Night low-illumination image restoration-light effect suppression method and intelligent camera device

Abstract

The invention discloses a night low-illumination image restoration-light effect suppression method and an intelligent camera device. In order to solve the problem of camera imaging in a night low-illumination environment, firstly, an image data set is self-made to provide accurate data support for network training, a lightweight CBAM-Net network is designed based on AI-ISP, and lightweight and accurate feature extraction is realized by using axial deep convolution in combination with a U-shaped basic network and a weight modulation module. Through a model acceleration strategy of concurrent reasoning by means of a pruning algorithm and a thread pool, computing resources of an embedded platform are adapted. The intelligent camera device integrates camera acquisition equipment, algorithm deployment and implantation, an edge computing platform and related driving and transmission interfaces, and realizes image acquisition, processing and display. The method can effectively improve the night low-illumination image quality and inhibit the light effect, and is suitable for the fields of security and protection, vehicle-mounted electronics and the like.

Description

Technical Field

[0001] The present invention relates to a method and an intelligent camera device capable of restoring a low-illuminance image in real time and suppressing light effects in a low-illuminance scene with a light source, and belongs to the technical field related to image processing and embedded systems. Background Art

[0002] Cameras are widely used in many industrial fields such as security and automotive electronics. However, in low-light environments at night, the images captured by cameras are often unclear and easily affected by various light effects, which poses a great challenge to the execution of high-level visual tasks.

[0003] Traditional ISP technology has obvious limitations when dealing with images under dark light conditions. Specifically, in key aspects such as light effect suppression, contrast enhancement, and dark light brightening, traditional ISPs are difficult to meet the actual needs of complex scenes. Most AI-ISP low-light image restoration methods and devices focus on the low-light area of ​​the scene, which makes the bright light source area prone to over-enhancement and saturation. The halo and brilliance around it will be magnified, causing light pollution, loss of image details, and reduced overall image quality. For example: the existing inventions "Imaging hardware system and method in extremely low-light environment based on Ascend" (CN118828223A), "A low-light visual enhancement method and device based on KinD algorithm" (CN118338135A) both have the disadvantage of being unable to suppress light effects. Moreover, most of the existing algorithm models have large number of parameters and high computing power requirements, and cannot be actually applied to industrial products.

[0004] The present invention discloses a method for restoring low-light images at night and suppressing light effects, and an intelligent camera device, in order to realize the design of a camera device that can restore images in low-light areas in real time and suppress light effects in light source areas in low-light scenes. The present invention builds a lightweight AI-ISP network with the above-mentioned restoration function, constructs a data set including low-light image restoration and light effect suppression effects, and provides data support for AI-ISP network training. Finally, through model pruning and thread pool parallel acceleration strategy, the algorithm is deployed based on an embedded platform to develop a method for restoring low-light images at night and suppressing light effects, and an intelligent camera device. Summary of the invention

[0005] The present invention aims to provide a nighttime low-illuminance image restoration-light effect suppression method and an intelligent camera device, which can effectively improve the quality of images collected by a camera in low-illuminance scenes with light sources at night.

[0006] The intelligent camera device designed by the present invention comprises:

[0007] Construction of datasets for low-light image restoration and suppression of light effects, design of lightweight AI-ISP algorithms, model deployment acceleration strategies, and construction of intelligent camera device systems.

[0008] When constructing image data for low-light image restoration and light effect suppression, first place a remotely controllable outdoor light source in the same scene, strictly ensure that the acquisition device remains in the same position and the surrounding environment is static, then increase the exposure time of the image acquisition device while adjusting the brightness of the outdoor light source to the minimum, and then take an RGB image of normal brightness as a clean label image for training. Then, gradually reduce the exposure time of the image acquisition device while adjusting the brightness of the outdoor light source to the highest level, and take a low-light RAW image at each level as training input. This cycle is repeated to capture the next set of data.

[0009] The design of the lightweight AI-ISP pipeline separates the color channels of the single-channel Bayer data collected by the sensor. After black level correction and desired magnification adjustment, it enters the CBAM-Net network for deep feature extraction and integration, and finally restores the image size through pixel shuffling technology.

[0010] The CMAM-Net combines a U-shaped basic network with a weight modulation module. The basic network adopts a lightweight U-shaped structure, which contains a specific number of encoding blocks, bottleneck layers, and decoding blocks. All of them adopt a lightweight structure based on axial depth separable convolution modules to reduce complexity and improve feature capture capabilities. The CBAM attention module is introduced in the encoding area, and the feature weights are dynamically adjusted according to the weighted operation of the spatial and channel information of the feature map, effectively integrating the features of the overexposed and light effect areas of the image into the global features.

[0011] The model acceleration strategy includes model structured pruning and concurrent inference deployment of board-side thread pool.

[0012] The model structured pruning utilizes a pruning algorithm to remove channels that contribute less to the model through three steps of L1 sparsification training, pruning, and fine-tuning, and selects important channels based on appropriate sparsity rate training to reduce the number of model parameters and the amount of calculation while ensuring model performance.

[0013] The board-side thread pool concurrent reasoning deployment, the present invention designs a thread pool strategy for concurrent reasoning, creates a fixed number of working threads, maintains the task queue, dynamically schedules tasks according to the CPU and NPU resource status, realizes dynamic task load balancing, and improves resource utilization efficiency. These threads remain active throughout the life cycle of the program, waiting for tasks to be assigned, avoiding the high overhead of frequently creating new threads each time parallel processing is required.

[0014] A task queue is maintained in the thread pool. When new inference requests arrive, they are efficiently packaged into tasks and placed in the task queue in sequence. The task queue uses the CPU to achieve precise synchronization control in a multi-threaded environment to ensure thread safety and prevent data inconsistency or conflict caused by multiple threads accessing and modifying the task queue at the same time.

[0015] The construction of the intelligent camera device system consists of an image acquisition module, an image real-time processing module and an image display module.

[0016] The real-time image processing module is composed of an embedded device deployed with an AI-ISP algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0019] Figure 2 This is the entire ISP pipeline structure for image enhancement and restoration of the present invention.

[0020] Figure 3 This is the CBAM-Net network structure of the present invention.

[0021] Figure 4 It is a diagram showing the effect of restoring the test processing of the algorithm designed by the present invention on a self-made data set.

[0022] Figure 5 It is a concurrent reasoning structure diagram of the thread pool designed by the present invention.

[0023] Figure 6 It is a structural diagram of the intelligent camera system device for low-light image restoration and light effect suppression designed by the present invention.

[0024] Figure 7 This is a test result diagram of an actual scene of the intelligent camera system device designed by the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0026] See also Figure 1 The present invention provides a nighttime low-light image restoration-light effect suppression method and an intelligent camera device, comprising the following steps:

[0027] S001, shoots a dataset with low-light restoration and light effect suppression capabilities.

[0028] First, place a remotely controllable outdoor light source in the same scene, strictly ensure that the acquisition device remains in the same position and the surrounding environment is static, then increase the exposure time of the image acquisition device while adjusting the brightness of the outdoor light source to the minimum, and take an RGB image of normal brightness as a clean label image for training. Then, gradually reduce the exposure time of the image acquisition device while adjusting the brightness of the outdoor light source to the highest level, and randomly take several low-light RAW images as training input. Cycle in this way to take about 370 sets of data in various scenes, of which about 30 sets are randomly used for test verification, and the rest are used for training.

[0029] S002, design and training optimization of AI-ISP network.

[0030] First, the single-channel Bayer data collected by the sensor is separated according to color and parsed into four color channel data to maximize the retention of the original information. To achieve more efficient processing, this operation reduces the spatial resolution by 2 times in each dimension. The data is then processed by black level correction and desired magnification adjustment (for example, x100 or x300) to optimize its input quality. Next, the four-channel data enters the CBAM-Net network designed in this paper for deep feature extraction and integration. The output of CBAM-Net is a 12-channel image with half the spatial resolution, which is restored to its original size through pixel shuffling technology to form a complete ISP pipeline structure, as shown in Figure 1. Figure 2 shown.

[0031] Specifically, the CBAM-Net network structure is as follows Figure 3As shown in the figure, the network consists of a U-shaped encoding-decoding network and a CBAM attention mechanism. The U-shaped network consists of 4 encoding blocks, 4 decoding blocks, and a bottom feature integration module, and a CMAM attention mechanism is added to each encoding block. The single-channel Bayer format raw image is separated into four color channels to obtain four-channel raw data with a size half of the original. This data is sent to the network for iterative training, and the hyperparameters are continuously adjusted to fine-tune and optimize the network performance.

[0032] In order to prove the effectiveness of the self-made dataset and network structure, the proposed CBAM-Net network and other networks were trained using the self-made dataset, and then the algorithm was compared and tested. Figure 4 As shown in the table below, the comparison results of related indicators are as follows, where the FLOPS indicator is measured under the condition of input size of 1*4*512*1024:

[0033]

[0034] The network structure proposed in this paper is compared with the other two algorithms in four indicators, where PSNR represents peak signal-to-noise ratio, SSIM represents structural similarity, Parameters represents network parameters, and FLOPS represents model calculation. The experimental results show that the algorithm proposed in this paper is better than U-net (Chen et al.) in PSNR and SSIM indicators, although slightly inferior to CAUNet, but the number of parameters and calculation is much smaller than the other two algorithms. Therefore, the network model of this paper is closer to the embedded deployment of practical applications.

[0035] S003, algorithm model deployment acceleration strategy (structured pruning + thread pool concurrent reasoning).

[0036] During the model training process, L1 sparsification training is first performed on the structured pruning part of the model. During the training, L1 sparsification training is performed on the gamma parameter of the BatchNorm layer according to the selected sparsity rate.

[0037] Specifically, during the training process, by adding the L1 regularization term to the loss function, the gamma parameter is gradually pushed close to 0, thereby achieving automatic screening of channels. During the training process, pay close attention to the distribution of gamma values, and ensure that channels that contribute more to the model are screened out by drawing gamma value distribution histograms, and then sort them according to the model contribution and perform pruning operations.

[0038] After pruning, use the original dataset to fine-tune the pruned model. During the fine-tuning process, dynamically adjust the number of training rounds based on actual test results, such as the PSNR and SSIM indicators on the validation set. Start with a smaller number of training rounds and gradually increase them while observing changes in model performance. When the model performance no longer improves or reaches the preset performance threshold, stop fine-tuning to ensure that the model maintains high performance indicators while reducing the number of parameters.

[0039] The algorithm is deployed and accelerated in the thread pool concurrent reasoning on the embedded board. The embedded device uses the RK358 platform and converts the pruned PyTorch model to the RKNN format to adapt to the RK3588 embedded platform. During the conversion process, the model configuration parameters are carefully set, such as the mean of the input data, normalization parameters, and quantization options, to ensure the accuracy of the model conversion and the final performance, which requires a trade-off between accuracy and resource utilization.

[0040] On the RK3588 platform, the model is deployed according to the thread pool concurrent reasoning strategy. The designed thread pool concurrent reasoning structure is as follows Figure 5 shown.

[0041] Specifically, when the model inference application is started, the designed thread pool will create a fixed number of worker threads after initialization. These threads remain active during the life cycle of the program, waiting for tasks to be assigned, avoiding the high cost of frequently creating new threads every time parallel processing is required. At the same time, a task queue is maintained in the thread pool. When new inference requests arrive, these requests are packaged into tasks and placed in the task queue in turn. The task queue is synchronized and controlled by the CPU in a multi-threaded environment to ensure thread safety. Each worker thread will continue to monitor the task queue. Once a new task is detected in the task queue, the idle thread will take a task from the queue and execute it. The working thread pool dynamically schedules tasks according to the current resource status of the CPU and NPU to ensure that all resources are fully utilized. When a thread completes a task, it immediately returns to the task queue to find the next task, thereby achieving dynamic load balancing of tasks. If the task queue is empty, the thread will enter a waiting state to save system resources. After completing all tasks, the thread pool will automatically recycle resources and manage threads to ensure efficient task execution and reasonable use of system resources.

[0042] The model is deployed on the RK3588 platform. The experimental test results show that the output frame rate is positively correlated with the number of threads when the thread pool is used for concurrent reasoning deployment. The size of the raw data collected by the camera is 1*768*1024. After image preprocessing, the original raw data is rearranged into 4*384*512 RGGB four-channel data and sent to the network. The model output data size is 3*768*1024. The relevant test results are shown in the following table:

[0043]

[0044] Among them, core0, core1, and core2 represent the three NPU computing cores of rk3588. According to the test results, this paper uses the thread pool strategy to efficiently utilize computing resources. Before reaching the maximum frame rate of the image acquisition device, the number of threads is increased. The thread pool strategy can increase the output frame rate by multiples, and can also achieve real-time processing effects including pre- and post-processing of image data. Therefore, this experiment verifies the effectiveness of using the thread pool strategy for concurrent reasoning deployment of multi-threaded tasks.

[0045] S004, Construction of intelligent camera system for low-light image restoration and light effect suppression

[0046] The system installation is as follows Figure 6 As shown in the figure, a camera with a Sony imx327 sensor is used to collect raw data through the V4L2 kernel driver development interface. Ensure that the camera parameters are set correctly, such as exposure time, gain, etc., to obtain high-quality raw image data. The collected data is preprocessed according to the design requirements, such as rearranging the original raw data into a format suitable for network input (such as 4*384*512 RGGB four-channel data).

[0047] The collected image data is pre-processed and deployed on the RK3588 platform with a lightweight AI-ISP algorithm for low-light image restoration and light effect suppression in real time. The processed image data is post-processed and then transmitted to the display device through the HDMI interface for real-time display, realizing the camera's high-quality image acquisition and display functions in low-light environments at night. The actual test results are as follows: Figure 7 shown.

[0048] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A method for restoring low-light images at night and suppressing light effects and an intelligent camera device, characterized in that: The following steps are involved: Construction of dataset for low-light image restoration and light effect suppression. Lightweight AI-ISP algorithm design. Model deployment acceleration strategy. Construction of intelligent camera device system.

2. The low-light image restoration and light effect suppression dataset construction as claimed in claim 1, characterized in that: First, place a remotely controllable outdoor light source in the same scene, strictly ensure that the acquisition device remains in the same position and the surrounding environment is static, then increase the exposure time of the image acquisition device while adjusting the brightness of the outdoor light source to the minimum, and take an RGB image of normal brightness as a clean label image for training. Then, gradually reduce the exposure time of the image acquisition device while adjusting the brightness of the outdoor light source to the maximum, and take a low-light RAW image at each level as training input. Repeat this method to capture the next set of data.

3. As described in claim 1, the lightweight AI-ISP algorithm design is characterized by: The single-channel Bayer data collected by the sensor is separated into color channels. After black level correction and desired magnification adjustment, it enters the CBAM-Net network for deep feature extraction and integration, and finally the image size is restored through pixel shuffling technology. The CMAM-Net combines a U-shaped basic network with a weight modulation module. The basic network adopts a lightweight U-shaped structure, which contains a specific number of encoding blocks, bottleneck layers, and decoding blocks. All of them adopt a lightweight structure based on axial depth separable convolution modules to reduce complexity and improve feature capture capabilities. The CBAM attention weight modulation module is introduced in the encoding area, and the feature weights are dynamically adjusted according to the weighted operation of the spatial and channel information of the feature map, effectively integrating the features of the overexposed and light effect areas of the image into the global features.

4. As described in claim 1, the model deployment acceleration strategy is characterized by: Model structured pruning and board-side thread pool concurrent inference deployment. The structured pruning of the model uses a pruning algorithm to remove channels that contribute less to the model through three steps: L1 sparsification training, pruning, and fine-tuning. Important channels are selected according to appropriate sparsity rate training to reduce the number of model parameters and calculations while ensuring model performance. The board-side thread pool concurrent reasoning deployment, thread pool strategy design for concurrent reasoning, create a fixed number of worker threads, maintain task queues, dynamically schedule tasks according to the CPU and NPU resource status, achieve dynamic task load balancing, and improve resource utilization efficiency. These threads remain active throughout the life cycle of the program, waiting for tasks to be assigned, avoiding the high overhead of frequently creating new threads every time parallel processing is required.

5. As claimed in claim 1, the construction of the intelligent camera device system is characterized by: The intelligent camera device system consists of an image acquisition module, an image real-time processing module and an image display module. The image real-time processing module is composed of an embedded device deployed with an AI-ISP algorithm.

Images