Highway monitoring video high dynamic range fusion enhancement method and system based on NPU

Abstract

The invention discloses an NPU-based expressway monitoring video high dynamic range fusion enhancement method and system, and relates to the technical field of image enhancement processing, and the method comprises the steps: obtaining RAW data frames outputted by an image sensor supporting the LOFIC technology in a single physical exposure period; extracting a bright frame, a middle frame and a dark frame at the same exposure moment to form three RAW subframes; respectively carrying out ISP preprocessing to obtain three frames of RGB; splicing the three frames of RGB into 9 channels for input according to channels, inputting the neural network model operated by the NPU for fusion and enhancement, and outputting 3-channel high-dynamic range RGB; and encoding the output image and outputting to a storage or transmission interface. Through the technical scheme of the invention, the loss of details such as ghosting, overexposure and underexposure is reduced, and the monitoring imaging stability of tunnel entrances, backlighting and night vehicle lamp scenes is improved.

Description

A method and system for high dynamic range fusion enhancement of highway surveillance video based on NPU Technical Field

[0001] This invention relates to the field of image enhancement processing technology, and in particular to an NPU-based high dynamic range fusion enhancement method and an NPU-based high dynamic range fusion enhancement system for highway surveillance videos. Background Technology

[0002] Highway video surveillance systems need to continuously image and collect evidence of vehicles, road conditions, and emergencies under all weather and climate conditions. Especially in scenarios such as sudden changes in brightness at tunnel entrances and exits, backlighting and strong reflections during the day, and direct illumination from vehicle headlights and streetlights at night, the surveillance images contain both bright and dark areas. This can easily lead to overexposure in bright areas resulting in loss of detail and underexposure in dark areas, making it difficult to reliably identify key targets such as license plate characters, vehicle outlines, lane lines, and debris.

[0003] Existing wide dynamic range (WDR) enhancement solutions often employ methods such as multi-frame exposure fusion, continuous multi-frame fusion, or single-frame multi-gain synthesis to extend the dynamic range. Multi-frame exposure or continuous multi-frame fusion is prone to inter-frame displacement in high-speed moving target scenes, resulting in ghosting, trailing images, and blurring, which are difficult to completely eliminate in post-processing. Single-frame multi-gain solutions still have shortcomings in terms of dynamic range extension, strong light glare suppression, and low-light noise control. Meanwhile, strong beams directly hitting the lens at night can easily cause glare, halo diffusion, and localized blinding, obscuring effective target information; in low-light scenes, image noise is significant, and traditional noise reduction processing may introduce detail loss or motion blur. Other methods improve imaging through light-blocking structures or supplemental lighting, but these suffer from poor adaptability, unstable effects, light pollution, or high maintenance costs.

[0004] Therefore, existing technologies still struggle to simultaneously meet the requirements of ultra-high dynamic range extension, strong light glare suppression, low-light noise control, and real-time processing under the complex lighting conditions of highways. There is an urgent need for a video high dynamic range fusion enhancement scheme for highway monitoring to achieve stable and engineerable high-quality output under the constraints of edge computing power and latency. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a high dynamic range (HDR) fusion enhancement method for highway surveillance videos based on an NPU. This method utilizes a LOFIC image sensor to output bright, mid, and dark frames at the same physical exposure time within a single physical exposure cycle, extracting three original RAW subframes. This avoids inter-frame shift and ghosting caused by multiple exposures, meeting the input consistency requirements of high-speed motion scenes. The three RAW frames are preprocessed using an ISP pipeline to obtain three RGB frames, which are then stitched together to form a 9-channel input. A preset neural network model is run on the NPU to perform fusion enhancement, outputting a 3-channel HDR RGB image. This achieves synchronous recovery of overexposed details in bright areas and underexposed information in dark areas, while also considering strong light glare suppression and noise reduction. The encoded output stream is then connected to a storage or transmission interface to form an end-deployable HDR video link, improving the stability and evidence collection effectiveness of surveillance imaging in scenarios with sudden changes in brightness at tunnel entrances, backlighting, and nighttime vehicle headlights.

[0006] To achieve the above objectives, this invention provides a high dynamic range (HDR) fusion enhancement method for highway surveillance videos based on an NPU, comprising: acquiring a sequence of raw RAW data frames output by an image sensor supporting LOFIC technology within a single physical exposure cycle; extracting bright, mid, and dark frames corresponding to the same physical exposure time from the raw RAW data frame sequence to obtain three raw RAW sub-frames; performing ISP pipeline preprocessing on the three raw RAW sub-frames respectively to obtain three RGB images corresponding to the bright, mid, and dark frames; stitching the three RGB images into a 9-channel input and inputting it into a preset neural network model running on the NPU for fusion enhancement processing, outputting a 3-channel HDR RGB image; performing video encoding on the HDR RGB image to obtain a bitstream, and outputting the bitstream to a storage unit or transmission interface.

[0007] In the above technical solution, preferably, the original RAW data frame sequence consists of bright frames, medium frames and dark frames at the same physical exposure time, so that the bright frames, medium frames and dark frames share the optical image at the same physical exposure time.

[0008] In the above technical solution, preferably, the bright frame is a long exposure high conversion gain HCG subframe, the medium frame is a long exposure low conversion gain LCG subframe, and the dark frame is a short exposure low conversion gain LCG subframe. The image sensor outputs the bright frame, medium frame, and dark frame through overlapping sampling within the single physical exposure cycle.

[0009] In the above technical solution, preferably, bright frames, medium frames, and dark frames corresponding to the same physical exposure time are extracted from the original RAW data frame sequence. The specific process includes: the CPU receives the original RAW data frame sequence through a high-speed data interface, parses the subframe identification information in the original RAW data frame sequence used to indicate the frame types of bright frames, medium frames, and dark frames, and splits the original RAW data frame sequence accordingly to obtain the three original RAW subframes.

[0010] In the above technical solution, preferably, the ISP pipeline preprocessing includes at least dynamic range compression, tone mapping, defogging, and local contrast enhancement, and the tone mapping is set after the dynamic range compression, and the local contrast enhancement is set after the defogging.

[0011] In the above technical solution, preferably, the ISP pipeline preprocessing further includes automatic exposure, white balance correction, noise reduction, gamma correction, saturation adjustment, and sharpening; in the ISP pipeline preprocessing, parameter configuration interfaces are set for automatic exposure, white balance correction, noise reduction, dynamic range compression, tone mapping, gamma correction, saturation adjustment, dehazing, local contrast enhancement, and sharpening, respectively, for receiving parameter tuning parameters and updating the processing parameters of the corresponding processing nodes.

[0012] In the above technical solution, preferably, when a model update instruction is received, the neural network model to be loaded is received through the model configuration and update interface, and the neural network model to be loaded is loaded into the neural network processor (NPU) to replace the preset neural network model, wherein the neural network model to be loaded satisfies the constraint of 9-channel input and 3-channel output.

[0013] In the above technical solution, preferably, the preset neural network model includes an initial feature extraction layer, a residual block group, a multi-scale feature enhancement module, a feature mapping layer, a feature fusion layer, a residual optimization layer, and an output layer connected in sequence; wherein, the initial feature extraction layer is used to perform convolutional feature extraction on the 9-channel input; the residual block group is used to perform residual learning on the feature extraction results; the multi-scale feature enhancement module is used to perform multi-scale enhancement on the features after residual learning; the feature mapping layer is used to perform channel mapping on the features output by the multi-scale feature enhancement block group to generate intermediate features with the same number of channels as the 9-channel input, so as to be concatenated with the 9-channel input; the feature fusion layer is used to perform nonlinear activation and convolutional fusion on the concatenated features to obtain fused features; the residual optimization layer is used to generate residual optimization results based on the 9-channel input and the fusion results; the output layer is used to perform convolutional mapping on the residual optimization results to output the 3-channel high dynamic range RGB image.

[0014] This invention also proposes an NPU-based high dynamic range fusion enhancement system for highway surveillance videos, used to implement the NPU-based high dynamic range fusion enhancement method for highway surveillance videos disclosed in any of the above technical solutions. The system includes an image sensor supporting LOFIC technology, a processor, an ISP processing unit, a neural network processor (NPU), and an encoding output unit. The image sensor outputs a sequence of raw RAW data frames within a single physical exposure cycle. The processor, connected to the image sensor, extracts bright, medium, and dark frames corresponding to the same physical exposure time from the raw RAW data frame sequence to obtain three raw RAW sub-frames. The ISP processing unit and the processor... The system is configured to perform ISP pipeline preprocessing on the three original RAW subframes to output three RGB images corresponding to the bright, medium, and dark frames. The neural network processor (NPU) is connected to the ISP processing unit and is configured to concatenate the three RGB images into a 9-channel input, and run a preset neural network model on the NPU to perform fusion enhancement processing on the 9-channel input to output a 3-channel high dynamic range RGB image. The encoding output unit is connected to the NPU and is configured to perform video encoding on the high dynamic range RGB image to obtain a bitstream, and output the bitstream to at least one of a storage unit and a transmission interface.

[0015] In the above technical solution, preferably, the NPU sets a model configuration and update interface to load a replacement model that meets the 9-channel input and 3-channel output dimension constraints; the ISP processing unit sets parameter configuration interfaces for automatic exposure, white balance correction, noise reduction, dynamic range compression, tone mapping, gamma correction, saturation adjustment, dehazing, local contrast enhancement and sharpening respectively.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By using an image sensor that supports LOFIC technology to output bright, medium and dark frames at the same physical exposure time in a single physical exposure cycle, and extracting three original RAW sub-frames, the input data shares the optical image at the same time, reducing the risk of inter-frame displacement and ghosting caused by multiple exposures or cross-frame fusion in high-speed motion scenes, and providing a consistent data basis for subsequent HDR fusion.

[0017] (2) By performing ISP pipeline preprocessing on three original RAW subframes respectively, and then arranging nodes such as dynamic range compression followed by tone mapping, dehazing followed by local contrast enhancement to output three RGB images, the basic visibility and local detail performance of bright and dark areas are improved, and more stable input features are provided for NPU-side fusion enhancement.

[0018] (3) By stitching three RGB images into a 9-channel input and running a preset neural network model on the NPU to perform fusion enhancement processing, the output of a 3-channel high dynamic range RGB image is achieved. This realizes the simultaneous restoration of overexposed details in bright areas and enhancement of underexposed information in dark areas, and suppresses strong light glare and noise, thereby improving the imaging clarity and usability in complex lighting scenarios.

[0019] (4) By encoding the high dynamic range RGB image into video and outputting it to the storage unit or transmission interface, a video output link that can be deployed on the end side is formed, which facilitates the real-time access and evidence collection of the monitoring platform, and improves the monitoring stability and evidence collection effectiveness in scenarios such as sudden changes in brightness at tunnel entrances and exits, backlighting and nighttime vehicle lights. Attached Figure Description

[0020] Figure 1 is a flowchart illustrating the high dynamic range fusion enhancement method for highway surveillance video based on NPU according to an embodiment of the present invention; Figure 2 is a flowchart illustrating the ISP pipeline preprocessing according to an embodiment of the present invention; Figure 3 is a flowchart illustrating the fusion enhancement processing of the neural network model according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The present invention will now be described in further detail with reference to the accompanying drawings: As shown in Figure 1, the present invention provides a method for enhancing high dynamic range (HDR) fusion of highway monitoring videos based on an NPU, comprising: deploying an image sensor supporting LOFIC technology at a highway monitoring point; acquiring a set of raw RAW data frame sequences output by the image sensor supporting LOFIC technology within a single physical exposure cycle; extracting bright frames, mid-frames, and dark frames corresponding to the same physical exposure time from the raw RAW data frame sequence to form three raw RAW sub-frames, thereby ensuring that the three frames are consistent in the time dimension from the source and reducing the risk of inter-frame displacement caused by high-speed vehicle movement.

[0023] Subsequently, ISP pipeline preprocessing is performed on the three original RAW subframes to obtain three RGB images corresponding to the bright, medium, and dark frames. The three RGB images are then stitched together to form a 9-channel input, which is then input to a preset neural network model running on the neural network processor NPU for fusion enhancement processing. The output is a 3-channel high dynamic range RGB image, which preserves the details in the bright areas and the information in the dark areas in the same output image.

[0024] The output end performs video encoding on the high dynamic range RGB image to obtain a bitstream, and outputs the bitstream to the storage unit or transmission interface to form a high dynamic range monitoring video link that can be directly landed on the end side.

[0025] In this implementation, the loss of key information caused by overexposure and underexposure is significantly reduced in scenarios such as backlighting, sudden changes in brightness at tunnel entrances, and strong headlights at night, thus improving the stability of video output.

[0026] In the above embodiments, preferably, the original RAW data frame sequence consists of bright frames, medium frames, and dark frames at the same physical exposure time, so that the bright frames, medium frames, and dark frames share the optical image at the same physical exposure time. Compared with HDR schemes that use multiple exposures or frame capture across time periods, the three input frames maintain consistency in the time dimension, making it easier to suppress detail contamination caused by ghosting, trailing images, and misalignment when facing high-speed moving targets, and providing a stable input boundary for subsequent ISP and NPU fusion enhancement.

[0027] In the above embodiments, preferably, the bright frame is a long exposure high conversion gain HCG subframe, the medium frame is a long exposure low conversion gain LCG subframe, and the dark frame is a short exposure low conversion gain LCG subframe. The image sensor outputs the bright frame, medium frame, and dark frame through overlapping sampling within a single physical exposure cycle.

[0028] In this implementation, HCG long exposure subframes enhance dark area signals and signal-to-noise ratio, LCG long exposure subframes improve intermediate grayscale levels and dynamic range, and short exposure LCG subframes preserve bright areas without overexposure details. The three work together within the same exposure cycle to cover the distribution of bright / medium / dark information in high dynamic range scenes.

[0029] In the above implementation, preferably, bright frames, mid-frames, and dark frames corresponding to the same physical exposure time are extracted from the original RAW data frame sequence. Specifically, the processing side receives the original RAW data frame sequence via a high-speed data interface from the CPU. The CPU parses the subframe identifier information in the original RAW data frame sequence that indicates the frame types of bright, mid, and dark frames, and splits the original RAW data frame sequence accordingly to obtain three original RAW subframes. This splitting based on frame type identifiers, with the splitting logic consistent with the sensor output encapsulation method, reduces the probability of splitting ambiguity and frame errors, ensuring frame type matching and data integrity of the subsequent three-channel ISP and NPU fusion input.

[0030] As shown in Figure 2, in the above embodiment, preferably, the three original RAW subframes are respectively entered into the ISP pipeline for preprocessing, which includes at least dynamic range compression, tone mapping, defogging, and local contrast enhancement. The tone mapping is set after dynamic range compression, and the local contrast enhancement is set after defogging.

[0031] During implementation, dynamic range compression first compresses and normalizes the dynamic range of highlights and shadows. Then, tone mapping redistributes the compressed brightness and darkness levels, reducing highlight clipping and shadow collapse. Defogging improves contrast reduction caused by fog, rain, or strong scattering. Subsequent local contrast enhancement further enhances local texture and edge detail. Tailored for high-dynamic-range and high-reflection monitoring scenarios on highways, it effectively suppresses strong light and halos, preserving brightness and darkness details and color realism while compressing the dynamic range, ensuring rich visual information is presented even on ordinary devices in high-contrast scenes (such as backlighting and night scenes). Through these implementation methods, the RGB base image levels are more stable in high-reflection, fog, and tunnel entrance contrast scenarios, and the detail contrast is more conducive to subsequent NPU fusion enhancement convergence.

[0032] In the above embodiments, preferably, the ISP pipeline preprocessing further includes automatic exposure, white balance correction, noise reduction, gamma correction, saturation adjustment, and sharpening; in the ISP pipeline preprocessing, parameter configuration interfaces are set for automatic exposure, white balance correction, noise reduction, dynamic range compression, tone mapping, gamma correction, saturation adjustment, dehazing, local contrast enhancement, and sharpening, respectively, for receiving parameter adjustment parameters and updating the processing parameters of the corresponding processing nodes.

[0033] During implementation, the parameter configuration interface updates the target brightness, color temperature gain, noise reduction intensity, compression curve, mapping curve, Gamma curve, saturation coefficient, defogging intensity, local contrast gain, and sharpening intensity of each node, ensuring that the ISP output maintains a consistent style and controllable boundaries under different operating conditions. In field deployment, ISP parameters can be adjusted according to road lighting characteristics and weather changes, reducing the risk of over-compression causing graying details or over-sharpening leading to noise amplification, thus improving long-term operational stability.

[0034] In the above embodiments, preferably, when a model update instruction is received, the neural network model to be loaded is received through the model configuration and update interface, and the neural network model to be loaded is loaded into the neural network processor (NPU) to replace the preset neural network model. The neural network model to be loaded satisfies the constraint of 9-channel input and 3-channel output.

[0035] During implementation, the updated model was integrated into the existing fusion enhancement process without changing the 9-channel input and 3-channel output data interfaces. The model iteration was completed without altering the sensor acquisition, ISP preprocessing, and encoding output links, thus maintaining the established data flow and interface stability, improving system maintainability and version evolution efficiency, and meeting the continuous optimization needs under different road sections and seasonal conditions.

[0036] As shown in Figure 3, in the above embodiment, preferably, the preset neural network model includes an initial feature extraction layer, a residual block group, a multi-scale feature enhancement module, a feature mapping layer, a feature fusion layer, a residual optimization layer, and an output layer connected in sequence.

[0037] In the implementation process, the initial feature extraction layer is used to extract convolutional features from the 9-channel input; the residual block group is used to perform residual learning on the feature extraction results to stabilize the gradient and preserve details; the multi-scale feature enhancement module is used to perform multi-scale enhancement on the residual-learned features to take into account both large-scale halos and small-scale textures; the feature mapping layer is used to perform channel mapping on the features output by the multi-scale feature enhancement block group to generate intermediate features with the same number of channels as the 9-channel input, so as to be concatenated with the 9-channel input; the feature fusion layer is used to perform nonlinear activation and convolutional fusion on the concatenated features to obtain fused features; the residual optimization layer is used to generate residual optimization results based on the 9-channel input and the fusion results to achieve joint suppression of noise, blur and strong light interference; the output layer is used to perform convolutional mapping on the residual optimization results to output a 3-channel high dynamic range RGB image.

[0038] In this embodiment, under the superposition of strong light glare, low illumination, high noise, and high speed motion blur, the detail integrity and clarity of the fusion enhancement output are improved, while the artifacts and halo diffusion phenomena are reduced.

[0039] Specifically, in one specific embodiment, a specific example of the preset neural network model is as follows: 1) Original input x: 9-channel image (3 frames of bright, medium and dark RGB images).

[0040] 2) Preliminary feature extraction convolutional layer (Conv Input): Conv2d(9→64, kernel=3×3, padding=1)+ BatchNorm2d + ReLU.

[0041] 3) Residual Blocks x4: Each block structure: - Conv2d(64→64, 3×3) ×2 - InstanceNorm2d(64) - LeakyReLU(0.2) - Identity Skip Connection.

[0042] 4) Multi-scale Feature Enhancement Module (MSFE Blocks x3): Each block structure: - Conv2d(64→64, 3×3, padding=2, dilation=2)+ BatchNorm2d + ReLU - Conv2d(64→64, 3×3, padding=1, dilation=1)+ BatchNorm2d + ReLU.

[0043] 5) Intermediate feature mapping and fusion convolutional layer (Conv Output: x1) Conv2d(64→9, 3×3, stride=1).

[0044] 6) Concatenation and Fusion: Concatenate the intermediate features with the initial map features, and perform hyperbolic tangent activation and convolution fusion. Concatenation: Original input x (9 channels) + concatenated x1 (9 channels) → Concatenated feature map: x_cat Fusion: nn.Tanh()Conv2d(18→9, 1×1, stride=1, padding=0) → Fusion feature map: x_fused.

[0045] 7) Residual optimization: original input x - x_fused * x1, noise reduction and blur removal, strong light suppression map.

[0046] 8) Output Convolutional Layer (Conv Output) Conv2d(9→3, 1×1, stride=1, padding=0).

[0047] Output a 3-channel HDR image (High Dynamic Range Result).

[0048] For the aforementioned neural network model, during implementation, the HiSilicon Hi3519AV200's built-in WDR mode was used to access sensor data and complete 3-frame fusion. Simultaneously, it outputs the three bright-mid-dark images before fusion and a corresponding fused image at the same time, ensuring the positions of the images before and after fusion remain unchanged. In the image lab, noise reduction, sharpening, and aperture were disabled, or ambient light was adjusted to obtain noisy and blurry low-quality bright-mid-dark frames. Noise reduction, sharpening, aperture, and ambient light were then enabled, and fusion parameters were adjusted to obtain low-noise, clear, high-quality fused HDR frames. Batch recordings were performed under different scenes and illumination levels. The footage was then further optimized using AI image-to-image software to obtain higher-quality HDR images. This process allowed for the selection of a large number of high- and low-quality image pairs from the lab (3 low-quality bright-mid-dark images with "high noise and blur" correspond to 1 high-quality ideal HDR image with "low noise, clear, and weak halo").

[0049] The same method can be used to obtain a large number of high- and low-quality image pairs of real-world scenes, which can then be fed into an improved deep convolutional neural network based on ADnet combined with residual networks for training. The training process and optimization strategies ensure model convergence stability and strong generalization ability. The training process includes the following key steps: a) Model initialization: The convolutional kernel weights are initialized using a He normal distribution, and the bias term is set to zero to improve the stability of the initial training; b) Dynamic learning rate adjustment: Combining cosine annealing and hot restart (SGDR) strategies, the initial learning rate is set to 10−5, and periodically decayed to escape local optima; c) Reconstruction loss (L1 / L2 Loss): Constraining the pixel-level consistency between the output and the target image; Perceptual loss: Based on the VGG feature space difference, improving visual fidelity; Halo suppression specific loss: Designing a region-weighted loss based on brightness gradient and ring mask to strengthen the suppression of artifacts around strong light sources such as car lights and billboards; d) Backpropagation and parameter update, model saving and breakpoint continuation mechanism: Through the above process, the network gradually establishes an end-to-end mapping function from low-quality observed images to ideal high-quality images in multiple iterations, improving training efficiency while achieving robust modeling and high-performance reconstruction of complex degradation patterns.

[0050] The image enhancement model trained by this invention is dedicated to solving the core visual degradation problem in highway monitoring scenarios. The specific training objectives are: (1) Strong halo suppression: effectively eliminate local overexposure and radial halo artifacts caused by various artificial light sources (such as vehicle high beams, tunnel lighting, construction warning lights, and luminous signs), and restore the road structure and target outline that are covered by strong light.

[0051] (2) Image clarity enhancement: Improves the recognizability of vehicle appearance details (such as license plate characters, body texture), pedestrian features (clothing, posture) and small foreign objects (such as spilled objects, scattered goods), supporting high-precision target detection and recognition; (3) Signal-to-noise ratio enhancement: Significantly suppresses readout noise and dark current noise in low light or no light environment, enhances image texture and edge continuity, and avoids key features being obscured by noise.

[0052] (4) Adaptability to all lighting conditions: Ensure that the model can output high-fidelity images with consistent visual quality in diverse environments such as normal lighting, backlight, nighttime, fog, haze, rain and snow.

[0053] (5) Reliable imaging in all weather conditions; Perception consistency is maintained: During the enhancement process, color authenticity, geometric invariance and scene semantic integrity are maintained to avoid the risk of misjudgment caused by over-sharpening or the introduction of artifacts.

[0054] This invention also proposes an NPU-based high dynamic range fusion enhancement system for highway surveillance videos, used to implement the NPU-based high dynamic range fusion enhancement method for highway surveillance videos disclosed in any of the above embodiments. The system includes an image sensor supporting LOFIC technology, a processor, an ISP processing unit, a neural network processor (NPU), and an encoding output unit. The image sensor outputs a sequence of raw RAW data frames within a single physical exposure cycle. The processor, connected to the image sensor, extracts bright, medium, and dark frames corresponding to the same physical exposure time from the raw RAW data frame sequence to obtain three raw RAW sub-frames. The ISP processing unit... The unit is connected to the processor and is used to perform ISP pipeline preprocessing on three original RAW subframes respectively to output three RGB images corresponding to bright, medium and dark frames. The neural network processor (NPU) is connected to the ISP processing unit and is used to stitch the three RGB images into a 9-channel input by channel, and run a preset neural network model on the NPU to perform fusion enhancement processing on the 9-channel input to output a 3-channel high dynamic range RGB image. The encoding output unit is connected to the NPU and is used to perform video encoding on the high dynamic range RGB image to obtain a bitstream, and output the bitstream to at least one of the storage unit and the transmission interface.

[0055] In this implementation, the system forms a closed-loop data collection, preprocessing, NPU fusion, and encoding output, meeting the requirements for real-time deployment and remote access of highway monitoring.

[0056] In the above embodiments, preferably, the NPU sets up a model configuration and update interface for loading a replacement model that meets the 9-channel input and 3-channel output dimension constraints; the ISP processing unit sets up parameter configuration interfaces for automatic exposure, white balance correction, noise reduction, dynamic range compression, tone mapping, gamma correction, saturation adjustment, dehazing, local contrast enhancement and sharpening respectively.

[0057] During implementation, the ISP side achieves on-site commissioning and stable output through parameter configuration interface, while the NPU side achieves model version upgrade through model configuration and update interface. The two achieve parameter-level adaptation and model-level iteration without changing the main data flow. Under long-term operation and multi-scenario deployment conditions, the system maintains image quality consistency and continuous optimization capability through two paths: parameters and models, which reduces operation and maintenance costs and improves project scalability.

[0058] The high dynamic range fusion enhancement method and system for highway surveillance video based on NPU disclosed in the above embodiments shall be implemented with reference to the detailed steps in the following embodiments.

[0059] Step 1: Outputting 3-Frame Image Data from a Single Exposure Based on LOFIC Technology. Using a CV3012 sensor supporting LOFIC technology, the sensor is initialized and configured into LOFIC mode via the CPU (integrated SPI master control interface). The appropriate exposure ratio is set in the image sensor registers via the AE module and SPI interface (overlapping readout of 3 subframes: long exposure HCG (bright frame), long exposure LCG (medium frame), and short exposure LCG (dark frame)). Within a single physical exposure time, 3 frames of image data with different exposures are output in an overlapping manner. (For example, dark frames capture highlight details, medium frames retain sub-dark details, and bright frames retain shadow information). Specifically, the sensor alternately outputs subframes with different exposure times during the exposure period, ultimately outputting a set of raw data frame sequences containing high dynamic range information. All subframes share the optical image at the same physical exposure time, avoiding the inter-frame shift problem of traditional multi-frame fusion technology.

[0060] Step 2: Split the mixed frame sequence into subframes. The CPU uses a high-speed interface to split the set of raw data frame sequences containing high dynamic range information output by the image sensor in Step 1 into 3 bright, medium and dark subframe image data.

[0061] Step 3: Multi-dimensional ISP pipeline processing based on NPU. The three frames of bright, medium, and dark image data obtained in Step 2 are processed using a multi-dimensional ISP pipeline within the NPU. The ISP pipeline is as follows: 1) Black level correction (calibration value removal); 2) Bad pixel correction (adaptive intensity); 3) Auto exposure (AE, baseline value 56); 4) White balance correction (calibrated white balance); 5) Color correction (calibrated LUT mapping); 6) Dynamic range compression (DRC, linked to ISO settings, dynamic intensity increases with ISO, resulting in smoother subsequent AI fusion and better enhancement of key details); 7) Tone mapping; 8) Noise reduction (linked to ISO settings, lower ISO results in lower noise reduction intensity, higher ISO results in higher noise reduction intensity; at night, adjusting temporal intensity and motion detection parameters prevents motion blur of high-speed vehicles). Large-particle random noise is significantly suppressed; spatial noise reduction intensity is adjusted to maintain detail without loss, and small noise is effectively removed); 9) Demosaic (Bayer→RGB); 10) Gamma correction (non-linear brightness adjustment, value 2.0); 11) Defogging (adaptive, adjusts defogging intensity based on fog level); 12) Local contrast enhancement (LDCI, linked to ISO setting, enhances LDCI in wide dynamic range scenes); 13) Saturation adjustment (CA, linked to ISO setting, reduces saturation at night when ISO is high, and increases saturation when ISO is low); 14) Sharpening (linked to ISO setting, ensures high detail and edge sharpening intensity and high image clarity when ISO is low under sufficient ambient light, and reduces sharpness when ISO is high in low light to avoid generating significant noise); 15) Output: High dynamic range RGB image (3 channels).

[0062] Among them, the parameters of AE, AWB, NR, DRC, Tonemapping, GAMMA, CA, LDCI, and EE steps are all designed with corresponding open and fine-tunable interfaces, which can be used to further optimize the parameters on site to adapt to the scenario.

[0063] Step 4: AI Image Fusion and Image Enhancement Based on NPU. The preprocessed image data from Step 3 is then processed using a lightweight AI image enhancement network model trained with embedded NPU (model structure shown in Figure 1) for image denoising and deblurring. Simultaneously, the RGB values ​​of overexposed pixels are mapped to a reasonable dynamic range, while glare halo diffusion is suppressed. The network's training data includes numerous high- and low-quality image pairs from real-world laboratory and highway scenes. Learning objectives include: Strong Halo Suppression: Effectively eliminating local overexposure and radial halo artifacts caused by various artificial light sources (such as vehicle high beams, tunnel lighting, construction warning lights, and illuminated signs), restoring road structures and target outlines obscured by strong light; Image Clarity Enhancement: Improving the discernibility of vehicle exterior details (such as body texture), pedestrian features (posture), and small foreign objects (such as spilled materials and scattered goods), supporting high-precision target detection and recognition; Signal-to-Noise Ratio Improvement: Significantly suppressing readout noise and dark current noise in low-light or no-light environments, enhancing image texture and edge continuity, and preventing key features from being obscured by noise.

[0064] Step 5: Video Encoding, Storage and Transmission. The video stream processed in Step 4 is compressed and encoded, then stored or transmitted in real time to the monitoring center.

[0065] Traditional solutions suffer from overexposure in bright areas, resulting in loss of detail, and underexposure in low-light areas at night, making them unrecognizable. When headlights or streetlights shine directly into the lens at night, glare (halo diffusion) is easily produced. In the video processed by this invention, the details in both bright and dark areas are complete, with a real-time frame rate of 30fps, meeting the needs of high-speed monitoring.

[0066] This invention significantly improves the imaging quality of highway monitoring under extreme lighting conditions through multi-technology collaborative innovation. It can be widely applied to intelligent monitoring systems in key areas such as highway mainlines, ramps, tunnels, and service areas, providing highly reliable visual data sources for vehicle-to-everything (V2X), autonomous driving assistance, traffic law enforcement evidence collection, and emergency rescue, and strongly supporting the goal of building a smart highway that is "visible, measurable, controllable, and serviceable".

[0067] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for high dynamic range fusion enhancement of highway surveillance video based on NPU, characterized in that, include: Acquire a sequence of raw RAW data frames output by an image sensor that supports LOFIC technology within a single physical exposure cycle; The bright frame, medium frame and dark frame corresponding to the same physical exposure time are extracted from the original RAW data frame sequence to obtain three original RAW sub-frames; The three original RAW subframes are preprocessed using an ISP pipeline to obtain three RGB images corresponding to the bright, medium, and dark frames. The three RGB images are then concatenated by channel to form a 9-channel input, which is then fed into a preset neural network model running on the Neural Processing Unit (NPU) for fusion enhancement processing, outputting a 3-channel high dynamic range (HVR) RGB image. The HVR RGB image is then video encoded to obtain a bitstream, which is then output to a storage unit or transmission interface.

2. The method for high dynamic range fusion enhancement of highway surveillance video based on NPU according to claim 1, characterized in that, The original RAW data frame sequence consists of bright frames, medium frames, and dark frames at the same physical exposure time, so that the bright frames, medium frames, and dark frames share the optical image at the same physical exposure time.

3. The method for high dynamic range fusion enhancement of highway surveillance video based on NPU according to claim 2, characterized in that, The bright frame is a long-exposure high-conversion-gain HCG subframe, the medium frame is a long-exposure low-conversion-gain LCG subframe, and the dark frame is a short-exposure low-conversion-gain LCG subframe. The image sensor outputs the bright frame, medium frame, and dark frame through overlapping sampling within the single physical exposure cycle.

4. The method for high dynamic range fusion enhancement of highway surveillance video based on NPU according to claim 1, characterized in that, Extracting bright, medium, and dark frames corresponding to the same physical exposure time from the original RAW data frame sequence involves the following steps: the CPU receives the original RAW data frame sequence via a high-speed data interface, parses the subframe identification information in the original RAW data frame sequence used to indicate the frame types of bright, medium, and dark frames, and then splits the original RAW data frame sequence into the three original RAW subframes accordingly.

5. The method for high dynamic range fusion enhancement of highway surveillance video based on NPU according to claim 1, characterized in that, The ISP pipeline preprocessing includes at least dynamic range compression, tone mapping, defogging, and local contrast enhancement, with tone mapping performed after dynamic range compression and local contrast enhancement performed after defogging.

6. The method for high dynamic range fusion enhancement of highway surveillance video based on NPU according to claim 5, characterized in that, The ISP pipeline preprocessing further includes automatic exposure, white balance correction, noise reduction, gamma correction, saturation adjustment, and sharpening. In the ISP pipeline preprocessing, parameter configuration interfaces are set for automatic exposure, white balance correction, noise reduction, dynamic range compression, tone mapping, gamma correction, saturation adjustment, dehazing, local contrast enhancement, and sharpening, respectively, to receive parameter tuning parameters and update the processing parameters of the corresponding processing nodes.

7. The method for high dynamic range fusion enhancement of highway surveillance video based on NPU according to claim 1, characterized in that, When a model update instruction is received, the neural network model to be loaded is received through the model configuration and update interface, and the neural network model to be loaded is loaded into the neural network processor (NPU) to replace the preset neural network model. The neural network model to be loaded satisfies the constraint of 9-channel input and 3-channel output.

8. The method for high dynamic range fusion enhancement of highway surveillance video based on NPU according to claim 7, characterized in that, The preset neural network model includes an initial feature extraction layer, a residual block group, a multi-scale feature enhancement module, a feature mapping layer, a feature fusion layer, a residual optimization layer, and an output layer connected in sequence. Specifically, the initial feature extraction layer performs convolutional feature extraction on the 9-channel input; the residual block group performs residual learning on the feature extraction results; the multi-scale feature enhancement module performs multi-scale enhancement on the residual-learned features; the feature mapping layer maps the features output by the multi-scale feature enhancement block group to generate intermediate features with the same number of channels as the 9-channel input, for concatenation with the 9-channel input; the feature fusion layer performs nonlinear activation and convolutional fusion on the concatenated features to obtain fused features; the residual optimization layer generates residual optimization results based on the 9-channel input and the fusion results; and the output layer performs convolutional mapping on the residual optimization results to output the 3-channel high dynamic range RGB image.

9. A high dynamic range fusion enhancement system for highway surveillance video based on an NPU, characterized in that, The method for implementing the NPU-based high dynamic range fusion enhancement method for highway surveillance videos as described in any one of claims 1 to 8 includes an image sensor supporting LOFIC technology, a processor, an ISP processing unit, a neural network processor (NPU), and an encoding output unit; the image sensor is used to output a sequence of raw RAW data frames within a single physical exposure cycle. The processor is connected to the image sensor and is used to extract bright frames, mid-frames, and dark frames corresponding to the same physical exposure time from the original RAW data frame sequence to obtain three original RAW sub-frames. The ISP processing unit is connected to the processor and is used to perform ISP pipeline preprocessing on the three original RAW sub-frames to output three RGB images corresponding to the bright, mid, and dark frames. The neural network processor (NPU) is connected to the ISP processing unit and is used to stitch the three RGB images into a 9-channel input by channel, and run a preset neural network model on the NPU to perform fusion enhancement processing on the 9-channel input to output a 3-channel high dynamic range RGB image. The encoding output unit is connected to the NPU and is used to perform video encoding on the high dynamic range RGB image to obtain a bitstream, and output the bitstream to at least one of the storage unit and the transmission interface.

10. The NPU-based high dynamic range fusion enhancement system for highway surveillance video according to claim 9, characterized in that, The NPU sets up a model configuration and update interface for loading replacement models that meet the constraints of 9-channel input and 3-channel output dimensions; the ISP processing unit sets up parameter configuration interfaces for automatic exposure, white balance correction, noise reduction, dynamic range compression, tone mapping, gamma correction, saturation adjustment, dehazing, local contrast enhancement, and sharpening.

Images