A dynamic light source adaptive camera system

Abstract

This invention discloses a dynamic light source adaptive camera system, including a multispectral sensor array, a spectral quality assessment module, a cross-spectral feature fusion unit, a scene detection module, an intelligent control unit, a light source driving module, and an image optimization module. Through simultaneous acquisition of three spectra and dynamic weight fusion, it perceives scene illumination and target information. Combined with a customized multi-task deep learning model optimized through mixed-precision quantization and structured pruning, it dynamically adjusts the brightness, emission angle, and spectral adaptation mode of multi-zone LED fill lights. It also coordinates multi-frame fusion, noise suppression, and local HDR fusion algorithms to achieve performance under adverse weather and low-light conditions. This system effectively solves problems of overexposure, underexposure, severe noise, and motion artifacts. It features low power consumption, high stability, and wide adaptability, making it suitable for various applications such as vehicle-mounted devices, smart door locks, security monitoring, drone aerial photography, medical endoscopes, and outdoor live streaming equipment.

Description

Technical Field

[0001] This invention relates to the field of camera imaging technology, specifically to a dynamic light source adaptive camera system that integrates multispectral sensing and deep learning lightweight technology, suitable for high-quality imaging under complex lighting conditions in various scenarios such as vehicles, security, smart homes, drones, and medical applications. Background Technology

[0002] With the rapid development of security monitoring, vehicle-mounted assisted driving, smart homes, and outdoor mobile devices, cameras, as core sensing components, directly determine the reliability and practicality of the system through their imaging quality in complex environments. While existing technologies such as automatic exposure (AE) and automatic gain control (AGC) can achieve basic brightness adjustment, they have significant drawbacks in scenarios such as nighttime, backlighting, and strong light interference: insufficient overall illumination at night leads to severe image noise and loss of detail; in backlighting scenarios, strong background light makes foreground objects too dark, making target identification difficult; and strong light interference easily causes localized overexposure and image distortion.

[0003] To address these issues, some systems employ infrared illumination or HDR algorithm optimization, but significant limitations remain: infrared illumination lacks specificity, often employing omnidirectional illumination, resulting in high ineffective power consumption and an inability to handle light scattering caused by adverse weather conditions such as fog, rain, and snow; traditional HDR algorithms require capturing multiple frames of images with different exposures, leading to motion artifacts and significantly increased power consumption; existing multispectral cameras are only used for image stitching, failing to form a closed loop with light source adjustment, and the fusion method involves fixed weight superposition, resulting in poor adaptability; deep learning models often employ generalized structures with redundant parameters, failing to meet the specific requirements of "light source adjustment + scene classification," leading to high latency and high power consumption on low-computing-power embedded devices.

[0004] Therefore, how to achieve accurate scene perception under severe weather conditions, efficient intelligent decision-making under low computing power conditions, and deep collaborative adaptive control of light source and imaging has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic light source adaptive camera system that, through the collaborative design of multispectral fusion perception and customized lightweight deep learning model, overcomes the limitations of single-spectral perception and the performance bottleneck of general models, and achieves low-power, high-definition, and highly stable imaging in complex lighting, harsh weather, and low-computing scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a dynamic light source adaptive camera system, comprising: A multispectral sensor array is used to synchronously acquire multispectral raw images of a scene. The multispectral sensor array includes at least a visible light sensor, a near-infrared sensor, and a thermal infrared sensor. The spectral quality assessment module is used to calculate the quality index of each spectral image and output scene type labels and spectral priority determination results. A cross-spectral feature fusion unit is used to dynamically allocate the weights of each spectrum based on an attention mechanism to generate a unified "lighting-target" fused feature map; The scene detection module is used to extract brightness distribution information, illumination defect type and target area coordinates based on the fused feature map. The scene detection module uses the YOLOv8 lightweight model for target recognition. The extracted brightness distribution information includes brightness histogram, highlight area ratio, shadow area ratio and contrast. The detection delay is ≤50ms. The intelligent control unit is used to calculate the light source adjustment parameters, spectral adaptation weights, and upper limit of supplementary lighting power based on a customized multi-task deep learning model, combined with the fused feature map and hardware power consumption data. The loss function of the customized multi-task deep learning model is a weighted joint loss, in which the supplementary lighting parameter loss weight is 0.5, the spectral weight loss is 0.3, and the power consumption loss is 0.2. The light source driving module is used to adjust the brightness, emission angle and spectral adaptation mode of the supplementary light source according to the light source adjustment parameters. The supplementary light source is a multi-zone LED array that supports zoned directional supplementary light. The LED array of the supplementary light source has at least a 4×4 zone structure, supports independent dimming of a single zone, and adapts to the directional supplementary light requirements in different scenarios. The image optimization module is used to perform multi-frame fusion, noise suppression, and local HDR fusion processing on the multispectral fusion image after supplemental lighting, and output the final image.

[0007] Preferably, in the multispectral sensor array: the visible light sensor is of CMOS type, with a wavelength range of 400-760nm, a resolution of ≥2 million pixels, a frame rate of ≥30fps, and a dynamic range of ≥85dB; The near-infrared sensor has a wavelength of 940nm, uses low-noise CMOS material, has a frame rate of ≥30fps, and a detection distance of 0.5-10m; The thermal infrared sensor has a wavelength range of 8-14μm, is uncooled, and has a temperature resolution of ≤0.05℃.

[0008] Preferably, the quality indicators of the spectral quality assessment module include the signal-to-noise ratio (SNR) of each spectrum, the scattering coefficient and the target contrast, the scene type label includes at least foggy day, rainy day, snowy day, extreme low light (≤1 lux) and normal scene, and the processing delay is ≤20ms.

[0009] Preferably, the cross-spectral feature fusion unit adopts a dynamic weight fusion network with an attention mechanism, and the fusion weight is dynamically calculated by two factors: scene type and spectral quality.

[0010] Preferably, the customized multi-task deep learning model includes: An improved MobileNetV3-Small feature extraction backbone network was developed, removing redundant SE modules and adding a cross-spectral feature adaptation layer. Three parallel task heads are used for supplementary lighting parameter prediction, spectral adaptation weight prediction, and power consumption status feedback, respectively. The scene-task attention guidance module is used to dynamically adjust the weight of each task header based on the scene type label. The model is optimized using a combination of hybrid precision quantization, structured pruning, knowledge distillation, and sparse training, resulting in a parameter size of ≤2 million and an inference latency of ≤10ms.

[0011] Preferably, the light source driving module includes a MEMS adjustable lens group and an LED driving chip, supports horizontal ±30° and vertical ±15° angle adjustment, single-zone brightness adjustment range of 5-50mW, total power consumption ≤500mW, and response delay ≤100ms.

[0012] Preferably, the multi-frame fusion of the image optimization module uses 3 images with different exposures and removes moving pixels based on optical flow; noise suppression uses the BM3D algorithm, which improves the noise reduction intensity in dark areas by 30%; the dynamic range of the image after local HDR fusion is ≥120dB and the artifact suppression rate is ≥85%.

[0013] Preferably, the multispectral sensor array further includes a spectral calibration unit with a built-in standard light source calibration module, which automatically calibrates every 100 hours with a spectral response error ≤2%.

[0014] Preferably, the intelligent control unit also includes a closed-loop verification mechanism, which re-evaluates the spectral quality and target area brightness deviation every 3-5 frames of images. If the deviation is ≥10% or the contrast improvement is less than 15%, the parameter adjustment process is re-triggered.

[0015] Preferably, the spectral adaptation mode of the light source driving module satisfies the following: the near-infrared supplementary light intensity is negatively correlated with the near-infrared sensor SNR, visible light supplementary light is turned off in thermal infrared scenes, and only directional infrared supplementary light is turned on.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. Significantly improved environmental adaptability: Through the closed-loop linkage of three-spectrum fusion perception and light source adjustment, it breaks through the perception limitations of single spectrum in severe weather conditions such as fog, rain, snow, and extreme low light (≤1 lux), and the target recognition accuracy is ≥90%, which is more than 180% higher than the traditional visible light solution; 2. Higher accuracy in intelligent decision-making: The customized multi-task model integrates three decision-making processes: supplementary lighting parameters, spectral adaptation, and power consumption control. Combined with a scene-task attention mechanism, it achieves precise matching of "scene type - spectral selection - supplementary lighting parameters," with an adjustment accuracy of 94.5%. 3. Outstanding advantages in low power consumption and low latency: The model has undergone end-to-end lightweight optimization, reducing the parameter size by 60.8%, inference latency by 73.3%, and operating power consumption ≤35mW; directional partitioned illumination reduces the proportion of invalid power consumption to below 22%, which is more than 70% lower than omnidirectional illumination; 4. Wider compatibility: Supports low-computing-power chips such as ARM Cortex-M7 (computing power ≤ 0.5 TOPS), and can be integrated into various devices such as automotive, security, smart door locks, drones, and medical endoscopes, taking into account the imaging needs of complex scenarios and embedded deployments; 5. Superior image quality: Local HDR fusion + multi-frame noise reduction collaborative optimization, dynamic range ≥120dB, motion artifact rate ≤5%, dark detail retention rate ≥90%, effectively solving problems such as overexposure, underexposure, and severe noise. Attached Figure Description

[0017] Figure 1 This is a block diagram of a dynamic light source adaptive camera system according to the present invention; Figure 2 This is a flowchart illustrating the operation of a dynamic light source adaptive camera system according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figures 1 to 2 The present invention provides a technical solution: Multispectral sensor array: Visible light sensor: CMOS type, 400-760nm wavelength, resolution ≥2 million pixels, frame rate ≥30fps, dynamic range ≥85dB, supports MIPICSI-2 interface; Near-infrared sensor: 940nm wavelength, low-noise CMOS, frame rate ≥30fps, detection distance 0.5-10m; Thermal infrared sensor: 8-14μm wavelength, uncooled, temperature resolution ≤0.05℃; Spectral calibration unit: Built-in standard light source calibration module, automatically calibrated every 100 hours, spectral response error ≤2%; Spectral quality assessment module: Detection indicators: SNR of each spectrum, scattering coefficient, target contrast; Evaluation algorithm: Support Vector Machine (SVM) spectral validity classifier; Output: Scene type label (fog / rain / snow / extreme low light / normal), spectral priority determination; Processing delay ≤ 20ms.

[0020] Cross-spectral feature fusion unit: Fusion Algorithm: Attention Mechanism Dynamic Weight Fusion Network; Input: Trispectral image features, spectral quality assessment score, target region coordinates; Output: Unified "lighting-target" fused feature map; Processing delay ≤ 20ms.

[0021] Scene detection module: Inspection content: brightness histogram (0-255 gray levels), highlight / shadow ratio, contrast, type of lighting defect, and coordinates of the target area; Core algorithm: YOLOv8 lightweight object detection model; processing latency ≤50ms.

[0022] Intelligent control unit: Model architecture: A customized multi-task deep learning model, including an improved MobileNetV3-Small backbone network, three parallel task heads (lighting parameter prediction / spectral adaptation weight prediction / power consumption state feedback), and a scene-task attention guidance module; Model optimization: Mixed precision quantization (INT8 + key layer FP16), structured pruning (gradient sensitivity threshold 0.01), knowledge distillation (12 million teacher model parameters), sparse training (L1 regularization λ=0.001). Performance metrics: Parameter size ≤ 2 million, inference latency ≤ 10ms, tuning accuracy ≥ 94.5%; Input: fused feature map, hardware power consumption data; Output: Complementary light brightness (0-1000lm), angle (horizontal ±30° / vertical ±15°), spectral adaptation weight, and upper limit of supplementary light power.

[0023] Light source driver module: Supplemental lighting source: 4×4 zone LED array (16 zones); Driver chip: TITPS61165; Angle adjustment: MEMS adjustable lens group, response time ≤100ms; Dimming method: PWM dimming (frequency ≥ 20kHz), single zone power 5-50mW, total power consumption ≤ 500mW; Spectral adaptation mode: Supports switching between visible light and near-infrared supplementary light, and the intensity of near-infrared supplementary light is negatively correlated with the SNR of the near-infrared sensor.

[0024] Image optimization module: Multi-frame fusion: 3 frames of images with different exposures (1 / 30s, 1 / 100s, 1 / 1000s), optical flow method to remove moving pixels; Noise suppression: BM3D algorithm improves noise reduction intensity in dark areas by 30%; Local HDR fusion: Dynamic range extended to ≥120dB, artifact suppression rate ≥85%; Processing delay ≤ 20ms.

[0025] Workflow Step 1: The multispectral sensor array synchronously acquires raw images of visible light, near-infrared, and thermal infrared spectra, and transmits them to the spectral quality assessment module through the MIPI interface, with a transmission delay of ≤15ms; Step 2: The spectral quality assessment module calculates the SNR, scattering coefficient and other indicators of each spectrum, and outputs the scene type label and spectral priority determination results, with a processing delay of ≤20ms; Step 3: The cross-spectral feature fusion unit calls the attention fusion network to dynamically allocate weights based on scene type and spectral quality, generating a fused feature map with a processing latency of ≤20ms; Step 4: The scene detection module extracts brightness distribution, illumination defect type and target area coordinates based on the fused feature map, with a processing latency of ≤50ms; Step 5: The intelligent control unit calls the customized multi-task model, combines the fused feature map and hardware power consumption data, and outputs the supplementary lighting parameters, spectral adaptation weights and supplementary lighting power limit, with a decision delay of ≤10ms; Step 6: The light source driving module receives the control signal and adjusts the LED array brightness, MEMS lens angle and spectral adaptation mode, with a response delay of ≤100ms; Step 7: The image optimization module performs multi-frame fusion, noise suppression, and local HDR processing on the fused image after supplemental lighting, and outputs the final image with an optimization latency of ≤20ms; Step 8: Closed-loop verification: Repeat steps 2-7 every 3-5 frames. If the brightness deviation of the target area is ≥10% or the contrast improvement is less than 15%, adjust the parameters in real time.

[0026] Example 1: Application of vehicle-mounted cameras in foggy weather Application scenario: Foggy weather (visibility ≤ 200m, ambient light intensity 5 lux, near-infrared SNR = 18dB, visible light SNR = 8dB), pedestrians are crossing the road 50m ahead; System operation process: The multispectral sensor array synchronously acquires three-spectral images. The spectral quality assessment module outputs the scene label "foggy day" and determines that near-infrared is the optimal channel (weight 0.85) and visible light is the auxiliary channel (weight 0.15). The cross-spectral feature fusion unit enhances pedestrian contour features in near-infrared images through an attention mechanism, suppresses visible light fog scattering noise, and generates a fused feature map (the contrast of the pedestrian region is increased from 8 to 35). The intelligent control unit outputs the following supplementary lighting parameters: visible light supplementary lighting is turned off in the front and middle sections, near-infrared supplementary lighting intensity is 1000lm, the angle is focused on the pedestrian area (horizontal ±5°, vertical ±3°), and the upper limit of supplementary lighting power is 250mW. The light source driving module adjusts the angle of the MEMS lens to output near-infrared supplementary light in a directional manner. The image optimization module performs multi-frame fusion and BM3D noise reduction on near-infrared + auxiliary visible light images; Implementation results: Pedestrian recognition accuracy reached 91% (compared to 32% in the original visible light solution), image contrast was improved by 337.5%, the proportion of ineffective power consumption of supplementary lighting was 22%, which is 70.7% lower than that of omnidirectional supplementary lighting, and the overexposure suppression rate in strong light areas was ≥90%.

[0027] Example 2: Extreme Low-Light Security Monitoring Application Application scenario: Warehouse at night with no light source (light intensity 0.5 lux, thermal infrared sensor detects personnel movement, body temperature 36.5℃); System operation process: The spectral quality assessment module determined thermal infrared as the optimal channel (weight 0.95) and near-infrared as the auxiliary channel (weight 0.05), with the scene label "extreme low light". The cross-spectral feature fusion unit extracts personnel contour and temperature features, while the scene detection module locates personnel movement trajectories. The intelligent control unit outputs the following supplementary lighting parameters: directional infrared supplementary lighting brightness of 800lm, angle following the movement of the person, and supplementary lighting power limit of 200mW; The image optimization module performs multi-frame fusion noise reduction based on thermal infrared + infrared supplementary light images; Implementation results: The retention rate of personnel details is ≥93%, the noise ratio is reduced from 40% in the original solution to 6%, the power consumption of supplementary lighting is reduced by 68% compared with omnidirectional supplementary lighting, and the facial features and movement trajectories of personnel can be clearly identified.

[0028] Example 3: Smart door lock camera (low computing power adaptation) Application scenario: The smart door lock is equipped with an ARM Cortex-M7 chip (0.3 TOPS computing power, power consumption limit of 300mW), and is used in foggy stairwells at night (3 lux of light, 5m of visibility). System operation process: Images are acquired by a multispectral sensor, and the spectral quality assessment module outputs a "foggy day" label with a near-infrared weight of 0.8 and a visible light weight of 0.2. The customized multi-task model completes inference within 8ms, and outputs: near-infrared supplementary light brightness of 600lm, angle focusing on the face area (horizontal ±10°, vertical ±5°), and supplementary light power upper limit of 250mW; The light source driving module controls four LED zones for directional supplemental lighting, and the image optimization module performs local HDR fusion. Implementation results: The face recognition accuracy rate is 95.2%, the model operation power consumption is 32mW, and the total power consumption of the supplementary light is 180mW, all of which are lower than the hardware limit. The system response time is ≤150ms, which meets the real-time recognition requirements of the door lock.

[0029] Example 4: Backlight Security Monitoring Application Application scenario: Outdoor backlight environment (background light intensity ≥ 5000 lux, target area light intensity ≤ 50 lux), target person's movement speed ≤ 1.5 m / s; System operation process: The multispectral sensor synchronously acquires images, and the spectral quality assessment module determines that visible light is the optimal channel (weight 0.7) and near-infrared is the auxiliary channel (weight 0.3), and the scene is labeled "backlight". The cross-spectral feature fusion unit enhances the features of the target region, and the scene detection module locates the bounding box of the target person. The intelligent control unit outputs the following supplementary lighting parameters: 500lm brightness for the target area, angle that follows the target movement, and a maximum supplementary lighting power of 300mW. The image optimization module uses short exposure (1 / 1000s) to capture background details and long exposure (1 / 30s) to capture target details, and the local HDR fusion weights are allocated according to the brightness gradient; Implementation results: The dynamic range of the target area is ≥110dB, the detail retention rate is ≥90%, the area of ​​motion artifacts is ≤5%, and the facial features and clothing details of the target person can be clearly identified.

Claims

1. A dynamic light source adaptive camera system, characterized in that, include: A multispectral sensor array is used to synchronously acquire multispectral raw images of a scene. The multispectral sensor array includes at least a visible light sensor, a near-infrared sensor, and a thermal infrared sensor. The spectral quality assessment module is used to calculate the quality index of each spectral image and output scene type labels and spectral priority determination results. A cross-spectral feature fusion unit is used to dynamically allocate the weights of each spectrum based on an attention mechanism to generate a unified "lighting-target" fused feature map; The scene detection module is used to extract brightness distribution information, illumination defect type and target area coordinates based on the fused feature map. The scene detection module uses the YOLOv8 lightweight model for target recognition. The extracted brightness distribution information includes brightness histogram, highlight area ratio, shadow area ratio and contrast. The detection delay is ≤50ms. The intelligent control unit is used to calculate the light source adjustment parameters, spectral adaptation weights, and upper limit of supplementary lighting power based on a customized multi-task deep learning model, combined with the fused feature map and hardware power consumption data. The loss function of the customized multi-task deep learning model is a weighted joint loss, in which the supplementary lighting parameter loss weight is 0.5, the spectral weight loss is 0.3, and the power consumption loss is 0.

2. The light source driving module is used to adjust the brightness, emission angle and spectral adaptation mode of the supplementary light source according to the light source adjustment parameters. The supplementary light source is a multi-zone LED array that supports zoned directional supplementary light. The LED array of the supplementary light source has at least a 4×4 zone structure, supports independent dimming of a single zone, and adapts to the directional supplementary light requirements in different scenarios. The image optimization module is used to perform multi-frame fusion, noise suppression, and local HDR fusion processing on the multispectral fusion image after supplemental lighting, and output the final image.

2. The dynamic light source adaptive camera system according to claim 1, characterized in that, In the multispectral sensor array: the visible light sensor is of CMOS type, with a wavelength range of 400-760nm, a resolution of ≥2 million pixels, a frame rate of ≥30fps, and a dynamic range of ≥85dB; The near-infrared sensor has a wavelength of 940nm, uses low-noise CMOS material, has a frame rate of ≥30fps, and a detection distance of 0.5-10m; The thermal infrared sensor has a wavelength range of 8-14μm, is uncooled, and has a temperature resolution of ≤0.05℃.

3. The dynamic light source adaptive camera system according to claim 1, characterized in that, The quality indicators of the spectral quality assessment module include the signal-to-noise ratio (SNR) of each spectrum, the scattering coefficient and the target contrast. The scene type label includes at least foggy, rainy, snowy, extreme low light (≤1 lux) and normal scene, and the processing delay is ≤20ms.

4. The dynamic light source adaptive camera system according to claim 1, characterized in that, The cross-spectral feature fusion unit adopts a dynamic weight fusion network with an attention mechanism, and the fusion weight is dynamically calculated by two factors: scene type and spectral quality.

5. The dynamic light source adaptive camera system according to claim 1, characterized in that, The customized multi-task deep learning model includes: An improved MobileNetV3-Small feature extraction backbone network was developed, removing redundant SE modules and adding a cross-spectral feature adaptation layer. Three parallel task heads are used for supplementary lighting parameter prediction, spectral adaptation weight prediction, and power consumption status feedback, respectively. The scene-task attention guidance module is used to dynamically adjust the weight of each task header based on the scene type label. The model is optimized using a combination of hybrid precision quantization, structured pruning, knowledge distillation, and sparse training, resulting in a parameter size of ≤2 million and an inference latency of ≤10ms.

6. The dynamic light source adaptive camera system according to claim 1, characterized in that, The light source driving module includes a MEMS adjustable lens group and an LED driver chip, which supports horizontal ±30° and vertical ±15° angle adjustment, single-zone brightness adjustment range of 5-50mW, total power consumption ≤500mW, and response delay ≤100ms.

7. The dynamic light source adaptive camera system according to claim 1, characterized in that, The image optimization module uses three frames with different exposures for multi-frame fusion and removes moving pixels based on optical flow. Noise suppression uses the BM3D algorithm, which improves noise reduction intensity in dark areas by 30%. The dynamic range of the image after local HDR fusion is ≥120dB and the artifact suppression rate is ≥85%.

8. A dynamic light source adaptive camera system according to claim 1, characterized in that, The multispectral sensor array also includes a spectral calibration unit with a built-in standard light source calibration module, which automatically calibrates every 100 hours with a spectral response error of ≤2%.

9. A dynamic light source adaptive camera system according to claim 1, characterized in that, The intelligent control unit also includes a closed-loop verification mechanism, which re-evaluates the spectral quality and target area brightness deviation every 3-5 frames of images. If the deviation is ≥10% or the contrast improvement is less than 15%, the parameter adjustment process is re-triggered.

10. A dynamic light source adaptive camera system according to claim 1, characterized in that, The spectral adaptation mode of the light source driving module satisfies the following: the near-infrared supplementary light intensity is negatively correlated with the near-infrared sensor SNR; in thermal infrared scenes, visible light supplementary light is turned off, and only directional infrared supplementary light is turned on.

Images