A low-light scene enhancement method and system based on event camera and three-dimensional gaussian scattering

Abstract

This invention discloses a method and system for enhancing low-light scenes based on event cameras and 3D Gaussian scattering. The method includes: explicitly modeling reflectivity and illumination properties for each Gaussian element within a 3D Gaussian scattering framework to achieve scene physical decomposition based on Retinex theory; effectively utilizing the high dynamic range of event streams and the color accuracy of image frames through an event-guided reflectivity smoothness constraint and a signal-to-noise ratio adaptive complementary fusion mechanism; performing degradation modeling on the two modalities separately using a degradation-aware alignment module and aligning them to the radiation field; jointly optimizing the radiation field to restore a high-quality scene representation under normal illumination and supporting new perspective synthesis and real-time rendering. This invention enables high-quality enhancement and new perspective synthesis of low-light scenes within a 3D Gaussian scattering framework.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D scene reconstruction technology, and in particular relates to a method and system for enhancing low-light scenes based on event cameras and 3D Gaussian scattering. Background Technology

[0002] Low-light enhancement is an important research direction in the field of computer vision, and it has a wide range of applications in scenarios such as autonomous driving, security monitoring, and nighttime robot navigation.

[0003] However, under extremely low light conditions, the limited amount of photons captured results in extremely low signal-to-noise ratio and severe loss of detail in images acquired by traditional RGB cameras, making low light enhancement a highly pathological inverse problem.

[0004] (a) Limitations of frame-based low-light enhancement methods.

[0005] Existing frame-based low-light enhancement methods can be divided into two categories: optimization-based and learning-based. Optimization-based methods include unsupervised methods and NeRF / 3DGS-based 3D reconstruction methods, which perform well under moderate lighting conditions but struggle to recover details in extremely low-light scenes. Learning-based methods improve enhancement performance through supervised training, but are often difficult to generalize to unseen degraded scenes due to dataset bias. All of the above methods rely solely on low-light images and struggle to effectively recover lost information when details in dark areas are missing.

[0006] (ii) Limitations of low-light enhancement methods based on event cameras.

[0007] Event cameras possess unique advantages in low-light conditions due to their high dynamic range and precise motion capture capabilities. Existing event-based low-light enhancement methods improve visibility through event-image fusion and temporal modeling, but they generally suffer from problems such as neglecting color fidelity and lacking cross-frame temporal consistency. Existing event fusion methods are mostly data-driven, sensitive to data bias, and have limited generalization ability; some methods rely solely on event signals and fail to fully utilize the complementary information of image modalities.

[0008] (III) Challenges of low-light enhancement within the framework of 3D reconstruction.

[0009] 3D reconstruction methods based on NeRF and 3DGS suppress noise through multi-view consistency constraints and have been introduced into low-light enhancement tasks in recent years. However, these methods still struggle to effectively recover structural details when reliable observations in extremely dark regions are severely lacking. Existing event-based NeRF / 3DGS methods often rely solely on events, require preset background brightness, and suffer from inaccurate radiometric measurements. Methods that introduce image input typically assume that events originate from inter-frame differences, ignoring the degradation differences between real low-light images and events.

[0010] (iv) The difficulty of directly integrating event streams with images.

[0011] Integrating event streams, image frames, and 3D reconstruction frameworks presents several challenges: First, under low-light conditions, event streams are susceptible to leakage noise, shot noise, and latency artifacts, while image frames suffer from severe noise and color cast, making them prone to introducing errors when directly used for multimodal constraints. Second, the information sources of the two modalities may overlap, making it difficult to accurately determine the more reliable modality in each region. Third, there is still a lack of systematic solutions for organically integrating Retinex physical priors, the high dynamic range characteristics of event streams, and multi-view geometric consistency within a unified framework. Summary of the Invention

[0012] To address the problems of weak dark area detail recovery, insufficient utilization of complementary event and image modalities, and difficulty in jointly modeling event noise and image degradation in existing low-light scene enhancement methods, this invention provides a low-light scene enhancement method and system based on event camera and three-dimensional Gaussian scattering, which can achieve high-quality enhancement and new perspective synthesis of low-light scenes within the framework of three-dimensional Gaussian scattering.

[0013] A low-light scene enhancement method based on event camera and 3D Gaussian scattering includes the following steps: (1) Acquire low-light image sequences from multiple perspectives and their synchronized low-light event stream data; (2) Perform camera pose estimation on low-light image sequences and initialize the three-dimensional Gaussian scene representation using a three-dimensional Gaussian scattering framework; (3) Based on Retinex theory, the 3D Gaussian scene representation is explicitly decomposed, and reflectivity and illumination properties are modeled for each Gaussian primitive. A 3D Gaussian radiation field with explicit decoupling of reflectivity and illumination is constructed, and a low-illumination image generation model is established. The reflectivity map of the scene is obtained through Alpha blending rendering. With illumination diagram Reflectance diagram With illumination diagram Element-wise multiplication yields scene radiance. , as a rendering observation of a three-dimensional Gaussian radiation field from the current perspective; (4) The decomposition process is guided by the event frame mixing guidance module, which includes event-guided reflectivity smoothness constraint and signal-to-noise ratio adaptive complementary fusion mechanism; (5) Based on the low-light generation model, the low-light event stream and low-light image are degraded and modeled respectively, and the two modes are aligned to the three-dimensional Gaussian radiation field. (6) Jointly optimize the radiation field to restore the enhanced scene representation under normal lighting.

[0014] In step (1), the low-light event stream data is generated by the event camera, and each event records the pixel-level logarithmic brightness change, and the event set... Represented as: ; in, The total number of events; Indicates the first Individual events; polarity ∈ {-1, +1}, indicating the direction of brightness change; timestamp Indicates the trigger time; These are pixel coordinates; when the logarithmic intensity change at a pixel exceeds a preset contrast threshold... The corresponding event is triggered at the specified time; the cumulative brightness change satisfies: ; in, The value represents the scene brightness of pixel u at time t, and the integral term represents the cumulative intensity change of the event stream.

[0015] In step (3), the low-light image generation model is represented as: ; in, Indicates the exposure time. Represents noise. Each three-dimensional Gaussian element Explicitly store reflectivity properties With lighting properties .

[0016] The reflectivity map of the scene is obtained through alpha blending rendering. With illumination diagram The formula is: ; ; in, For the first The reflectivity attribute corresponding to each Gaussian element; For the first The lighting properties corresponding to each Gaussian element; For the first The transmittance of each high-speed element, This represents the set of all Gaussian elements ordered from near to far along the light rays.

[0017] In step (4), the event-guided reflectivity smoothness constraint is achieved by applying a masking total variation regularization loss to the non-event region. Defined as: ; in, Represents spatial gradient, An edge mask is derived for the event, where H and W are the height and width of the image, respectively, u is the pixel coordinate, and ||·||1 represents the L1 norm. A penalty is applied to the reflectivity spatial gradient in event-free regions to encourage smoothness, while the penalty is suppressed at event-aligned edges to preserve structural information. In the construction... Event trailing suppression processing is applied to the event stream to eliminate delay artifacts caused by limited sensor bandwidth under low light conditions.

[0018] In step (4), the signal-to-noise ratio adaptive complementary fusion mechanism recovers the grayscale image of the scene radiance. As a confidence plot, the corresponding joint photometric consistency loss Defined as: ; in, For adaptive weighted event loss, For image loss, and All studies employed a weighted combination of L1 loss and D-SSIM. Indicates to The application stops gradient operations, and ⊙ represents element-wise multiplication; the signal-to-noise ratio adaptive complementary fusion mechanism prioritizes the event signal in dark areas and the image signal in bright areas.

[0019] In step (5), two multilayer perceptrons are used to perform degradation modeling on the low-light event stream and the low-light image respectively.

[0020] The image degradation process is modeled as follows: ; in For image degradation operators, model the reconstructed scene radiosity. Images observed in low light The complete degradation process includes sensor noise and ISP nonlinear effects; the event degradation model is as follows: ; in The event degradation operator aligns the rendered brightness changes with the brightness changes perceived by the event camera in the linear radiance domain; the two multilayer perceptrons learn their respective degradation maps through end-to-end optimization, effectively reducing the propagation of noise in cross-modal constraints.

[0021] In step (6), during the joint optimization of the radiation field, the total loss function Defined as: ; in, This refers to the joint photometric consistency loss corresponding to the signal-to-noise ratio adaptive complementary fusion mechanism. For brightness loss, Loss to the gray world The masking total variation regularization loss is used to guide the event-guided reflectivity smoothness constraint. , , The weights for each loss item.

[0022] Brightness loss The global mean brightness of the constrained restored radiance is close to the target brightness value. : ; Gray World Loss Minimize the variance of each channel to suppress color cast: ; in, The global mean. For channel variance, This is the stability coefficient.

[0023] A low-light scene enhancement system based on an event camera and 3D Gaussian scattering includes: The data acquisition module is used to collect multi-view low-light image sequences and their synchronous event stream data; The scene initialization module is used to perform camera pose estimation on low-light image sequences and initialize a 3D Gaussian scene representation using a 3D Gaussian scattering framework. The Retinex decomposition module is used to explicitly model the reflectivity and illumination properties of a 3D Gaussian scene representation, constructing a 3D Gaussian radiation field that is explicitly decoupled from reflectivity and illumination, and establishing a low-illumination image generation model; it renders reflectivity maps and illumination maps using alpha blending; reflectivity map With illumination diagram Element-wise multiplication yields scene radiance. , as the rendered observation of the three-dimensional Gaussian radiation field from the current perspective; The event frame hybrid guidance module is used to achieve complementary fusion of event guidance reflectivity smoothness constraints and signal-to-noise ratio adaptiveness; The degradation-aware alignment module is used to perform degradation modeling on low-light images and event streams respectively, and align the two modes to the radiation field. The optimization and rendering module is used to jointly optimize the radiation field, supporting low-light scene enhancement and new perspective compositing.

[0024] Compared with the prior art, the present invention has the following beneficial effects: 1. Based on Retinex theory, this invention explicitly models the reflectivity and illumination properties of 3D Gaussian primitives, achieving physical decoupling of scene representation, effectively suppressing reflectivity-illumination leakage, and exhibiting higher color fidelity and detail recovery capabilities in low-light scene reconstruction.

[0025] 2. This invention effectively utilizes the high temporal resolution and structure awareness of event streams by using event-guided reflectivity smoothness constraints, thereby stabilizing Retinex decomposition without relying on explicit annotations and significantly improving the reconstruction quality of extreme dark areas.

[0026] 3. This invention proposes an adaptive complementary fusion mechanism based on signal-to-noise ratio. It dynamically weights the signal-to-noise ratios of the two modes at each pixel, giving full play to the complementary advantages of events and images in different illumination intensity regions, and achieving the best performance on both synthetic and real datasets.

[0027] 4. This invention introduces a degradation-aware alignment module, which models the degradation mechanism of events and images separately through a multilayer perceptron, effectively improving the robustness of cross-modal constraints and solving the alignment difficulties caused by the degradation difference between the two modes under low light conditions.

[0028] 5. This invention is built on the 3DGS framework and naturally supports real-time rendering. It can reach 83 FPS on a single NVIDIA RTX 3090 GPU, balancing enhanced quality and rendering efficiency, and has good practicality and scalability. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the overall framework of a low-light scene enhancement method based on an event camera and three-dimensional Gaussian scattering, according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the complementary clue analysis of event frames in this invention.

[0032] Figure 3 This is a schematic diagram of the Retinex decomposition analysis and ablation experiment of the present invention.

[0033] Figure 4 This is a visual schematic diagram of the degradation mapping curves of F-MLP and G-MLP in this invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0036] Given a monocular low-light sequence and its synchronous event stream The objective of this invention is to restore the enhanced radiation field under normal illumination. It also supports high-quality new perspective synthesis. Training is conducted in two phases: the first phase uses standard 3DGS to preheat and initialize a rough 3D scene; the second phase introduces event-guided Retinex decomposition to refine the radiation field, ultimately obtaining a high-quality scene representation with explicit decoupling of reflectivity and illumination.

[0037] like Figure 1 As shown, a low-light scene enhancement method based on event camera and 3D Gaussian scattering includes the following steps: S01: Acquire low-light multi-view image sequences and their synchronization event stream data, and perform synchronization calibration.

[0038] In this embodiment, a DAVIS346C camera is used to acquire a sequence of low-light images. and corresponding event stream The exposure time is set to 5 ms or 10 ms, and the image pixel intensity value is usually below 50.

[0039] The event camera asynchronously records pixel-level brightness changes with microsecond-level temporal resolution; each event includes the trigger time. Pixel coordinates and changing polarity When the logarithmic brightness change at a pixel exceeds the contrast threshold The corresponding event is triggered when (set to 0.2).

[0040] In this embodiment, the image sequence can undergo conventional preprocessing operations after input, including but not limited to noise reduction and white balance correction, but the above preprocessing operations do not constitute a limitation on the technical solution of the present invention.

[0041] S02 performs camera pose estimation on low-light image sequences and initializes the 3D scene representation using a 3D Gaussian scattering framework.

[0042] In this embodiment, the COLMAP structure-based motion recovery algorithm is used to process low-light image sequences. Camera pose estimation is performed to obtain the extrinsic and intrinsic parameter matrices for each frame. In the first stage (iterations 0–3000), the scene is initialized based on the above pose using the standard 3DGS framework to obtain a rough 3D scene representation, expressed as a Gaussian set. Describe the scene, where each primitive has basic attributes such as position, covariance, and opacity.

[0043] S03, based on Retinex theory, performs explicit decomposition of the 3D Gaussian scene representation and models reflectivity and illumination properties.

[0044] In this embodiment, each three-dimensional Gaussian element is... Added reflectivity attribute With lighting properties The former is a view-independent constant, while the latter is parameterized using spherical harmonic functions. Reflectance maps are rendered using alpha blending. With illumination diagram : ; Scene radiation We obtain the result by multiplying the two elements one by one: ; in, It is a viewpoint-independent constant that represents the inherent properties of the material; A spherical harmonic function is used for view-dependent parameterization to characterize the interaction between illumination and surface normals. This physics-driven decoupling design effectively compresses the solution space and suppresses the reflectivity-illuminance leakage problem.

[0045] For visualization, the rendered image is obtained through Gamma correction: .

[0046] S04, the decomposition process is guided and optimized through the event frame mixing guidance module.

[0047] This invention utilizes two complementary cues—event streams and image frames—under low light conditions to construct an event frame hybrid guidance strategy, such as... Figure 2 As shown, (a) is a complementary clue analysis diagram of the event frame structure, and (b) is a comparison diagram of the quantization error of the event frame.

[0048] (1) Event-guided reflectivity smoothness constraint.

[0049] Under low-light conditions, regions with no events within a short time window typically correspond to regions with smooth reflectivity, because events only occur when the logarithmic change in brightness exceeds a threshold. Triggered by time. Based on this prior, this invention applies a masking total variation regularization loss to non-event regions. : ; Constructing an event export edge mask First, ETS processing is applied to the event stream to eliminate delay trailing; reflectivity gradients are penalized in event-free regions to encourage smoothness, and penalties are suppressed at event-aligned edges to preserve structure, thereby stabilizing Retinex decomposition under low illumination conditions.

[0050] (2) Signal-to-noise ratio adaptive complementary fusion.

[0051] Events and images exhibit higher signal-to-noise ratios in dark and bright areas, respectively. This invention aims to recover radiometric grayscale images. To achieve complementary fusion of adaptive confidence maps and joint photometric consistency loss. for: ; ; ; in, and Both use a weighted combination of L1 loss and D-SSIM, with weights... Set to 0.8. ⊙ represents element-wise multiplication; in the dark area ( Prioritize the use of event signals In the bright area ( Prioritize the use of image signals This will give full play to the complementary advantages of the two modes.

[0052] like Figure 3 As shown in the figure, (a) shows the comparison of decomposition components, (b) shows the event-assisted decomposition analysis, and (c) shows the ablation results of each module and loss item.

[0053] S05, the degradation modeling of the two modes is performed separately by the degradation perception alignment module and aligned to the radiation field.

[0054] In low-light conditions, event streams and image frames degrade in different ways, and directly using them for cross-modal constraints can lead to noise propagation.

[0055] like Figure 4 As shown, in this embodiment, two lightweight multilayer perceptrons are used to model the degradation mechanisms of events and images, respectively. Image degradation: G-MLP outputs low-light image prediction. Modeling Gaussian-Poisson sensor noise (parameters) , ~ U(0.01, 0.04) and ISP nonlinearity effects and color bias. Event degradation: F-MLP output event prediction. The log-differential sensing process of the aligned event camera is performed within the linear radiance domain.

[0056] Two multilayer perceptrons effectively align two types of degraded low-quality observations to the radiation field through end-to-end learning, significantly reducing noise propagation in cross-modal constraints.

[0057] S06, jointly optimizes the radiation field, restores the enhanced scene representation under normal lighting, and supports new perspective synthesis.

[0058] In this embodiment, the second stage (iterations 3000–30000) minimizes the total loss function. : ; in, Constrain global brightness, target brightness The synthetic dataset was set to 0.55, and the real dataset was set to 0.4. Minimize the variance of each channel to suppress color cast, stability coefficient = 0.5; loss weight set to = 1.0, =0.1, = 0.1.

[0059] Through multiple rounds of iterative optimization, the radiation field gradually converges to a high-quality scene representation under normal lighting. All experiments were conducted on a single NVIDIA RTX 3090 GPU, with each scene requiring approximately one hour of training and supporting real-time rendering at 83 FPS. On the synthetic dataset (LLFF, 8 scenes), this invention achieves state-of-the-art performance in both low-light enhancement and novel perspective synthesis tasks: PSNR of 23.45 dB, SSIM of 0.8312, and LPIPS of 0.0888.

[0060] This invention achieves high-quality enhancement and new perspective compositing in low-light scenes and supports real-time rendering by introducing 3DGS explicit decomposition based on Retinex theory, event-guided reflectivity smoothness constraint and signal-to-noise ratio adaptive complementary fusion, and a degradation-aware alignment module.

[0061] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A low-light scene enhancement method based on event camera and 3D Gaussian scattering, characterized in that, Includes the following steps: (1) Acquire low-light image sequences from multiple perspectives and their synchronized low-light event stream data; (2) Perform camera pose estimation on low-light image sequences and initialize the three-dimensional Gaussian scene representation using a three-dimensional Gaussian scattering framework; (3) Based on Retinex theory, the 3D Gaussian scene representation is explicitly decomposed, and reflectivity and illumination properties are modeled for each Gaussian primitive. A 3D Gaussian radiation field with explicit decoupling of reflectivity and illumination is constructed, and a low-illumination image generation model is established. The reflectivity map of the scene is obtained through Alpha blending rendering. With illumination diagram Reflectance diagram With illumination diagram Element-wise multiplication yields scene radiance. , as a rendering observation of a three-dimensional Gaussian radiation field from the current perspective; (4) The decomposition process is guided by the event frame mixing guidance module, which includes event-guided reflectivity smoothness constraint and signal-to-noise ratio adaptive complementary fusion mechanism; (5) Based on the low-light image generation model, the low-light event stream and low-light image are degraded and modeled respectively, and the two modes are aligned to the three-dimensional Gaussian radiation field; (6) Jointly optimize the radiation field to restore the enhanced scene representation under normal lighting.

2. The low-light scene enhancement method based on event camera and 3D Gaussian scattering according to claim 1, characterized in that, In step (1), the low-light event stream data is generated by the event camera, and each event records the pixel-level logarithmic brightness change, and the event set... Represented as: ； in, The total number of events; Indicates the first Individual events; polarity ∈ {-1, +1}, indicating the direction of brightness change; timestamp Indicates the trigger time; These are pixel coordinates; when the logarithmic intensity change at a pixel exceeds a preset contrast threshold... The corresponding event is triggered at the specified time; the cumulative brightness change satisfies: ； in, The value represents the scene brightness of pixel u at time t, and the integral term represents the cumulative intensity change of the event stream.

3. The low-light scene enhancement method based on event camera and 3D Gaussian scattering according to claim 1, characterized in that, In step (3), the reflectivity map of the scene is obtained through Alpha blending rendering. With illumination diagram The formula is: ； ； in, For the first The reflectivity attribute corresponding to each Gaussian element; For the first The lighting properties corresponding to each Gaussian element; For the first The transmittance of each high-speed element, This represents the set of all Gaussian elements ordered from near to far along the light rays.

4. The low-light scene enhancement method based on event camera and 3D Gaussian scattering according to claim 1, characterized in that, In step (4), the event-guided reflectivity smoothness constraint is achieved by applying a masking total variation regularization loss to the non-event region. Defined as: ； in, Represents spatial gradient, An edge mask is derived for the event, where H and W are the height and width of the image, respectively, u is the pixel coordinate, and ||·||1 represents the L1 norm. A penalty is applied to the reflectivity spatial gradient in event-free regions to encourage smoothness, while the penalty is suppressed at event-aligned edges to preserve structural information. In the construction... Event trailing suppression processing is applied to the event stream to eliminate delay artifacts caused by limited sensor bandwidth under low light conditions.

5. The low-light scene enhancement method based on event camera and 3D Gaussian scattering according to claim 1, characterized in that, In step (4), the signal-to-noise ratio adaptive complementary fusion mechanism recovers the grayscale image of the scene radiance. As a confidence plot, the corresponding joint photometric consistency loss Defined as: ； in, For adaptive weighted event loss, For image loss, and All studies employed a weighted combination of L1 loss and D-SSIM. Indicates to The application stops gradient operations, and ⊙ represents element-wise multiplication; the signal-to-noise ratio adaptive complementary fusion mechanism prioritizes the event signal in dark areas and the image signal in bright areas.

6. The low-light scene enhancement method based on event camera and 3D Gaussian scattering according to claim 1, characterized in that, In step (5), two multilayer perceptrons are used to perform degradation modeling on the low-light event stream and the low-light image respectively.

7. The low-light scene enhancement method based on event camera and 3D Gaussian scattering according to claim 1, characterized in that, In step (6), during the joint optimization of the radiation field, the total loss function Defined as: ； in, This refers to the joint photometric consistency loss corresponding to the signal-to-noise ratio adaptive complementary fusion mechanism. To minimize brightness loss, the global mean brightness of the recovered radiance is constrained to be close to the target brightness value; To minimize the variance of each channel as gray world loss, color bias is suppressed. The masking total variation regularization loss is used to guide the event-guided reflectivity smoothness constraint. , , The weights for each loss item.

8. A low-light scene enhancement system based on an event camera and three-dimensional Gaussian scattering, characterized in that, include: The data acquisition module is used to collect multi-view low-light image sequences and their synchronous event stream data; The scene initialization module is used to perform camera pose estimation on low-light image sequences and initialize a 3D Gaussian scene representation using a 3D Gaussian scattering framework. The Retinex decomposition module is used to explicitly model the reflectivity and illumination properties of a 3D Gaussian scene representation, constructing a 3D Gaussian radiation field that is explicitly decoupled from reflectivity and illumination, and establishing a low-illumination image generation model; it renders reflectivity maps and illumination maps using alpha blending; reflectivity map With illumination diagram Element-wise multiplication yields scene radiance. , as a rendering observation of a three-dimensional Gaussian radiation field from the current perspective; The event frame hybrid guidance module is used to achieve complementary fusion of event guidance reflectivity smoothness constraints and signal-to-noise ratio adaptiveness; The degradation-aware alignment module is used to perform degradation modeling on low-light images and event streams respectively, and align the two modes to the radiation field. The optimization and rendering module is used to jointly optimize the radiation field, supporting low-light scene enhancement and new perspective compositing.

Images