Event stream guided low-light image deblurring method

Abstract

This invention proposes an event-stream-guided low-light image deblurring method, comprising: preparing a dataset containing event stream data, low-light blurred image data, and corresponding clear images; preprocessing the event data into dense spatiotemporal voxel form; constructing an end-to-end low-light blur enhancement network with dual branches for events and images, the core of which is to construct an event denoising module, which learns pixel-level event reliability weights by utilizing prior features of image frame-to-frame feature changes and spatiotemporal consistency features of events, and selectively suppresses noisy events to obtain denoised event features; constructing a spatiotemporal trajectory parsing module, which analyzes the spatiotemporal diffusion phenomenon caused by longer exposure time and non-steady motion of objects under low light conditions based on the statistical distribution characteristics of events in the time dimension, and transforms the originally blurred trail into effective motion cues for enhanced image restoration by decoupling the steady-state structure and dynamic changes in the trajectory, thereby achieving in-depth mining of potential spatiotemporal patterns; and performing cross-modal fusion of the event features and image features after trajectory parsing and feature reconstruction, and outputting the enhanced image through decoding and reconstruction. This invention aims to achieve in-depth mining and collaborative utilization of the implicit spatiotemporal patterns in event streams under conditions of significant low-light noise and complex motion trajectories, thereby improving the robustness and reconstruction quality of event-guided low-light deblurring and making it applicable to low-light imaging scenarios such as nighttime surveillance, robot vision, and autonomous driving.

Description

Technical Field

[0001] This invention relates to the fields of event cameras, artificial intelligence computer vision, and autonomous driving technology, and specifically to an event stream-guided method for deblurring low-light images. Background Technology

[0002] In low-light scenarios such as nighttime surveillance, robot vision, and autonomous driving, imaging systems often face significant signal-to-noise ratio degradation and motion blur. Traditional frame cameras, limited by fixed sampling frequencies and finite dynamic ranges, typically require extended exposure times to improve brightness in low-light conditions, making them more prone to noticeable motion blur and detail loss when the target or camera moves. Simultaneously, low-light noise and underexposure further weaken texture and edge information, significantly reducing the robustness of deblurring and subsequent visual tasks. To overcome the perception limitations of frame cameras in moving scenes and low-light variation scenarios, event cameras have gradually gained attention as a novel visual sensor. Event cameras asynchronously record pixel brightness changes with microsecond-level temporal resolution, offering advantages such as high dynamic range and low latency, providing richer spatiotemporal dynamic cues in complex scenes where traditional cameras are prone to failure.

[0003] However, existing event-based deblurring methods still face significant challenges in low-light scenes. On the one hand, low light leads to increased random fluctuations near the event trigger threshold, resulting in a large number of noisy events and background activity events in the event stream. This makes the motion indicators contained in the event data unreliable, and directly using events as motion cues can lead to bias. On the other hand, low-light deblurring is often accompanied by longer exposure / integration time windows. The cumulative distribution of events in the time dimension is no longer an ideal instantaneous response, but rather forms highly diffuse, non-steady-state spatiotemporal trajectories due to the coupling and superposition of real physical motion and sensor photoelectric hysteresis. This complex trajectory interweaving and aliasing introduces erroneous motion cues during reconstruction, causing problems such as edge ghosting and texture distortion. Although some methods adopt the idea of ​​accumulating events into frames or voxels and fusing them with image features, they often focus more on the fusion structure itself and lack a targeted modeling mechanism for decoupling event noise and non-steady-state motion trajectories under low-light conditions, thus limiting the stability and reconstruction quality in low-light nighttime scenes.

[0004] Furthermore, existing research has explored the modeling and utilization of event data from different perspectives, but it still falls short of covering the more challenging deblurring scenario of "low illumination + blur". For example, Seeing Motion at Nighttime with an Event Camera (H. Liu, et al., CVPR, 2024) is mainly based on the single assumption of sensor photoelectric response hysteresis, attempting to alleviate nighttime diffusion and improve structural clarity by forcibly realigning event timestamps. However, this hard correction strategy often disrupts the spatiotemporal physical continuity of real motion when dealing with nonlinear accelerated motion or complex deformation of objects, and fails to fully exploit the prior motion distribution contained in the divergent trajectory. Towards RobustEvent-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach (G. Liang, et al., CVPR, 2024) proposed EvLight for low-light image enhancement, but its adaptability to deblurring scenes with both low light and motion blur is limited. CMTA: Cross-Modal Temporal Alignment for Event-guided Video Deblurring (T. Kim, et al., 2024) uses cross-modal temporal alignment and attention interaction to enhance the connection between events and images, but attention modeling brings high computational overhead.

[0005] Therefore, this invention targets low-light scenarios with significant blur degradation. Under the end-to-end low-light blur enhancement framework, it simultaneously introduces event reliability denoising and spatiotemporal trajectory analysis mechanisms to achieve synergistic optimization of noise event suppression and decoupling of static and dynamic features of non-steady motion trajectories, thereby improving the stability of deblurring and reconstruction quality. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an event-driven low-light image deblurring method. On one hand, an end-to-end low-light blur enhancement network framework with dual branches for events and images is used to achieve collaborative modeling of steady-state texture and spatiotemporal dynamic information of events in low-light blurred scenes. On the other hand, by collaboratively introducing an event denoising module and a spatiotemporal trajectory analysis module, under extreme conditions of significant low-light noise events and prominent long-exposure non-steady-state motion diffusion, not only is the event information source selectively purified, but statistical analysis and decoupling of complex non-steady-state motion trajectories are also achieved. This mechanism can effectively suppress random noise and trajectory aliasing artifacts at the physical level, while maximizing the reverse inference and preservation of real physical motion cues, thereby significantly improving the stability and reconstruction quality of deblurring. This invention helps improve the clarity of imaging and the robustness of visual perception in low-light imaging scenarios such as nighttime surveillance, robot vision, and autonomous driving.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] This invention is an event-stream-guided method for deblurring low-light images, comprising the following steps:

[0009] Step 1: Dataset Preparation. Prepare a dataset containing event stream data, low-light blurred image data, and corresponding sharp images.

[0010] Step 2: Preprocess the raw event data and represent it as dense spatiotemporal voxels. .

[0011] Step 3: Design an event-guided end-to-end low-light blur enhancement network, including an event denoising module for noise event suppression, a spatiotemporal motion trajectory parsing module, and a cross-modal fusion module.

[0012] Step 4: Divide the data into training, validation, and test sets; use the training set to train the model, continuously adjusting the model's parameters through backpropagation to minimize the loss function; during training, use the validation set to monitor the model's performance.

[0013] Step 5: Use the trained optimal model to test low-light blurred images and events over a period of time to generate enhanced images.

[0014] In step 1, the dataset was acquired using three cameras: two high-resolution RGB cameras and one event camera. One RGB camera used images taken under normal lighting conditions with a 2ms exposure time as ground truth labels; the other RGB camera and the event camera were combined with a beam splitter to simulate low-light scenes, and their exposure times were set to 16ms to obtain blurred images. The original event data format is... ,in For pixel position, For the first The timestamp of each event For the first The polarity of an event.

[0015] Step 2, which involves preprocessing the raw event data into spatiotemporal voxels, is a given time window. Divide it into B sub-time windows and embed the events as tensors using the following method. :

[0016] ,

[0017]

[0018]

[0019] in, The time window corresponding to the current dense spatiotemporal voxels. and These are the timestamps corresponding to the start and end times of the time window, respectively. For the first The timestamp of the event trigger The number of sub-time windows obtained by dividing the time window along the time dimension. For the generated spacetime voxels in the first A time channel, a spatial location voxel values ​​at the location, For the index of the time channel, and These are the x and y coordinates of the spatial location, respectively. For normalized timestamps, For the first The polarity of an event, For the first The pixel coordinates of each event. It is a one-dimensional linear interpolation kernel function. , .

[0020] The low-light blur enhancement network in step 3 consists of two stages. The first stage includes the event denoising module and spatiotemporal trajectory parsing module proposed in this invention. The second stage extends the alignment module and feature enhancement module in ELEDNet.

[0021] First, the shallow features of the image are analyzed separately. and shallow features of events Apply convolutional projection to the latent space to obtain image features. and event characteristics The event denoising module denoises events by analyzing event features. Applying 3D depthwise separable convolutions to model the temporal and spatial coherence of events, then... Convolutional processing performs channel blending to obtain event context features. This branch relies solely on the spatiotemporal consistency of the event itself to extract intrinsic reliability clues about the event modality. pass Convolution output This yields the confidence level of the validity of the event branches.

[0022] Based on image features Constructing inter-frame variation For each moment definition:

[0023]

[0024] Again Applying a 3D depthwise separable convolution consistent with the event branch yields the time-varying prior context. Calculate the difference between the event context and the changing prior context. Related items Among them, the difference term utilizes the spatiotemporal distribution independence of image thermal noise and event background noise to quantify the degree of inconsistency between modes in order to suppress random interference; the correlation term is based on the physical isomorphism between the brightness gradient field and the event flow trajectory to quantify the spatiotemporal coupling strength between modes in order to guide the network to focus on high-confidence motion edges.

[0025]

[0026] in, This is an element-wise product. Then, it is further concatenated along the channel dimension:

[0027]

[0028] Then, after convolution, the cross-modal score is output. .

[0029] Given the dynamic differences in the reliability of information sources under different scenarios, this invention further introduces learnable parameters. and A dynamic weighting mechanism is constructed to enable the model to adaptively adjust its dependence on intrinsic consistency and cross-modal verification based on local texture and lighting conditions, thereby achieving optimal denoising performance across all scenarios.

[0030] The weighted fusion of the two scores is then processed using a Sigmoid algorithm to obtain pixel-level reliability weights.

[0031]

[0032] Using time-extended pixel-level reliability weights The shallow features of the event are weighted pixel by pixel, and then residual connections are used to obtain the denoised event features:

[0033]

[0034] in Parameters for controlling the suppression intensity, for The pixel-level event reliability weight map obtained after time expansion These are denoised event features, which serve as input features for subsequent spatiotemporal trajectory analysis and cross-modal fusion modules.

[0035] To address the diffuse blur trajectory of objects generated under low-light, long-exposure conditions, this invention differs from existing methods, such as NER-Net, which primarily rely on the single assumption of sensor photoelectric response hysteresis. This approach attempts to force all trailing events to a single moment by learning timestamp offsets. However, this rigid strategy has significant limitations when handling nonlinear accelerated motion or complex deformations of objects. Forcibly compressing distributions with spatiotemporal physical continuity often destroys realistic motion information and leads to texture geometric distortion. This invention proposes a soft decoupling paradigm based on spatiotemporal distribution statistics. This paradigm abandons the simple "error correction" perspective and no longer treats trajectory divergence as a single hardware noise. Instead, it considers this non-steady, diffuse trajectory, resulting from the combined effects of photoelectric delay and long-exposure displacement, as a probability density projection of the object's motion state in the time domain. Based on this, a three-level progressive analytical strategy is constructed: First, a high-order temporal moment statistical model is built. By calculating the first and second moments of the event intensity within the time window, the focus and discrete uncertainty of the motion trajectory in the time domain are quantified, thereby generating pixel-level motion trajectory priors. Subsequently, this prior is used to drive the structural feature decoupling unit. Through linear weighted aggregation and Gaussian low-pass filtering constraints, high-frequency non-steady-state noise interference is dynamically suppressed in the feature space, achieving the separation of steady-state structural features and transient motion information. Finally, a spatiotemporal evolution synthesis strategy is adopted, using differential 3D convolution to extract transient motion increments and map them back onto the steady-state structure. This analytical strategy suppresses non-steady-state fuzziness while enabling reverse deduction of clear, continuous, and realistic motion trajectories. It not only avoids artificial artifacts introduced by forced coordinate alignment but also preserves the physical fidelity of texture details and the authenticity of spatiotemporal logic to the greatest extent while ensuring edge sharpness.

[0036] To adapt to the 3D convolution operation in the spatiotemporal trajectory analysis module, the denoised event features... By performing temporal dimension expansion and channel permutation, a standard five-dimensional spatiotemporal tensor input is constructed. :

[0037]

[0038] The high-dimensional hidden layer features of multiple channels are transformed into a scalar field that can be statistically analyzed, and then a learnable feature dimensionality reduction projection operator is used. Event features are mapped to a temporal response intensity space to extract pixel-level response amplitudes for spatiotemporal dispersion distribution calculations. :

[0039]

[0040] To address the non-steady-state spatiotemporal dispersion phenomenon of moving objects in event streams under low-light, long-integral conditions, the spatiotemporal trajectory analysis module abandons the traditional black-box feature extraction paradigm and instead proposes a trajectory analysis and complementary hierarchical representation mechanism based on physical statistical moments. Its aim is to utilize the statistical characteristics of event distribution to explicitly distinguish between steady-state structures and dynamic trajectories; that is, real motion exhibits a high-variance, diffuse distribution on the time axis, while background structures exhibit a low-variance, concentrated distribution.

[0041] First, based on the aforementioned response amplitude, time moment analysis is introduced to parametrically describe the evolution of the event. The zeroth moment, first normalized moment, and second central moment along the time dimension are calculated to quantify the central moment of the event trigger and its diffusion extent, respectively.

[0042]

[0043]

[0044]

[0045] in, The discrete energy within the time window. This refers to the center position of the event flow within the time window. For spatiotemporal discreteness, For the first A slice of response at a time index. To prevent stable terms with a denominator of zero.

[0046] Secondly, considering and The numerical differences between the two metrics can lead to computational imbalance if directly fused. Therefore, this module unifies the two metrics to... The standard evaluation criteria, which will soon Normalization ,in, The size of the continuous input time window, which takes a value of 3, and for... Perform stable compression Calculate the spatiotemporal dispersion score:

[0047]

[0048] The first item describes the lag in the occurrence of the event. The larger the value, the earlier the trigger point of the event within the time window for that pixel, and the more likely it belongs to a non-stationary trajectory left over from a historical moment; the second item This characterizes the morphological dispersion of event distribution; the larger the value, the more significant the spatiotemporal divergence caused by local motion. (Coefficient) As a balancing factor, it is set to 0.5 in the module to suppress... Sensitivity to local high-frequency noise ensures that the final score is dominated by robust temporal location cues. Center of time The normalized value, Spatiotemporal discreteness The value obtained after stable compression.

[0049] Finally, after adaptive scaling and truncation, pixel-level spatiotemporal dispersion scores are output. This is used to guide subsequent feature decoupling:

[0050]

[0051] in, and All of these are learnable parameters.

[0052] based on Generate complementary pixel-level gate weights and :

[0053]

[0054]

[0055] in, This is a decoupling threshold used to adaptively define the boundary between steady-state and non-steady-state characteristics; This is the gating sharpness coefficient. For steady-state structure gating weights, This represents the motion gating weights. By constructing this parameterized mapping, the module can dynamically optimize the decoupling strategy based on the actual noise distribution of the local scene: gating. It can filter out dynamic disturbances with high variance and guide the feature flow to focus on the low-frequency steady-state structure region; while complementary gating Then it is activated synchronously, specifically for extracting and utilizing high-frequency unsteady motion trajectories.

[0056] To address the spatiotemporal sparsity and trajectory fragmentation of event streams under low-light conditions, this module employs a context-aware dynamic propagation mechanism to extract high-fidelity motion evolution features. Specifically, if gating weights are directly used... Multiplying by the feature flow will cause the feature response to drop directly to zero in regions of insignificant motion, thereby disrupting the global coherence of the spatiotemporal context. Therefore, this module introduces a processing strategy based on residual soft modulation:

[0057]

[0058] in, By preserving the bias for context, an information response lower bound is provided for the cross-temporal propagation of feature flows, thus ensuring that the connectivity of the global spatiotemporal topology is maintained even in relatively static background regions. This soft modulation strategy enables the network to utilize... While amplifying the nonsteady motion response, it effectively avoids information blockage in relatively static regions, ultimately outputting frame-by-frame motion features. .

[0059] While extracting non-steady-state motion features, this module constructs an independent steady-state structure feature extraction branch for relatively static low-frequency background regions. Considering that the spatial texture clarity of event features varies significantly at different times within a continuous time window due to changes in illumination and motion blur, directly performing temporal averaging easily introduces motion ghosting and destroys high-frequency details. Therefore, this module introduces an adaptive temporal attention aggregation mechanism. The size of the continuous input time window is... The extracted time-by-time event features are represented as ,in By introducing a learnable temporal distribution control factor, an asymmetric attention weight distribution is constructed to perform a weighted projection on the original event features:

[0060]

[0061]

[0062] in, To be assigned to the Attention weight at any moment This is a learnable decay control factor. This mechanism allows the network to break the symmetry of the temporal dimension, autonomously learning and focusing on the dominant feature slices within the window that are richest in information and have the clearest texture, thereby suppressing noise interference from low-quality moments and compressing multi-frame events into single-frame aggregated features rich in high-fidelity structural information. .

[0063] Considering that in long-exposure imaging, the true steady-state structure typically exhibits low-frequency intrinsic texture without significant spatiotemporal displacement, while motion artifacts and random thermal noise mainly exist in high-frequency forms. Therefore, in order to... This core feature is extracted from the data, and this module sequentially applies a spatial two-dimensional enhancement module containing a channel attention mechanism, as well as a frequency-domain Gaussian low-pass filter to it. By actively filtering out high-frequency interference components, the module can lock onto and output structural features. :

[0064]

[0065] In low-light conditions with motion blur, extracting both clean structural and motion features directly from a highly mixed event stream presents significant optimization challenges. To mitigate the coupling difficulty of representation learning, this module introduces a complementary signal decoupling mechanism of "steady-state-incremental." This mechanism aims to guide the network to actively learn and utilize the physical evolution of the event stream, explicitly decomposing scene features into relatively static steady-state structural terms and time-varying dynamic incremental terms, thereby providing a clearer optimization objective.

[0066] First, a spatial two-dimensional augmentation module incorporating a channel attention mechanism is used to analyze structural features. Extracting steady-state structure terms :

[0067]

[0068] The inherent low-frequency textures in the scene that do not change drastically over time provide stable cues for the network to learn the spatial structure.

[0069] Secondly, in order to capture purely transient changes, Input is fed into a 3D convolutional block initialized with a temporal difference operator, a temporal zero-mean constraint is applied, and the dynamic increment term is extracted. :

[0070]

[0071] in, Representing the expectation along the time dimension, this constraint aims to strip away the static background DC component, ensuring It only responds to edges and high-frequency changes where real motion occurs.

[0072] After explicitly decoupling the structure and dynamic increments, this module first decouples the steady-state structure terms. Broadcasting is performed along the time dimension and linked to a dynamic increment term at each time step. By performing linear superposition in the spatiotemporal domain, transient motion increments are used to compensate for the temporal loss of steady-state features, thus synthesizing a spatiotemporal evolution representation that combines texture structure fidelity with physical motion coherence. :

[0073]

[0074] To achieve deep interaction of information across all scenarios, this module will share steady-state structural features across time periods. As a static prior for spatiotemporal sharing, a full-time time-series mapping is performed on the time axis to obtain a result consistent with the dimension of motion features. Extracted frame-by-frame motion features and spatiotemporal evolution representation The features are cascaded along the channel dimension. This joint feature stream is then fed into a fusion network, where it is aggregated via context to generate dynamically perceptual features. :

[0075]

[0076] Finally, by combining the complementary gating generated earlier, the following is applied: With structural features Adaptive feature splitting and weighted modulation are performed to output the decoupled structural features. With motion characteristics :

[0077]

[0078]

[0079] The spatiotemporal trajectory analysis module completes the analysis of highly coupled mixed event streams into steady-state structures and non-steady-state motion trajectories. By introducing physical quantization priors and dynamic soft modulation mechanisms, the module effectively overcomes the bottleneck of direct optimization of mixed signals under low-light long exposure conditions, thereby improving the high-fidelity extraction capability of real motion trajectories.

[0080] In acquiring and Then, a deep spatiotemporal feature encoding network based on transposed attention is first employed. Cross-modal features are encoded. To fully capture global contextual information across different receptive fields, this process incorporates downsampling operations to construct a multi-scale pyramid feature representation:

[0081]

[0082] in, For shallow image features, These are shallow features of the event. For a feature encoder based on transpose attention, and The first Image pyramid features and event pyramid features at various scales.

[0083] Meanwhile, in order to provide clear physical motion guidance for subsequent alignment tasks, this invention additionally incorporates the non-steady-state motion trajectory features output by the preceding spatiotemporal trajectory analysis module. Incorporate it into the coding system and simultaneously generate a motion-guided feature pyramid of the corresponding scale:

[0084]

[0085] in, For the corresponding level index of the multi-scale pyramid, its value set is set as follows: , For the first Motion pyramid features at various scales. Hierarchy. The larger the value, the lower the spatial resolution of the pyramid feature and the wider the spatiotemporal receptive field. In the interaction constraints at different scale levels, the low-resolution level is responsible for capturing the global large-scale motion trend in the scene and upsampling its predicted spatial offset level by level to the high-resolution level, thereby providing reliable initial motion constraints for fine-grained local spatial alignment, thus constructing a coarse-to-fine cascade correction mechanism across multiple scales.

[0086] After acquiring the pyramid features of the three modalities mentioned above, this invention introduces an event-guided deformable temporal alignment module to perform coarse-to-fine spatial alignment. This module integrates image features... For reference, utilize event characteristics With motion guidance features Learn and utilize the real motion evolution patterns between consecutive frames. Specifically, taking the current moment... For reference frames of adjacent times When aligning features, at any scale Below, the network predicts the network. Joint estimation of spatial sampling offset With modulation mask :

[0087]

[0088] in, This represents the upsampling operator, used to propagate the offset predicted at the coarse-scale level to the fine-scale level for continuous correction. In the first At this scale, the near moment Features aligned to reference time The spatial sampling offset predicted at that time, The modulation mask is used to represent the spatial sampling offset. Subsequently, a deformable alignment operator with offset and mask is employed. Spatially align the features of neighboring frames and output the aligned features:

[0089]

[0090] Ultimately, at the original resolution scale At this point, the module aggregates the features of all aligned neighboring frames and the reference frame, and outputs a spatially aligned image feature stream. .

[0091] After the alignment module, a cross-modal feature enhancement module (SFCM-FE) is used to collaboratively recover the main structure of the scene by utilizing aligned frames and event features under low illumination and low signal-to-noise ratio conditions. The difference from ELEDNet is that this invention further utilizes the structural features output by the spatiotemporal trajectory analysis module. As a structural prior, it is also input to enhance the stability of the main structure recovery.

[0092] The structural features output by the spatiotemporal trajectory analysis module As prior input to the structure, the input features are first encoded using a CNN to construct a multi-scale feature pyramid:

[0093]

[0094] in, In scale The alignment frame features below, In order to scale The following event characteristics, In scale The structural features below, It is a multi-scale feature encoder. , , As input to the feature enhancement module SFCM-FE, the scale is obtained. Enhanced features .

[0095] First, compare with the previous scale. Enhancement results Upsampling:

[0096]

[0097] in It is a 4×4 deconvolution upsampling layer. For the previous scale Enhanced features, for The current scale is obtained after deconvolution upsampling. The features are then concatenated and convolved with the event features, aligned frame features, structural features, and upsampled features to obtain the fused features:

[0098]

[0099] Will The input is a spectral filtering-based feature enhancement module SFCM-FE, which ultimately yields the enhanced multi-scale pyramid features. .

[0100] Will The input pyramid decoding and reconstruction network performs stepwise upsampling and multi-scale fusion to finally generate the enhanced image. .

[0101] The model is trained using backpropagation with gradient descent. Multi-scale supervision is employed for the training loss. The network output includes reconstruction results at three scales: full resolution, 1 / 2 resolution, and 1 / 4 resolution. The pixel reconstruction loss is calculated separately for each scale and then summed with weights. The loss is defined as:

[0102]

[0103] in, and They represent the first The predicted image at each scale and the ground truth image at the corresponding scale are weighted by the following coefficients: , This is the L1 loss.

[0104] Compared with existing methods, the advantages of this invention are:

[0105] This invention proposes an event-driven low-light image deblurring method. It introduces cross-modal reliability verification and a non-steady-state spatiotemporal trajectory decoupling mechanism into an end-to-end low-light blur enhancement network framework, focusing on overcoming two key degradation challenges: significant event random noise and complex motion blur during long exposures under low-light conditions. On one hand, through an event denoising module, pixel-level reliability weights are learned by comprehensively utilizing prior information on temporal variations of image features and intrinsic event information. This achieves selective suppression of complex background clutter and occasional false signals, preventing modal conflicts from interfering with deblurring reconstruction. On the other hand, through a spatiotemporal trajectory parsing module, pixel-level motion trajectory priors are generated based on the high-order statistical distribution of events in the time dimension. Adaptive decoupling and differential reconstruction of steady-state structure and transient motion increments are then performed in the feature space. This mechanism effectively suppresses high-frequency trajectory aliasing artifacts at the physical level while preserving realistic physical dynamic cues, thus significantly improving the structural stability and texture reconstruction quality of the deblurred image. It is highly suitable for challenging low-light imaging scenarios such as nighttime surveillance, robot vision, and autonomous driving.

[0106] Compared to existing enhancement, deblurring, and joint restoration methods, this invention demonstrates superior overall performance in restoration. Experimental results, in quantitative comparisons on the RELED dataset, show that the proposed method achieves a PSNR of 32.07 dB and a SSIM of 0.913. Compared to the baseline method ELEDNet, the PSNR of this invention is improved from 31.13 dB to 32.07 dB, an increase of 0.94 dB, and the SSIM is improved from 0.907 to 0.913. Compared to the representative event-guided deblurring method MAT, the PSNR of this invention is improved by 1.23 dB, and the SSIM by 0.009. These results demonstrate that this invention not only effectively improves reconstruction accuracy but also achieves or surpasses existing state-of-the-art methods in terms of structural similarity, validating the effectiveness of the proposed technical solution in low-light joint enhancement and deblurring tasks.

[0107] The method of this invention has a parameter size of 13.95 MB, which is only a slight increase compared to the baseline ELEDNet's 12.8 MB, but it achieves a significant performance improvement, indicating that the reliability verification mechanism and dynamic-static decoupling mechanism introduced in this invention have high parameter utilization efficiency. This feature enables the deployment of this invention in application scenarios that require real-time performance, stability, and low resource consumption, such as nighttime surveillance, mobile robots, intelligent driving sensing terminals, and edge vision devices.

[0108] The event denoising module and spatiotemporal trajectory parsing module proposed in this invention can both be inserted as functional units in an end-to-end network, and can be combined with existing frame-based, event-based, or frame-event joint recovery networks without changing the basic input-output format of the original backbone network. Therefore, this invention is not only applicable to low-light image deblurring tasks, but can also be further extended to related visual tasks such as low-light video enhancement, event-guided video restoration, and nighttime dynamic target perception, and has strong potential for widespread application. Attached Figure Description

[0109] Figure 1 This is a flowchart of an embodiment of the present invention.

[0110] Figure 2 This is the overall network architecture of an embodiment of the present invention.

[0111] Figure 3 This is the event denoising module in this embodiment of the invention.

[0112] Figure 4 This is the spatiotemporal trajectory analysis module in an embodiment of the present invention. Detailed Implementation

[0113] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0114] like Figure 1 As shown, this embodiment provides an event-stream-guided low-light image deblurring method, including the following steps:

[0115] (1) Prepare the dataset

[0116] Prepare a dataset containing event stream data, low-light blurred image data, and corresponding sharp images. This example uses the public dataset RELED.

[0117] (2) Preprocess raw event data

[0118] The step of preprocessing raw event data into spatiotemporal voxels is to give a time window. Divide it into B sub-time windows and embed the events as tensors using the following method. :

[0119] ,

[0120]

[0121]

[0122] in, The time window corresponding to the current dense spatiotemporal voxels. and These are the timestamps corresponding to the start and end times of the time window, respectively. For the first The timestamp of the event trigger The number of sub-time windows obtained by dividing the time window along the time dimension. For the generated spacetime voxels in the first A time channel, a spatial location voxel values ​​at the location, For the index of the time channel, and These are the x and y coordinates of the spatial location, respectively. For normalized timestamps, For the first The polarity of an event, For the first The pixel coordinates of each event. It is a one-dimensional linear interpolation kernel function. , .

[0123] (3) Model Design

[0124] The low-light blur enhancement network consists of two main parts. The first part includes an event denoising module and a spatiotemporal trajectory parsing module. The second part extends the design of ELEDNet by adding structural and motion information obtained from the first part to the input.

[0125] First, the shallow features of the image are analyzed separately. and shallow features of events Apply convolutional projection to the latent space to obtain image features. and event characteristics The first part, the event denoising module, denoises the event features... Applying 3D depthwise separable convolutions to model the temporal and spatial coherence of events, then... Convolutional processing performs channel blending to obtain event context features. This branch relies solely on the spatiotemporal consistency of the event itself to extract intrinsic reliability clues about the event modality. pass Convolution output This yields the confidence level of the validity of the event branches.

[0126] Based on image features Constructing inter-frame variation For each moment definition:

[0127]

[0128] right Applying a 3D depthwise separable convolution consistent with the event branch yields the time-varying prior context. Calculate the difference between the event context and the changing prior context. Related items Among them, the difference term utilizes the spatiotemporal distribution independence of image thermal noise and event background noise to quantify the degree of inconsistency between modes in order to suppress random interference; the correlation term is based on the physical isomorphism between the brightness gradient field and the event flow trajectory to quantify the spatiotemporal coupling strength between modes in order to guide the network to focus on high-confidence motion edges.

[0129]

[0130] in, This is an element-wise product. Then, it is further concatenated along the channel dimension:

[0131]

[0132] Then, after convolution, the cross-modal score is output. .

[0133] Given the dynamic differences in the reliability of information sources under different scenarios, this invention further introduces learnable parameters. and A dynamic weighting mechanism is constructed to enable the model to adaptively adjust its dependence on intrinsic consistency and cross-modal verification based on local texture and lighting conditions, thereby achieving optimal denoising performance across all scenes. The weighted fusion of the two scores is then processed using a Sigmoid algorithm to obtain pixel-level reliability weights. :

[0134]

[0135] Using time-extended pixel-level reliability weights The shallow features of the event are weighted pixel by pixel, and then residual connections are used to obtain the denoised event features:

[0136]

[0137] in, Parameters for controlling the suppression intensity, for The pixel-level event reliability weight map obtained after time expansion These are denoised event features, which serve as input features for subsequent spatiotemporal trajectory analysis and cross-modal fusion modules.

[0138] The core of the first part, the spatiotemporal trajectory analysis module, lies in establishing a dynamic-static decoupling mechanism for non-steady-state spatiotemporal trajectories based on statistical distribution. Addressing the diffuse blurry trajectories generated by objects under low-light, long-exposure environments, this invention differs from existing methods, such as NER-Net, which primarily rely on the single assumption of sensor photoelectric response hysteresis. This approach attempts to force all trailing events to a single moment by learning timestamp offsets. However, this rigid strategy has significant limitations when handling nonlinear accelerated motion or complex deformations of objects. Forcibly compressing distributions with spatiotemporal physical continuity often destroys realistic motion information and leads to texture geometric distortion. This invention proposes a soft decoupling paradigm based on spatiotemporal distribution statistics. This paradigm abandons the simple "error correction" perspective and no longer treats trajectory divergence as a single hardware noise. Instead, it considers this non-steady, diffuse trajectory, resulting from the combined effects of photoelectric delay and long-exposure displacement, as a probability density projection of the object's motion state in the time domain. Based on this, a three-level progressive analytical strategy is constructed: First, a high-order temporal moment statistical model is built. By calculating the first and second moments of the event intensity within the time window, the focus and discrete uncertainty of the motion trajectory in the time domain are quantified, thereby generating pixel-level motion trajectory priors. Subsequently, this prior is used to drive the structural feature decoupling unit. Through linear weighted aggregation and Gaussian low-pass filtering constraints, high-frequency non-steady-state noise interference is dynamically suppressed in the feature space, achieving the separation of steady-state structural features and transient motion information. Finally, a spatiotemporal evolution synthesis strategy is adopted, using differential 3D convolution to extract transient motion increments and map them back onto the steady-state structure. This analytical strategy suppresses non-steady-state fuzziness while enabling reverse deduction of clear, continuous, and realistic motion trajectories. It not only avoids artificial artifacts introduced by forced coordinate alignment but also preserves the physical fidelity of texture details and the authenticity of spatiotemporal logic to the greatest extent while ensuring edge sharpness.

[0139] To adapt to the 3D convolution operation in the spatiotemporal trajectory analysis module, the denoised event features... By performing temporal dimension expansion and channel permutation, a standard five-dimensional spatiotemporal tensor input is constructed. :

[0140]

[0141] Furthermore, in order to transform the high-dimensional hidden layer features of multiple channels into a scalar field that can be statistically analyzed, a learnable feature dimensionality reduction projection operator is used. Event features are mapped to a temporal response intensity space to extract pixel-level response amplitudes for spatiotemporal dispersion distribution calculations. :

[0142]

[0143] To address the non-steady-state spatiotemporal dispersion phenomenon of moving objects in event streams under low-light, long-integral conditions, the spatiotemporal trajectory analysis module abandons the traditional black-box feature extraction paradigm and instead proposes a trajectory analysis and complementary hierarchical representation mechanism based on physical statistical moments. Its aim is to utilize the statistical characteristics of event distribution to explicitly distinguish between steady-state structures and dynamic trajectories; that is, real motion exhibits a high-variance, diffuse distribution on the time axis, while background structures exhibit a low-variance, concentrated distribution.

[0144] First, based on the aforementioned response amplitude, time moment analysis is introduced to parametrically describe the evolution of the event. The zeroth moment, first normalized moment, and second central moment along the time dimension are calculated to quantify the central moment of the event trigger and its diffusion extent, respectively.

[0145]

[0146]

[0147]

[0148] in, The discrete energy within the time window. This refers to the center position of the event flow within the time window. For spatiotemporal discreteness, For the first A slice of response at a time index. To prevent stable terms with a denominator of zero.

[0149] Secondly, considering and The numerical differences between the two metrics can lead to computational imbalance if directly fused. Therefore, this module unifies the two metrics to... The standard evaluation metric is to normalize the centroid to And perform stable compression of variance. Calculate the spatiotemporal dispersion score:

[0150]

[0151] The first item describes the lag in the occurrence of the event. The larger the value, the earlier the trigger point of the event within the time window for that pixel, and the more likely it belongs to a non-stationary trajectory left over from a historical moment; the second item This characterizes the morphological dispersion of event distribution; the larger the value, the more significant the spatiotemporal divergence caused by local motion. (Coefficient) As a balancing factor, it is set to 0.5 in the module to suppress... Sensitivity to local high-frequency noise ensures that the final score is dominated by robust temporal location cues. Center of time The normalized value, Spatiotemporal discreteness The value obtained after stable compression.

[0152] Finally, after adaptive scaling and truncation, the spatiotemporal dispersion score is output. This is used to guide subsequent feature decoupling:

[0153]

[0154] based on Generate complementary pixel-level gate weights:

[0155]

[0156]

[0157] in, This is a decoupling threshold used to adaptively define the boundary between steady-state and non-steady-state characteristics; This represents the gated sharpness coefficient. By constructing this parameterized mapping, the module can dynamically optimize the decoupling strategy based on the actual noise distribution of the local scene: steady-state structure gate weights. It can filter out dynamic interference with high variance and guide the feature flow to focus on the low-frequency steady-state structure region; while motion-gated weights Then it is activated synchronously, specifically for extracting and utilizing high-frequency unsteady motion trajectories.

[0158] To address the spatiotemporal sparsity and trajectory fragmentation of event streams under low-light conditions, this module employs a context-aware dynamic propagation mechanism to extract high-fidelity motion evolution features. Specifically, if gating weights are directly used... Multiplying by the feature flow will cause the feature response to drop directly to zero in regions of insignificant motion, thereby disrupting the global coherence of the spatiotemporal context. Therefore, this module introduces a processing strategy based on residual soft modulation to obtain frame-by-frame motion features. :

[0159]

[0160] in, As a context-preserving bias, it provides an information response lower bound for the cross-temporal propagation of feature flows, thereby ensuring that the connectivity of the global spatiotemporal topology is maintained even in relatively static background regions. This soft modulation strategy enables the network to utilize... While amplifying the unsteady motion response, it effectively avoids information blockage in relatively static regions.

[0161] While extracting non-steady-state motion features, this module constructs an independent steady-state structure feature extraction branch for relatively static low-frequency background regions. Considering that the spatial texture clarity of event features varies significantly at different times within a continuous time window due to changes in illumination and motion blur, directly performing temporal averaging easily introduces motion ghosting and destroys high-frequency details. Therefore, this module introduces an adaptive temporal attention aggregation mechanism. The size of the continuous input time window is... The extracted time-by-time event features are represented as ,in By introducing a learnable temporal distribution control factor, an asymmetric attention weight distribution is constructed to perform a weighted projection on the original event features:

[0162]

[0163]

[0164] in, To be assigned to the Attention weight at any moment This is a learnable decay control factor. This mechanism allows the network to break the symmetry of the temporal dimension, autonomously learning and focusing on the dominant feature slices within the window that are richest in information and have the clearest texture, thereby suppressing noise interference from low-quality moments and compressing multi-frame events into single-frame aggregated features rich in high-fidelity structural information. .

[0165] Considering that in long-exposure imaging, the true steady-state structure typically exhibits low-frequency intrinsic texture without significant spatiotemporal displacement, while motion artifacts and random thermal noise mainly exist in high-frequency forms. Therefore, in order to... This core feature is extracted from the data, and this module sequentially applies a spatial two-dimensional enhancement module containing a channel attention mechanism, as well as a frequency-domain Gaussian low-pass filter to it. By actively filtering out high-frequency interference components, the module can lock onto and output structural features. :

[0166]

[0167] In low-light conditions with motion blur, extracting both clean structural and motion features directly from a highly mixed event stream presents significant optimization challenges. To mitigate the coupling difficulty of representation learning, this module introduces a complementary signal decoupling mechanism of "steady-state-incremental." This mechanism aims to guide the network to actively learn and utilize the physical evolution of the event stream, explicitly decomposing scene features into relatively static steady-state structural terms and time-varying dynamic incremental terms, thereby providing a clearer optimization objective.

[0168] First, a spatial two-dimensional augmentation module incorporating a channel attention mechanism is used to analyze structural features. Extracting steady-state structure terms :

[0169]

[0170] The inherent low-frequency textures in the scene that do not change drastically over time provide stable cues for the network to learn the spatial structure.

[0171] Secondly, in order to capture purely transient changes, Input is fed into a 3D convolutional block initialized with a temporal difference operator, a temporal zero-mean constraint is applied, and the dynamic increment term is extracted. :

[0172]

[0173] in, Representing the expectation along the time dimension, this constraint aims to strip away the static background DC component, ensuring It only responds to edges and high-frequency changes where real motion occurs.

[0174] After explicitly decoupling the structure and dynamic increments, this module first decouples the steady-state structure terms. Broadcasting is performed along the time dimension and linked to a dynamic increment term at each time step. By performing linear superposition in the spatiotemporal domain, transient motion increments are used to compensate for the temporal loss of steady-state features, thus synthesizing a spatiotemporal evolution representation that combines texture structure fidelity with physical motion coherence. :

[0175]

[0176] To achieve deep interaction of information across all scenarios, this module will share steady-state structural features across time periods. As a static prior for spatiotemporal sharing, a full-time time-series mapping is performed on the time axis to obtain a result consistent with the dimension of motion features. Extracted frame-by-frame motion features and spatiotemporal evolution representation The features are cascaded along the channel dimension. This joint feature stream is then fed into a fusion network, where it is aggregated via context to generate dynamically perceptual features. :

[0177]

[0178] Finally, by combining the complementary gating generated earlier, the following is applied: With structural features Adaptive feature splitting and weighted modulation are performed to output the decoupled structural features. With motion characteristics :

[0179]

[0180]

[0181] The spatiotemporal trajectory analysis module completes the analysis of highly coupled mixed event streams into steady-state structures and non-steady-state motion trajectories. By introducing physical quantization priors and dynamic soft modulation mechanisms, the module effectively overcomes the bottleneck of direct optimization of mixed signals under low-light long exposure conditions, thereby improving the high-fidelity extraction capability of real motion trajectories.

[0182] In video deblurring tasks, temporal alignment is a crucial step in recovering the current frame using information from adjacent frames. Existing alignment methods can be broadly categorized into several types, but they all have limitations in low-light and high-speed motion scenarios. The first type is optical flow-based alignment methods: Early video deblurring methods, such as SPyNet, typically estimated inter-frame optical flow first, then performed a wrapping operation on adjacent frames based on the optical flow field to achieve alignment. However, under low-light conditions, images are noisy and lack texture, and the severe blurring caused by high-speed motion makes it extremely difficult to match the pixel correspondence between adjacent frames. Optical flow estimation often fails in this situation, leading to alignment errors and artifacts in the reconstructed image. The second type is pure image alignment methods based on deformable convolutional networks (DCNs): To address the problem of inaccurate optical flow estimation, methods such as EDVR introduce deformable convolutional networks to implicitly align features by learning offsets. Although DCNs are more robust than optical flow, their computational complexity is extremely high when processing high-resolution features. More importantly, image-driven DCNs struggle to extract accurate geometric structure information at severely blurred edges, leading to a decrease in alignment accuracy in extremely blurred scenes due to a lack of high-frequency guidance. The third category of alignment methods is based on cross-modal attention mechanisms: recent methods such as CMTA attempt to leverage the high temporal resolution of event cameras, proposing cross-modal temporal alignment strategies. These strategies enhance intra- and inter-frame feature interactions through Transformers or complex attention mechanisms, avoiding explicit optical flow computation. While CMTA improves performance by utilizing event information, it introduces complex recursive attention structures and cascaded alignment modules, significantly increasing the number of model parameters and memory usage during inference. Furthermore, these methods typically treat events as reliable auxiliary inputs, often ignoring noise and non-stationary spatiotemporal trajectories inherent in low-light environments. Directly applying these to strong attention interactions may introduce erroneous guidance signals. Given the failure of optical flow methods in low-light blur and the limitations of image-driven DCNs, event data must be introduced to guide alignment. Although methods like CMTA provide high-performance alignment schemes, their complex attention mechanisms may contain significant noise in the event signals themselves. Therefore, in the subsequent reconstruction stage, this invention selects the Event-Guided Deformable Temporal Alignment Module (EDTFA) proposed in ELEDNet. This is because EDTFA employs an efficient "coarse-to-fine" pyramid alignment strategy. Compared to complex global attention mechanisms, its deformable convolution-based structure is easier to converge, and its computational cost is relatively controllable. Robust event guidance: EDTFA explicitly uses event features as conditional inputs for offset prediction, enabling it to directly utilize high-frequency edge information of events to correct alignment deviations in blurred frames, making it very suitable for the scenario described in this invention. The core innovation of this invention lies in the front-end "event reliability denoising" and "spatiotemporal trajectory parsing".Using EDTFA, a proven and standard alignment module, as a baseline component, a unified and reproducible reconstruction platform can be provided. This ensures that the final performance improvement is definitively derived from the cleansing of event data and spatiotemporal trajectory analysis of this invention, thus more objectively demonstrating the effectiveness of the front-end processing module proposed in this invention.

[0183] The second part mainly includes: in acquiring and Then, a deep spatiotemporal feature encoding network based on transposed attention is first employed. Cross-modal features are encoded. To fully capture global contextual information across different receptive fields, this process incorporates downsampling operations to construct a multi-scale pyramid feature representation:

[0184]

[0185] in, For shallow image features, These are shallow features of the event. For a feature encoder based on transpose attention, and The first Image pyramid features and event pyramid features at various scales.

[0186] Meanwhile, in order to provide clear physical motion guidance for subsequent alignment tasks, this invention additionally incorporates the non-steady-state motion trajectory features output by the preceding spatiotemporal trajectory analysis module. Incorporate it into the coding system and simultaneously generate a motion-guided feature pyramid of the corresponding scale:

[0187]

[0188] in, For the corresponding level index of the multi-scale pyramid, its value set is set as follows: , For the first Motion pyramid features at various scales. Hierarchy. The larger the value, the lower the spatial resolution of the pyramid feature and the wider the spatiotemporal receptive field. In the interaction constraints at different scale levels, the low-resolution level is responsible for capturing the global large-scale motion trend in the scene and upsampling its predicted spatial offset level by level to the high-resolution level, thereby providing reliable initial motion constraints for fine-grained local spatial alignment, thus constructing a coarse-to-fine cascade correction mechanism across multiple scales.

[0189] After acquiring the pyramid features of the three modalities mentioned above, this invention introduces an event-guided deformable temporal alignment module to perform coarse-to-fine spatial alignment. This module integrates image features... For reference, utilize event characteristics With motion guidance features Learn and utilize the real motion evolution patterns between consecutive frames. Specifically, taking the current moment... For reference frames of adjacent times When aligning features, at any scale Below, the network predicts the network. Joint estimation of spatial sampling offset With modulation mask :

[0190]

[0191] in, This represents the upsampling operator, used to propagate the offset predicted at the coarse-scale level to the fine-scale level for continuous correction. In the first At this scale, the near moment Features aligned to reference time The spatial sampling offset predicted at that time, The modulation mask is used to represent the spatial sampling offset. Subsequently, a deformable alignment operator with offset and mask is employed. Spatially align the features of neighboring frames and output the aligned features:

[0192]

[0193] By progressively propagating and correcting the offset across multiple scales, this mechanism effectively alleviates the challenges of dense correspondence caused by low light noise and long exposures. Ultimately, at the original resolution scale... At this point, the module aggregates the features of all aligned neighboring frames and reference frames, and outputs the highest-precision spatially aligned image feature stream. .

[0194] After the alignment module EDTFA, a cross-modal feature enhancement module SFCM-FE is used to collaboratively recover the main structure of the scene by utilizing aligned frames and event features under low illumination and low signal-to-noise ratio conditions. The difference from ELEDNet is that this invention further incorporates the structural features output by the spatiotemporal trajectory analysis module. As a structural prior, it is also input to enhance the stability of the main structure recovery.

[0195] SFCM-FE first performs CNN encoding on the input features to construct a multi-scale feature pyramid:

[0196]

[0197] in, In scale The alignment frame features below, In scale The following event characteristics, In scale The structural features below, It is a multi-scale feature encoder. , , As input to the feature enhancement module SFCM-FE, the scale is obtained. Enhanced features .

[0198] First, compare with the previous scale. Enhancement results Upsampling:

[0199]

[0200] in It is a 4×4 deconvolution upsampling layer. For the previous scale Enhanced features, for The current scale is obtained after deconvolution upsampling. The features are then concatenated and convolved with the event features, aligned frame features, structural features, and upsampled features to obtain the fused features:

[0201]

[0202] Will The input is a spectral filtering-based feature enhancement module SFCM-FE, which ultimately yields the enhanced multi-scale pyramid features. .

[0203] Will The input pyramid decoding and reconstruction network performs stepwise upsampling and multi-scale fusion to finally generate the enhanced image. .

[0204] (3) Model training

[0205] Backpropagation training was performed using gradient descent, with the Adam optimizer used. The training data was randomly pruned to 256×256 for data augmentation. The backbone learning rate was set to... Cosine annealing is used for attenuation, and the minimum learning rate is set to [value missing]. Furthermore, gradients are pruned after each backpropagation to stabilize training, with a pruning threshold set to 5.0. The denoising module parameter set uses a higher initial learning rate. To ensure faster convergence in the early and mid-stages of training, while maintaining effective updates to event reliability weights in the later stages, multi-scale supervision is employed in the training loss. The network output includes reconstruction results at three scales: full resolution, 1 / 2 resolution, and 1 / 4 resolution. The pixel reconstruction loss is calculated separately for each scale and then weighted and summed. The loss is defined as:

[0206]

[0207] in, and The first The predicted image at each scale and the ground truth image at the corresponding scale are weighted by the following coefficients: , This is the L1 loss.

[0208] Experimental test:

[0209] To verify the effectiveness of the method of this invention, it was compared with several existing state-of-the-art methods on the RELED dataset. These low-light enhancement methods include SNRNet, LLFormer, RetinexFormer, SDSDNet, and EvLight++. These methods primarily focus on brightness recovery, noise suppression, and visibility enhancement under low-light conditions, and can improve image quality in low-light scenes to some extent. However, their recovery capabilities remain limited for complex blur degradation caused by long exposures and target motion.

[0210] Deblurring methods include MPRNet, MIMOUNet+, NAFNet, RNN-MBP, DSTNet, e-SLNet, REDNet, EFNet, MAT, UEVD, REFID, and GEM. These methods primarily target motion blur recovery. Some methods rely solely on image frame information, while others combine image frame and event information for assisted reconstruction. Compared to simple enhancement methods, these methods focus more on blur trajectory modeling and detail recovery; however, they are still susceptible to noise interference, texture loss, and event diffusion artifacts in low-light scenes.

[0211] Joint methods include LEDNet. This method attempts to simultaneously handle low-light degradation and blur degradation within a unified framework, but its ability to model fast-moving details in complex dynamic low-light environments remains limited due to its primary reliance on information from the image frames themselves. The ELEDNet method, by introducing event information, provides additional temporal cues for the joint low-light enhancement and deblurring recovery, thus generally exhibiting better recovery potential in complex dynamic scenes. However, event streams under low-light conditions are prone to random noise, spurious responses, and spatiotemporal diffusion, which places higher demands on the effective utilization of event information. This invention, based on these limitations, further improves the recovery performance and structural stability in the joint low-light enhancement and deblurring task by introducing cross-modal reliability verification and spatiotemporal trajectory analysis.

[0212] Table 1 Evaluation on the RELED dataset

[0213]

[0214]

[0215] Table 3 Ablation Experiment of Spatiotemporal Trajectory Analysis Module

[0216]

[0217]

[0218] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. An event-stream-guided method for deblurring low-light images, characterized in that, The method includes the following steps: Step 1: Dataset preparation. Prepare a dataset containing event stream data, low-light blurred image data, and corresponding sharp images. Step 2: Preprocess the raw event data and represent it as dense spatiotemporal voxels. , Step 3: Design an event-guided end-to-end low-light blur enhancement network, including an event denoising module for noise event suppression, a spatiotemporal trajectory resolution module, and a cross-modal fusion module. The end-to-end low-light blur enhancement network in step 3 consists of two stages. The first stage includes an event denoising module and a spatiotemporal trajectory parsing module. The second stage extends the alignment module and feature enhancement module in ELEDNet. Step 4: Divide the data into training, validation, and test sets; train the model using the training set, continuously adjusting the model's parameters through backpropagation to minimize the loss function; during training, use the validation set to monitor the model's performance. Step 5: Use the trained optimal model to test low-light blurred images and events over a period of time to generate enhanced images.

2. The event-stream-guided low-light image deblurring method according to claim 1, characterized in that: In step 1, the dataset was obtained using three cameras, including two high-resolution RGB cameras and one event camera. One of the RGB cameras used images taken during a 2ms exposure time under normal lighting as ground truth labels. Another RGB camera and an event camera were added with a beam splitter to simulate low-light scenes. The exposure time of both cameras was set to 16ms to obtain a blurred image. The original event data format was... ,in For pixel position, For the first The timestamp of the event trigger For the first The polarity of an event.

3. The event-stream-guided low-light image deblurring method according to claim 1, characterized in that: In step 2, the step of preprocessing the raw event data into spatiotemporal voxels is to give a time window. Divide it into B sub-time windows and embed the events as tensors using the following method. : ， in, The time window corresponding to the current dense spatiotemporal voxels. and These are the timestamps corresponding to the start and end times of the time window, respectively. For the first The timestamp of the event trigger The number of sub-time windows obtained by dividing the time window along the time dimension. For the generated spacetime voxels in the first A time channel, a spatial location voxel values ​​at the location, For the index of the time channel, and These are the x and y coordinates of the spatial location, respectively. For normalized timestamps, For the first The polarity of an event, For the first The pixel coordinates of each event. It is a one-dimensional linear interpolation kernel function. , .

4. The event-stream-guided low-light image deblurring method according to claim 1, characterized in that: In step 3, an end-to-end low-light blur enhancement network with two branches (event and image) is designed to output the enhanced image. The overall network flow is as follows: 1) Input a sequence of 3 consecutive blurred images With corresponding dense spatiotemporal voxels Its spatial resolution is , Given the spatial height of the input image and voxels, Given the spatial width of the input image and voxels, 2) By inputting the image branch shallow encoder and the event branch shallow encoder respectively, shallow image features are obtained. and shallow features of events , 3) In the event branch, an event denoising module and a spatiotemporal trajectory analysis module are introduced sequentially. The event denoising module uses prior knowledge of inter-frame feature changes in images and spatiotemporal consistency cues to learn pixel-level reliability weights. Selective suppression of shallow event features yields denoised event features. The spatiotemporal trajectory analysis module is based on Spatiotemporal dispersion scores are generated through statistical distribution in the time dimension. Adaptive decoupling and reweighting are performed on the unsteady motion trajectory to obtain the structural features after trajectory analysis. With motion characteristics , 4) , , and After cross-modal fusion, the input is the decoding and reconstruction network, and the output is the enhanced image. .

5. The event-stream-guided low-light image deblurring method according to claim 4, characterized in that: The event denoising module processes the data as follows: First, it performs the following steps respectively: and Apply Convolutional projection onto the latent space yields image features. and event characteristics , Then, the event characteristics Applying 3D depthwise separable convolution to model the temporal and spatial coherence of events, and then... Convolutional processing performs channel blending to obtain event context features. Finally, regarding pass Convolution output score The validity confidence score of the event branch is obtained. Based on image features Constructing inter-frame variation For each moment definition: Again Applying the same 3D depthwise separable convolution as the event branch yields the time-varying prior context. Calculate the difference between the event context and the changing prior context. Related items , in, The algorithm performs element-wise multiplication, then concatenates the data along the channel dimension, and finally outputs a cross-modal score after convolution. : Introducing learnable parameters and A dynamic weighting mechanism is constructed to fuse the two scores mentioned above and then pass them through a Sigmoid function to obtain pixel-level reliability weights. : Will Extending along the time dimension yields The shallow features of the event are weighted pixel by pixel, and then residual connections are used to obtain the denoised event features: in, Parameters for controlling the suppression intensity, for The pixel-level event reliability weight map obtained after time expansion These are denoised event features, which serve as input features for subsequent spatiotemporal trajectory analysis and cross-modal fusion modules.

6. The event-stream-guided low-light image deblurring method according to claim 5, characterized in that: The spatiotemporal trajectory analysis module specifically establishes a decoupling mechanism for the dynamic and static features of unsteady spatiotemporal trajectories based on statistical distribution. Will By performing temporal dimension expansion and channel permutation, a standard five-dimensional spatiotemporal tensor input is constructed. : The high-dimensional hidden layer features of multiple channels are transformed into a scalar field that can be statistically analyzed, and then a learnable feature dimensionality reduction projection operator is used. Event features are mapped to a temporal response intensity space to extract pixel-level response amplitudes for spatiotemporal dispersion distribution calculations. : Based on response amplitude We introduce time moment analysis to parametrically describe the evolution of events. By calculating the zeroth moment, the first normalized moment, and the second central moment along the time dimension, we quantify the central moment of event triggering and the diffusion extent, respectively. in, The discrete energy within the time window. The position of the center of gravity of the event stream within the time window. For spatiotemporal discreteness, For the first A slice of response at a time index. To prevent stable terms with a denominator of zero, Secondly, considering and The numerical differences between the two indicators mean that direct fusion will lead to computational imbalance. Therefore, unifying the two indicators to... The standard evaluation criteria, which will soon Normalization ,in, The size of the continuous input time window, which takes a value of 3, and for... Perform stable compression Calculate the spatiotemporal dispersion score : in, As a balance factor, it is set to 0.5 in the module. Center of time The normalized value, Spatiotemporal discreteness The value obtained after stable compression is finally adaptively scaled and truncated. Internally, output spatiotemporal dispersion score : in, and All are learnable parameters. based on Generate complementary pixel-level gate weights and : in, This is a decoupling threshold used to adaptively define the boundary between steady-state and non-steady-state characteristics; This is the gating sharpness coefficient. For steady-state structure gating weights, For motion gating weights, A residual soft modulation-based processing strategy is introduced to obtain frame-by-frame motion features. : in, Preserve bias for context. For 3D convolution, An adaptive temporal attention aggregation mechanism is introduced, with the time window size for continuous input being [value missing]. , The time-by-time event features in the representation are as follows ,in By introducing a learnable temporal distribution control factor, an asymmetric attention weight distribution is constructed, and the original event features are weighted and projected to obtain single-frame aggregated features. : in, To be assigned to the Attention weight at any moment As a learnable decay control factor, right By applying a spatial two-dimensional enhancement module incorporating a channel attention mechanism, and a frequency-domain Gaussian low-pass filter, structural features are obtained. : in, For frequency domain Gaussian low-pass filtering, For a spatial two-dimensional augmentation module that includes a channel attention mechanism, Utilizing a spatial two-dimensional augmentation module incorporating a channel attention mechanism to analyze structural features Extracting steady-state structure terms : Will Input is fed into a 3D convolutional block initialized with a temporal difference operator, a temporal zero-mean constraint is applied, and the dynamic increment term is extracted. : in, For the expectation along the time dimension, steady-state structure terms Broadcasting along the time dimension, with dynamic incremental terms. Linear superposition in the spatiotemporal domain outputs a spatiotemporal evolution representation. : Structural features As a static prior for spatiotemporal sharing, a full-time time-series mapping is performed on the time axis to obtain a result consistent with the dimension of motion features. Frame-by-frame motion features and spatiotemporal evolution representation The channels are concatenated and then 3D convolution is used to output dynamically perceptual features. : Finally, by combining the complementary gating generated earlier, the following is applied: With structural features Adaptive feature splitting and weighted modulation are performed to output the decoupled structural features. With motion characteristics : Subsequent stages will extend ELEDNet's alignment and feature enhancement modules.

7. The event-stream-guided low-light image deblurring method according to claim 6, characterized in that: In acquiring and Then, a deep spatiotemporal feature encoding network based on transposed attention is first employed. Cross-modal features are encoded, and a multi-scale pyramid feature representation is constructed by combining downsampling operations: in, For shallow image features, These are shallow features of the event. For a feature encoder based on transpose attention, and The first Image pyramid features and event pyramid features at various scales At the same time, the output of the pre-spatiotemporal trajectory analysis module Incorporate it into the coding system and simultaneously generate a motion-guided feature pyramid of the corresponding scale: in, For the corresponding level index of the multi-scale pyramid, its value set is set as follows: , For the first Motion pyramid features at various scales After acquiring the pyramid features of the three modalities mentioned above, an event-guided deformable temporal alignment module is introduced to perform coarse-to-fine spatial alignment, based on the current time. For reference frames of adjacent times When aligning features, at any scale Below, the network predicts the network. Joint estimation of spatial sampling offset With modulation mask : in, Indicates the upsampling operator, In the first At this scale, the near moment Features aligned to reference time The spatial sampling offset predicted at that time, The modulation mask corresponding to the spatial sampling offset. Subsequently, a deformable alignment operator with offset and mask is used. Spatially align the features of neighboring frames and output the aligned features: Ultimately, at the original resolution scale At this point, the module aggregates the features of all aligned neighboring frames and the reference frame, and outputs a spatially aligned image feature stream. .

8. The event-stream-guided low-light image deblurring method according to claim 7, characterized in that: After the alignment module EDTFA, the cross-modal feature enhancement module SFCM-FE is used to collaboratively recover the main structure of the scene by utilizing the alignment frame and event features, and then the structural features output by the spatiotemporal trajectory analysis module are applied. As a structural prior input, the input features are first encoded using a CNN to construct a multi-scale feature pyramid: in, In scale The alignment frame features below, In scale The following event characteristics, In order to scale The structural features below, It is a multi-scale feature encoder. , , As input to the feature enhancement module SFCM-FE, the scale is obtained. Enhanced features , First, compare with the previous scale. Enhancement results Upsampling: in It is a 4×4 deconvolution upsampling layer. For the previous scale Enhanced features, for The current scale is obtained after deconvolution upsampling. The features below are then concatenated and convolved with event features, aligned frame features, structural features, and upsampled features to obtain fused features: Will The input is a spectral filtering-based feature enhancement module SFCM-FE, which ultimately yields the enhanced multi-scale pyramid features. , Will The input pyramid decoding and reconstruction network performs stepwise upsampling and multi-scale fusion to finally generate the enhanced image. .

9. The event-stream-guided low-light image deblurring method according to claim 7, characterized in that: The training input consists of a three-frame blurred image sequence and its corresponding dense spatiotemporal voxels. During training, the input is randomly cropped to a size of 256×256, with a batch size of 1. The total number of training epochs is set to 201, and the number of data loading threads is 8. The network parameters are updated using the Adam optimizer, and the backbone learning rate is set to... Cosine annealing is used to decay the learning rate, with the minimum learning rate set to [value missing]. Furthermore, gradient pruning is performed after each backpropagation to stabilize training, with a pruning threshold set to 5.

0. Multi-scale supervision is used for training loss, and the network output includes reconstruction results at three scales: full resolution, 1 / 2 resolution, and 1 / 4 resolution. Pixel reconstruction loss is calculated separately for each scale and then weighted and summed. The loss is defined as: in, and The first The predicted image at each scale and the ground truth image at the corresponding scale are weighted by the following coefficients: The backbone parameter set uses cosine annealing scheduling, while the denoising module parameter set uses a higher initial learning rate. , This is the L1 loss.

Images