Multi-speed scale scene imaging method based on event camera

Abstract

The invention discloses a multi-speed scale scene imaging method based on an event camera, and the method comprises the steps: determining an optimal event triggering threshold value for a static scene and a dynamic scene, and enabling a new event to come from a threshold value of which the proportion of a new pixel approaches zero when the threshold value is reduced, the dynamic optimal threshold value is a threshold value that the proportion of resting isolated pixels starts to rapidly decrease due to threshold value increase; reconstructing a static background frame according to the static optimal threshold value; collecting a multi-speed event stream according to the dynamic optimal threshold value, and extracting and fusing the multi-speed event stream into a dynamic event voxel lattice by adopting a heterogeneous time window; and splicing the static background frame and the dynamic voxel grid, inputting the spliced static background frame and dynamic voxel grid into an LSTM (Long Short Term Memory) loop network based on a U-Net architecture, and reconstructing a high-quality video sequence containing a clear static background and multi-speed motion at the same time. According to the method, the problem of data redundancy or information loss caused by a fixed threshold value of a standard event camera is solved, and efficient and high-quality reconstruction of a multi-speed scale scene is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an imaging method for multi-velocity scale scenes based on an event camera. Background Technology

[0002] Modern imaging systems aim to faithfully capture real-world scenes encompassing a wide range of lighting and motion variations. To address the exposure limitations caused by brightness variations, researchers have developed High Dynamic Range (HDR) imaging techniques, which extend the brightness range by fusing multiple exposures. Similarly, real-world scenes require Multi-Velocity Scale Imaging (MVSI) to handle the vast variations in motion speed from static backgrounds to fast-moving objects. That is, treating motion at different speeds within a scene as a single imaging target, the explicit goal is to capture sharp and consistent images in a data-efficient manner, even when motion speeds span multiple orders of magnitude (e.g., 1–1000 pixels / second).

[0003] Traditional frame-based cameras rely on global exposure mechanisms and fixed frame rates. This design leads to an inherent trade-off between speed range, image fidelity, and data efficiency when attempting to achieve MVSI. When shooting scenes with rapidly changing motion, the camera must operate at a high frame rate to capture the fastest movement. This not only results in significant data redundancy due to oversampling of static or slow-moving areas but also limits the exposure time per frame, degrading image quality in low light or fast-moving conditions. Therefore, they are inherently unsuitable for shooting scenes with multiple speed scales. It's important to note that the multi-bracketing exposure strategy used in HDR imaging is not applicable here because fast motion causes severe motion blur, making frame alignment difficult. While high frame rate cameras can handle high-speed motion, they inevitably introduce data redundancy, which greatly complicates subsequent data processing.

[0004] Event cameras are a novel type of neuromorphic visual sensor. Unlike traditional frame cameras, they are based on asynchronously triggered event streams caused by changes in brightness. Specifically, each pixel of an event camera operates independently, generating events only when a local brightness change is detected, thus avoiding redundancy and latency. Simultaneously, this event stream boasts high temporal resolution at the microsecond level, enabling event cameras to capture a wider range of motion than frame cameras. Recent research has further demonstrated their ability to capture both static and fast-moving scenes. These characteristics make event cameras ideal for capturing motion at varying speeds, enabling multi-velocity scale imaging.

[0005] Despite the numerous advantages of event cameras, they cannot be directly applied to MVSI tasks because standard event cameras employ a global threshold. This hinders local adaptation and introduces a fundamental trade-off: lowering the camera threshold increases pixel sensitivity, enhancing its ability to perceive static scenes, but leads to event avalanche and bandwidth saturation when faced with fast motion; while increasing the camera threshold effectively reduces data volume, it suppresses static or slow-moving information, resulting in scene reconstruction failure. This conflict between threshold setting and scene requirements demonstrates that a single global threshold cannot meet the imaging needs across velocity ranges in multi-velocity scale scenes. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by proposing a multi-velocity-scale scene imaging method based on an event camera. This method dynamically adjusts the threshold of each pixel based on its local motion features, thereby balancing pixel sensitivity and data efficiency. Ideally, a multi-threshold event camera should assign different thresholds to motions of different scales in the scene, enabling fine-grained adaptive sampling of static, slow, and fast dynamics. This configuration maintains both temporal sensitivity and robustness, ensuring that no region in the scene is over- or under-represented in the event stream. The resulting event data provides a more uniform spatiotemporal representation of the scene, avoiding the bias introduced by a single global threshold.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An imaging method for multi-velocity scale scenes based on an event camera includes the following steps:

[0009] In a first aspect, the present invention provides an imaging method for multi-velocity scale scenes based on an event camera, comprising the following steps:

[0010] S1. Determining the Optimal Threshold: Determine the optimal event trigger threshold for both static and dynamic scenes. When determining the optimal threshold for the static scene, decrease the threshold sequentially and calculate the proportion of pixels from the first triggered event in the newly added events after each threshold decrease. The threshold corresponding to when this proportion approaches 0 is taken as the optimal threshold for the static scene. When determining the optimal threshold for the dynamic scene, increase the threshold sequentially and calculate the proportion of spatiotemporally isolated pixels among the pixels that stop triggering events due to the threshold increase after each threshold increase. The threshold corresponding to when this proportion begins to decrease rapidly is taken as the optimal threshold for the dynamic scene.

[0011] S2. Static background reconstruction: Collect static event streams based on the optimal threshold of the static scene, obtain the original static event frames through convolutional integral method, and input them into the denoising network to obtain clean static background frames.

[0012] S3. Dynamic event fusion: Collect dynamic event streams at different speeds based on the optimal threshold of the dynamic scene, select different time windows for extraction of event streams at different speeds, normalize the extracted event streams on the time axis and fuse them into a dynamic event voxel.

[0013] S4. Video Reconstruction: Static background frames and dynamic event voxels are concatenated along the channel dimension to form a fused tensor, which is then input into an LSTM recurrent network based on the U-Net architecture to reconstruct a high-quality video sequence.

[0014] Further, in step S4, the LSTM recurrent network based on the U-Net architecture includes:

[0015] The encoding layer, consisting of multiple downsampling modules and a long short-term memory (LSTM) module, is used to extract temporal feature information from the fused tensor.

[0016] The intermediate layer is used for deep feature extraction.

[0017] The decoding layer recovers the spatial resolution step by step through deconvolution and skip connections, and outputs the reconstructed frame sequence in the prediction layer.

[0018] Furthermore, the downsampling module employs a feature extraction module based on NAF-Block; the core calculation form of NAF-Block is as follows: ,in, It is a convolution kernel of size w, where d is the number of output channels. represents the convolution operation, x is the input feature tensor, b is the bias, IN(·) represents the normalization operation, and SGU is the simple gated unit.

[0019] Furthermore, the intermediate layer introduces a channel attention mechanism, the calculation of which is expressed as follows: Where σ is the Sigmoid function, , Let be a linear transformation matrix.

[0020] Furthermore, the LSTM module is a Convolutional Long Short-Term Memory (ConvLSTM) module.

[0021] Furthermore, when training the LSTM recurrent network based on the U-Net architecture, spatial augmentation and temporal perturbation are applied to the input sequence; wherein, the spatial augmentation includes random pruning, flipping, and rotation, and the temporal perturbation includes random pauses, noise injection, and hot pixel simulation.

[0022] Furthermore, when training the LSTM recurrent network based on the U-Net architecture, mean squared error (MSE) is used as the supervision loss, and the first and last frames of each training sequence are pruned to enhance network stability.

[0023] In a second aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any one of claims 1 to 7.

[0024] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 7.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1. The present invention provides an imaging method for multi-velocity scale scenes based on an event camera. Different thresholds are assigned to each pixel according to different scene content, so as to achieve the shooting of multi-velocity scale scenes with the lowest data volume and bandwidth consumption. This effectively solves the problems of data redundancy (using low threshold settings to shoot high-speed motion) or information loss (using high threshold settings to shoot static or slow motion) faced by standard event cameras due to threshold mismatch.

[0027] 2. This invention calibrates the relationship between event pixel response and scene light intensity, scene contrast, threshold setting, and motion speed. It proposes two different optimal threshold finding strategies for static and dynamic scenes. The optimal low threshold is defined as the threshold setting when, during static scene shooting, the proportion of new events originating from "new pixels" approaches zero as the threshold decreases. The optimal high threshold is defined as the threshold setting when, during dynamic scene shooting, the proportion of isolated pixels among resting pixels rapidly decreases as the threshold increases. This solves the problems of noise or data redundancy encountered when using a uniform threshold to shoot different scenes.

[0028] 3. This invention proposes a heterogeneous time window fusion algorithm. When using a multi-threshold event camera to capture scenes at multiple speed scales, it extracts and fuses events using a matching time window for motion at different speed scales, effectively solving the problem of unified representation of event streams with uneven density triggered by motion at different speeds.

[0029] 4. This invention proposes a fully convolutional neural network based on the UNet architecture and adopts an end-to-end reconstruction method. After inputting fused event data into the network, a video with a static background and motion at different speeds can be obtained. The dynamic event streams triggered by motion at different speeds are fused into a unified event voxel grid through a heterogeneous time window fusion algorithm. Then, the two are connected to form a fusion tensor, which is input into the fully convolutional neural network based on the U-Net architecture and outputs a video with a static background and motion at different speeds. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1 A flowchart of an imaging method for multi-velocity scale scenes based on an event camera, provided in an embodiment of the present invention. Detailed Implementation

[0032] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] This invention provides an imaging method for multi-velocity scale scenes based on an event camera. Instead of using a global threshold setting, the threshold of each event pixel is adjustable. It assigns the optimal threshold to the corresponding pixel based on different scene content (scene light intensity, contrast, motion speed, etc.), thereby achieving the faithful recording of all scene information with the lowest data volume. This breaks the trade-off between velocity range, image fidelity, and data efficiency faced by traditional imaging models when capturing multi-velocity scale scenes.

[0034] like Figure 1 As shown, the overall technical solution of this invention mainly includes four core steps: optimal threshold determination, static background reconstruction, dynamic event fusion at different speeds, and video reconstruction. Each step is implemented through different modules, as detailed below:

[0035] S1. Determining the Optimal Threshold: When using a multi-threshold imaging model to capture scenes at multiple velocity scales, different thresholds need to be assigned to each pixel based on the different scene content. When capturing static scenes, the threshold needs to be lowered to improve the camera's perception capability. Each time the threshold is lowered, some pixels that have never triggered events before will start triggering events. These pixels that go from resting to active are called "new pixels" activated by this threshold reduction. In contrast, pixels that have already triggered events will trigger more events after the threshold is lowered due to increased sensitivity. These pixels are called "old pixels". By statistically analyzing the proportion of new events from "new pixels" in this threshold reduction, we can determine whether this threshold reduction has brought new scene information. When this proportion approaches 0, it indicates that subsequent threshold reductions will no longer add new scene information, but will only bring duplicate information, causing data redundancy and more noise. Therefore, the optimal threshold for capturing static backgrounds is the threshold setting when the proportion of new events from "new pixels" approaches 0.

[0036] When capturing dynamic scenes, it's necessary to increase the threshold to reduce the number of events, while ensuring that information is not lost due to an excessively high threshold. Each time the threshold is increased, some pixels that would normally trigger events will stop triggering and become inactive. Compared to pixels that represent scene information (which often have spatiotemporal correlation), pixels that do not represent any effective scene information (which often exhibit spatiotemporal isolation) are more susceptible to the impact of a threshold increase and will become inactive first. By statistically analyzing the proportion of isolated pixels among those that become inactive due to the threshold increase, we can determine whether scene information was lost during this threshold increase. Therefore, the optimal threshold for capturing dynamic backgrounds is the threshold setting when this proportion begins to decrease rapidly, indicating that subsequent threshold increases will lead to the loss of scene information.

[0037] S2. Static Background Reconstruction: The static event stream is processed using convolution integral to obtain the original event frames, which are then input into the denoising module to obtain clean static background frames. The denoising module can be the publicly available denoising module ESR-Net (Gao Q, Duan P, Lou H, et al. Unified Reconstruction of Static and DynamicScenes from Events[C] / / Proceedings of the Computer Vision and PatternRecognition Conference. 2025: 27914-27923).

[0038] S3. Fusion of Dynamic Events at Different Speeds: Event streams triggered by motion at different speeds have uneven density in the time dimension. High-speed event streams are often dense, thus requiring narrower time windows for extraction to avoid motion blur. In contrast, slow-speed event streams are often sparse, thus requiring wider time windows for extraction to obtain sufficient event information for reconstruction. Different time windows are selected for extraction of event streams at different speeds. Then, the event streams from multiple time windows are normalized on the time axis and fused into a single dynamic event voxel.

[0039] S4. Video Reconstruction: The fused dynamic event voxel grid is concatenated with the static background frame to form a frame + voxel grid fusion tensor, which is then input into an LSTM recurrent network based on the U-Net architecture. The encoding layer contains multiple downsampling and LSTM modules to extract temporal feature information, while the intermediate layers perform deep feature extraction while simultaneously increasing the feature dimension. The decoding layer reconstructs scene information from the feature map and reconstructs high-quality scene frames in the prediction layer.

[0040] The multi-threshold event imaging method proposed in this invention includes a spatiotemporal reconstruction network based on the U-Net framework and incorporating LSTM recurrent units, used to generate high-quality video sequences from dynamic-static fused tensors. This network is first trained on the publicly available E-StaDyn synthetic dataset, and then tested and deployed on local event data collected by the patented system. The training process is as follows:

[0041] (1) Network Structure Design. During the input phase, the system converts static events into static frames through integration, and then concatenates them with dynamic voxel data channel by channel to form a 3D fusion feature, which is then input into the network's encoding section. The encoding end employs multiple NAF-Block-based feature extraction modules, using a convolutional structure without non-linear activation to improve computational efficiency. The core computational form of a single NAF-Block can be described as... ,in, It is a convolution kernel of size w, where d is the number of output channels; The expression represents a convolution operation; x is the input feature tensor; b is the bias; IN(·) represents a normalization operation used to scale the feature values; SGU (Simple Gated Unit) is responsible for feature selection during the encoding stage, enabling the network to adaptively retain key information relevant to the reconstruction task. After multi-layer encoding, the features are further fed into an intermediate layer composed of stacked NAF-Blocks. An additional channel attention mechanism is introduced here to enhance the response relationship between channels. Its calculation can be represented as... Where σ is the Sigmoid function, , This is a linear transformation matrix. This module enables the network to automatically increase the weights of important channels, thereby enhancing the joint representation of dynamic textures and static backgrounds. Simultaneously, each coding layer is equipped with a ConvLSTM structure to capture the natural temporal dependencies of the event stream, ensuring structural consistency across multiple frame sequences.

[0042] (2) Decoding and Prediction. In the decoding stage, spatial resolution is restored step by step through deconvolution and skip connections. The skip structure fuses the shallow detail features from the encoding end with the high-dimensional semantic features, so that the final reconstructed image maintains the same size as the input. The prediction head is responsible for outputting the final reconstructed frame sequence, realizing a unified dynamic-static restoration of the scene.

[0043] (3) Training and Optimization Strategies. During training, the standard mean squared error (MSE) was used as the sole supervised loss to measure the difference between the generated and reference images. The optimizer used was AdamW, and the initial learning rate was set to... During training, spatial augmentation (random cropping, flipping, rotation) and temporal perturbation (random pauses, noise injection, hot pixel simulation) are applied simultaneously to improve the model's generalization ability. To prevent unstable propagation of recurrent structures at sequence boundaries, the first and last frames of each training sequence are cropped, thereby enhancing the stability of the LSTM's internal state. After multiple rounds of training iterations, the network can generate temporally smooth and detailed reconstructed images on real data, simultaneously presenting clear static backgrounds and moving targets across velocity scales, meeting the core requirements of this patented multi-threshold event imaging system.

[0044] Since there are currently no locally adjustable event cameras, we use time-division multiplexing to validate the proposed imaging method. For a multi-velocity scale scene (repetitive motion), we first use a Prophesee EVK4 event camera to capture a static background. This process requires gradually lowering the event camera's threshold from 0 and recording the event triggering situation during each lowering process. Then, we use the proposed search strategy to determine the optimal static threshold and use this threshold to recapture the static background.

[0045] Then, low-speed and high-speed motion were captured separately, and the proposed search strategy was used to determine the optimal dynamic threshold. For motion at different speeds, different time windows were used to segment the dynamic event stream, which was then fused into a dynamic event stream sequence using a heterogeneous time window fusion algorithm. This sequence was then concatenated with a static background frame to form a fusion tensor, which was fed into the reconstruction network to generate a video with a static background and motion at different speeds.

[0046] For multi-velocity scale scenes with non-repeatable motion, we propose a dual-event camera system for reconstruction. Two Prophesee EVK4 event cameras are combined using a beam splitter to form an imaging system with a shared field of view but different thresholds. One camera is set to a low threshold for capturing static backgrounds and slow motion (the specific threshold setting is determined using the aforementioned static optimal threshold finding strategy), while the other is set to a high threshold for capturing fast motion (the specific threshold setting is determined using the aforementioned dynamic optimal threshold finding strategy). Since the event stream recorded by the low-threshold camera includes slow motion, it needs to be integrated into event frames using an appropriate time window and then processed through a denoising network to obtain a sequence of static background frames. The event stream recorded by the high-threshold camera is then converted into voxel grids and concatenated with the corresponding static background frames to form a fusion tensor, which is input into the reconstruction network for reconstruction.

[0047] The imaging method for multi-velocity scale scenes based on event cameras proposed in this invention has a per-pixel threshold adjustable mechanism. Compared with traditional event cameras, it can faithfully record all information in multi-velocity scale scenes with a lower data volume, breaking through the inherent trade-off between velocity range, imaging quality and data volume.

[0048] This invention measures the relationship between event rate and threshold at illuminance levels ranging from 170 Lux to 670 Lux and motion speeds from static to 1000 pixels per second. A calibration and estimation procedure is designed to determine these operational boundaries in both synthetic and real-world scenes. The proposed optimal threshold-finding strategy determines the optimal threshold for different scenes, overcoming noise and data redundancy caused by shooting with a uniform threshold.

[0049] The heterogeneous time window fusion algorithm proposed in this invention constructs a unified multi-threshold fusion framework, rather than processing each threshold independently. This framework can seamlessly integrate data from any number of thresholds, including but not limited to the dual-threshold case. This framework provides a theoretical basis for reconstructing high-fidelity intensity and motion information at different velocity scales.

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An imaging method for multi-velocity scale scenes based on an event camera, characterized in that, Includes the following steps: S1. Determining the optimal threshold: Determine the optimal event triggering threshold for both static and dynamic scenes. When determining the optimal threshold for the static scene, decrease the threshold sequentially and count the proportion of pixels from the first triggered event in the newly added events after each threshold decrease. The threshold corresponding to the proportion that approaches 0 is taken as the optimal threshold for the static scene. When determining the optimal threshold for a dynamic scene, the threshold is increased sequentially and the proportion of spatiotemporally isolated pixels among the pixels that stop triggering events due to the threshold increase is counted after each increase. The threshold corresponding to the point where this proportion begins to decrease rapidly is taken as the optimal threshold for the dynamic scene. S2. Static background reconstruction: Collect static event streams based on the optimal threshold of the static scene, obtain the original static event frames through convolutional integral method, and input them into the denoising network to obtain clean static background frames. S3. Dynamic event fusion: Collect dynamic event streams at different speeds based on the optimal threshold of the dynamic scene, select different time windows for extraction of event streams at different speeds, normalize the extracted event streams on the time axis and fuse them into a dynamic event voxel. S4. Video Reconstruction: Static background frames and dynamic event voxels are concatenated along the channel dimension to form a fused tensor, which is then input into an LSTM recurrent network based on the U-Net architecture to reconstruct a high-quality video sequence.

2. The imaging method for multi-velocity scale scenes based on an event camera according to claim 1, characterized in that, In step S4, the LSTM recurrent network based on the U-Net architecture includes: The encoding layer, consisting of multiple downsampling modules and a long short-term memory (LSTM) module, is used to extract temporal feature information from the fused tensor. The intermediate layer is used for deep feature extraction. The decoding layer recovers the spatial resolution step by step through deconvolution and skip connections, and outputs the reconstructed frame sequence in the prediction layer.

3. The imaging method for multi-velocity scale scenes based on an event camera according to claim 2, characterized in that, The downsampling module employs a feature extraction module based on NAF-Block. The core computational form of the NAF-Block is as follows: ,in, It is a convolution kernel of size w, where d is the number of output channels. represents the convolution operation, x is the input feature tensor, b is the bias, IN(·) represents the normalization operation, and SGU is the simple gated unit.

4. The imaging method for multi-velocity scale scenes based on an event camera according to claim 2, characterized in that, The intermediate layer introduces a channel attention mechanism, the computation of which is expressed as follows: Where σ is the Sigmoid function, , Let be a linear transformation matrix.

5. The imaging method for multi-velocity scale scenes based on an event camera according to claim 1, characterized in that, The LSTM module is a Convolutional Long Short-Term Memory (ConvLSTM) module.

6. The imaging method for multi-velocity scale scenes based on an event camera according to claim 1, characterized in that, When training the U-Net-based LSTM recurrent network, spatial augmentation and temporal perturbation are applied to the input sequence; wherein, the spatial augmentation includes random pruning, flipping, and rotation, and the temporal perturbation includes random pauses, noise injection, and hot pixel simulation.

7. The imaging method for multi-velocity scale scenes based on an event camera according to claim 1, characterized in that, When training the LSTM recurrent network based on the U-Net architecture, mean squared error (MSE) is used as the supervision loss, and the first and last frames of each training sequence are pruned to enhance network stability.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images