A method for simultaneous reconstruction of dynamic and static scenes based on event cameras

Abstract

The application discloses a method for simultaneously reconstructing dynamic and static scenes based on an event camera, and belongs to the field of image processing.The method comprises the following steps: dividing an original event stream into a space-time grid, setting an event quantity threshold, determining events exceeding the threshold as dynamic events triggered by motion, and determining events not exceeding the threshold as static events triggered by a background; performing convolution integration on the static events to obtain an initial static reconstruction image, and further optimizing the initial static reconstruction image through a static reconstruction denoising network to obtain a static background image; dividing the dynamic events into a voxel grid through a space-time window, fusing the static background image and the voxel grid under the premise of introducing an event label tensor to obtain a fusion tensor; and inputting the fusion tensor into a dynamic and static video reconstruction network to obtain a dynamic video with a static background.The application improves the quality of the static reconstruction image, eliminates visual inconsistency caused by separate reconstruction and then fusion, and avoids model overfitting caused by monotonous data set elements.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for simultaneous reconstruction of dynamic and static scenes based on an event camera. Background Technology

[0002] Event cameras are a novel type of neuromorphic vision sensor. Unlike traditional frame cameras, they do not capture global images at fixed time intervals. Instead, they asynchronously trigger event streams based on changes in brightness. Specifically, each pixel of an event camera operates independently, generating events only when a local brightness change is detected. This event stream has a high temporal resolution at the microsecond level, effectively capturing fast-moving scenes. Therefore, event cameras have significant advantages in high-speed vision tasks, such as robot perception, object tracking, and autonomous driving. However, due to the fundamental difference in the data format output by event cameras compared to traditional frame cameras, their application is limited by compatibility issues with existing vision algorithms.

[0003] To fully leverage the advantages of event cameras while maintaining compatibility with traditional vision algorithms, researchers have recently proposed video reconstruction methods based on event data. These methods utilize event streams to reconstruct grayscale images, giving them visual representation capabilities similar to frame-based cameras. However, existing event-based video reconstruction methods primarily focus on restoring moving regions, often neglecting the reconstruction of static backgrounds, resulting in videos lacking background detail. In many practical applications, such as autonomous driving, robot navigation, and industrial inspection, the lack of background information can affect the stability and robustness of the system.

[0004] Existing research on static scene reconstruction can be broadly categorized into two types: methods relying on external devices and methods based purely on event data. The former typically utilizes external active lighting or special optical devices to enhance the event camera's perception of static scenes. However, the application of these methods is limited by specific hardware and difficult to scale in complex environments. Another type of method attempts to eliminate dependence on external devices and directly recover static scenes from event data. By constructing a statistical relationship between noisy events and scene brightness, static scene reconstruction under constant illumination has been achieved. However, these methods suffer from significant noise interference, resulting in low-quality reconstruction results. Furthermore, existing event data-based video reconstruction methods have significant limitations when handling complex scenes containing both dynamic foreground and static background. For example, recent research has combined static background information with dynamic video reconstructed using a mask fusion method based on an E2VID (Events-to-Video Network, a neural network for reconstructing grayscale images from event data) model. However, this method is limited by the performance of the external neural network and is prone to artifacts at mask boundaries.

[0005] Existing video reconstruction methods based on event data struggle to simultaneously recover both static backgrounds and dynamic targets. The main challenges include:

[0006] 1. Limitations of the event triggering mechanism: The event camera relies on changes in brightness to trigger events. Therefore, in static scenes, the event triggering frequency is low, resulting in less static background information.

[0007] 2. Noise interference problem: Due to hardware characteristics, event cameras are prone to generating noise events under low or high light conditions, which impairs the quality of static scene restoration.

[0008] 3. Difficulty in dynamic-static fusion: Most existing methods use separate modeling to handle static and dynamic regions, and fail to achieve end-to-end unified reconstruction within the same framework.

[0009] The closest existing technology to this invention is "Cao R, Galor D, Kohli A, et al. Noise2Image: noise-enabled static scene recovery for event cameras[J]. Optica, 2025, 12(1): 46-55." This paper proposes a statistical noise model to describe how noise event generation is related to scene intensity. Due to the logarithmic sensitivity of the sensor, the number of events triggered by photon noise is mostly negatively correlated with the illuminance level. The noise events are mapped to scene intensity by statistically counting the number of noise events per pixel. To avoid ambiguity caused by a one-to-many mapping, the Noise2Image network model is proposed to recover the static part of the scene with dynamics from noise events separately. Its specific implementation process is as follows:

[0010] (1) First, use an event camera to capture static scenes and static events triggered by noise. Then, estimate the actual noise event count based on the empirical noise event count measured by the event camera and record the positive and negative polarities of each event.

[0011] (2) The Noise2Image network model is proposed to receive input with two spatial channels for positive event counting and negative event counting. The network is trained to directly map the estimated event counts to the corresponding intensity images.

[0012] (3) The dynamic foreground of the scene is reconstructed into continuous frames through the E2VID network and fused with the background reconstructed by Noise2Image by image segmentation to obtain a dynamic video with a static background.

[0013] The techniques disclosed in the aforementioned literature do not consider the pixel extremum problem faced by pixel-by-pixel integration methods, resulting in low-quality reconstructed images. Furthermore, this technique directly employs the E2VID model when reconstructing dynamic foregrounds in scenes, and the difference between the two reconstruction methods leads to visual inconsistencies between static backgrounds and moving foregrounds. Moreover, the proposed dataset only contains facial elements, lacking diversity. Summary of the Invention

[0014] To address the technical problems of existing methods, this invention proposes a method for simultaneous reconstruction of dynamic and static scenes based on event cameras. By establishing a unified modeling strategy for dynamic and static scenes, high-precision reconstruction of event data is achieved. This invention not only improves the reconstruction quality of static backgrounds but also effectively preserves the detailed information of dynamic targets, making the reconstruction capabilities of event cameras more practical.

[0015] The technical solution adopted by this invention to solve the technical problem is as follows:

[0016] This invention provides a method for simultaneous reconstruction of dynamic and static scenes based on an event camera, comprising the following steps:

[0017] Step 1: Separate dynamic and static events;

[0018] The original event stream is divided into a spatiotemporal grid. An event quantity threshold is set. Events exceeding the threshold within the spatiotemporal grid are judged as dynamic events triggered by motion, while events not exceeding the threshold are judged as static events triggered by the background.

[0019] Step 2: Static background reconstruction;

[0020] The static event is first processed by convolution integral method to calculate the initial static reconstructed image, and then further optimized by static reconstruction and denoising network to obtain static background image;

[0021] Step 3: Integration of static and dynamic events;

[0022] The dynamic event is first divided into a voxel grid through a spatiotemporal window, and then the static background image and the voxel grid are fused together with the event label tensor to obtain a fused tensor.

[0023] Step 4: Video reconstruction;

[0024] The fusion tensor is input into the static video reconstruction network to obtain dynamic video with static background.

[0025] Furthermore, the time span of the spatiotemporal grid is a fixed time step, and the spatial size is a fixed number of pixels.

[0026] Furthermore, the specific implementation process of the convolution integral method is as follows: the convolution kernel slides through the entire pixel plane, calculates the number of events in each convolution kernel region, and assigns the statistical value of the number of events to the center pixel to obtain the initial static reconstructed image.

[0027] Furthermore, the static reconstruction and denoising network includes an input layer, an encoding layer, an intermediate layer, a decoding layer, and a prediction layer. The encoding layer consists of multiple custom SRD-Blocks and adopts a U-Net structure and a non-linear activation design. The encoded features are fed into multiple SRD-Blocks without activation functions. A channel attention mechanism is used to enhance the feature representation ability between channels. During decoding, deconvolution and skip connections are used to restore image details.

[0028] Furthermore, the size of the voxel grid is consistent with that of the static background image, i.e., both are pixel-level resolution.

[0029] Furthermore, the event label tensor is used to identify the static / dynamic attributes of each event.

[0030] Furthermore, the motion and still video reconstruction network adopts a convolutional neural network with a U-Net structure.

[0031] Furthermore, in the convolutional neural network, the encoder integrates an LSTM module to extract global temporal features and calculate the feature map of the continuous fused tensor; the intermediate layer keeps the feature map size unchanged and only performs depth feature extraction; the decoder restores high-quality dynamic continuous frames with background, presenting dynamic foreground while retaining static background information.

[0032] Furthermore, the static reconstruction and denoising network is trained on the real E-Static dataset.

[0033] Furthermore, the motion and still video reconstruction network is trained on the E-StaDyn synthetic dataset.

[0034] The beneficial effects of this invention are:

[0035] (1) This invention employs a two-step reconstruction strategy combining convolutional integral method and a static reconstruction denoising network, which solves the pixel extremum problem and improves the quality of statically reconstructed images. Specifically, the static scene image reconstructed using the convolutional integral method significantly improves the quality and information richness of the initial reconstruction, providing learnable samples for subsequent denoising operations. The static reconstruction denoising network used in this invention can effectively learn the noise response model of static scenes, thereby performing targeted denoising processing on the statically reconstructed image to obtain a high-quality static background image.

[0036] (2) The present invention adopts an end-to-end reconstruction method. After the event data is input into a unified framework, dynamic video with static background can be obtained directly, eliminating the visual inconsistency caused by separate reconstruction and fusion.

[0037] (3) The dataset used in this invention contains more than 200 different static scenes indoors and outdoors, all of which are actually captured by the camera, thus avoiding model overfitting caused by the monotony of the dataset elements.

[0038] (4) By constructing a unified framework for simultaneously reconstructing static background and moving foreground from event data, this invention enables the simultaneous reconstruction of static background and moving foreground, solves the problem of background information loss in traditional video reconstruction technology, and achieves the goal of capturing real scenes using a single event camera. Attached Figure Description

[0039] Figure 1 The present invention provides a flowchart of a method for simultaneous reconstruction of dynamic and static scenes based on an event camera.

[0040] Figure 2 The flowchart illustrates a method for simultaneously reconstructing dynamic and static scenes based on an event camera, as provided in this embodiment. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the accompanying drawings.

[0042] This invention provides a method for simultaneous reconstruction of dynamic and static scenes based on event cameras. First, it constructs a unified framework (Unified Reconstruction of Static and Dynamic Scenes from Events, URSEE) for simultaneously reconstructing static backgrounds and moving foregrounds from event data. Then, it achieves simultaneous reconstruction of dynamic and static scenes through four core steps. For example... Figure 1 As shown, the overall technical solution of this invention mainly includes four steps: separation of dynamic and static events, reconstruction of static background, fusion of dynamic and static events, and video reconstruction. Each step is implemented through different modules, as detailed below:

[0043] Step 1: Separate dynamic and static events;

[0044] The raw event data is first divided into spatiotemporal grids, each with a fixed time span (e.g., 10ms) and a fixed number of pixels (e.g., 20×20 pixels). The number of events contained within each spatiotemporal grid varies. When the number of events exceeds a preset threshold, the events within that grid are classified as "dynamic events" triggered by motion; otherwise, they are categorized as "static events" triggered by the background. Through this dynamic-static event separation process, the raw event stream is effectively separated into a clean static event stream and a dynamic event stream, which are then input into different reconstruction branches for targeted reconstruction processing.

[0045] Step 2: Static background reconstruction;

[0046] The static event stream extracted in step one is first processed by convolutional integral method to calculate the initial static reconstructed image. Specifically, a 3×3 convolutional kernel is used to slide across the entire pixel plane, and the number of events within each kernel region is calculated. This statistical value is then assigned to the center pixel (Padding = 1, stride = 1). Subsequently, the resulting initial static reconstructed image is further optimized by a static reconstruction and denoising network to obtain a higher quality static background image.

[0047] Step 3: Integration of static and dynamic events;

[0048] The dynamic events extracted in step one are divided into a voxel grid through a spatiotemporal window. The size of this voxel grid is consistent with the reconstructed static background image, i.e., both are pixel-level resolution. Therefore, the two can be directly fused to form a fusion tensor. This fusion tensor not only contains static background information, but also retains the spatiotemporal features of the dynamic events due to the characteristics of the voxel grid. To further accurately distinguish between the moving foreground and the static background, an event label tensor is additionally introduced during the fusion process to identify the motion / static attributes of each event. The final fusion tensor consists of three parts: the static background image, the dynamic event voxel grid, and the event label tensor, and serves as the input to the subsequent motion / static video reconstruction network.

[0049] Step 4: Video reconstruction;

[0050] A convolutional neural network with a U-Net structure is used as the network for reconstructing static and dynamic videos. The fusion tensor is input into the U-Net structure convolutional neural network. In this convolutional neural network, the encoder integrates an LSTM module to extract global temporal features and calculate the feature map of the continuous fusion tensor. The intermediate layer keeps the feature map size unchanged and only performs depth feature extraction. Finally, the decoder restores high-quality dynamic continuous frames with background, ensuring that both static background information is preserved and dynamic foreground is accurately presented.

[0051] This invention mainly includes two neural network models: a static reconstruction and denoising network and a dynamic and static video reconstruction network, which are trained on the E-Static real dataset and the E-StaDyn synthetic dataset, respectively. The specific training process is as follows:

[0052] (1) Real E-Static dataset;

[0053] E-Static is a dataset collected from real-world static scenes, covering various indoor and outdoor environments. Data acquisition was accomplished using a hybrid event-frame camera system, consisting of a traditional RGB camera (Alvium1800 U-240c) and an event camera (Sony IMX636). The two cameras achieved field-of-view alignment using a beam splitter with a 1:9 light ratio (event camera:frame camera), effectively avoiding overexposure of event pixels due to excessive lighting, thereby improving the reconstructability of static event images. Based on empirical parameter settings, this invention sets the event camera's ON threshold to -17 and OFF threshold to -50 to ensure a sufficient number of static events are generated for reconstruction. This E-Static real-world dataset contains 200 sets of original event streams and their corresponding high-quality frame images as ground reality data. For ease of training, the original 1280×720 resolution images were divided into six sub-images, each 512×512 pixels, thus constructing 1290 training samples and 96 test samples.

[0054] (2) E-StaDyn synthetic dataset;

[0055] E-StaDyn is a synthetic dataset designed specifically for static and dynamic event reconstruction tasks. It contains 130 scenes, each with a unique static background and dynamic foreground. The data is generated by pairing multiple high-quality static images with multiple 3D models exhibiting random motion, and then rendering them using Blender software. Each scene renders 600 consecutive frames as ground truth frames. Subsequently, the rendered frames are event-driven using the DVS-Voltmeter simulator to generate corresponding synthetic event streams. The final dataset is divided into 115 scenes for training and 15 scenes for testing, covering diverse combinations of static and dynamic events and providing ample support for training networks for static-dynamic fusion and static-dynamic video reconstruction.

[0056] (3) The training process of neural networks;

[0057] S1: For the static reconstruction and denoising network, the original event stream of each scene in the E-Static real dataset is reconstructed into an initial image by convolution integral method. This initial image contains both scene grayscale information and noise information, and serves as the input to the static reconstruction and denoising network.

[0058] S2: The entire static reconstruction and denoising network consists of an input layer, an encoding layer, intermediate layers, a decoding layer, and a prediction layer. The encoding layer comprises multiple custom SRD-Blocks and employs a U-Net (a U-shaped encoder-decoder structure with cross-connection) architecture and a non-linear activation-free design to reduce computational complexity and improve inference speed. The computation of each SRD-Block is as follows: in This represents a convolution kernel with a window size of w, and d represents the number of channels in the output tensor. The convolution operation is represented by , x represents the input tensor or the output of the upper feature convolutional layer, b represents the bias term, IN(·) is the batch normalization operation, which normalizes the existing tensor, SGU (Simple Gated Unit) is used for feature selection, and Y is the encoded feature. The encoded feature enters multiple SRD-Blocks without activation functions, and a channel attention mechanism is used to enhance the feature representation ability between channels, calculated as follows: Y = X·σ(W1(W0X)), where X represents the input feature tensor, σ represents the sigmoid activation function, and W1 and W0 both represent fully connected layers. This channel attention mechanism enables the model to adaptively focus on important feature channels.

[0059] S3: During decoding, deconvolution and skip connections are used to recover image details. The decoder output is a reconstructed image of the same size as the input. If the mean squared error (MSE) is used as the sole loss function during training, then the loss function is defined as follows: Where X i Indicates the reference target of the input image. The output image represents the network, and N represents the total number of pixels in the image. During optimization, the Adam optimizer is used to update the network parameters through backpropagation, with a learning rate set to 0.0001. After multiple iterations on the training set, the network converges stably and effectively completes the image restoration task.

[0060] S4: The dynamic and static video reconstruction network has a similar structure to the static reconstruction and denoising network. It is trained using the E-StaDyn synthetic dataset and validated on real data. Each scene in the E-StaDyn synthetic dataset generates a synthetic event stream through the DVS-Voltmeter simulator. This event stream is divided into 40 groups of continuous voxel grids and input into the dynamic and static video reconstruction network. The encoder's LSTM module extracts the temporal features of the fusion tensor. The output of the LSTM module forms a composite feature tensor, which is then input into the decoder.

[0061] The decoder uses deconvolution to progressively recover image details and incorporates skip connections to preserve low-level feature information. Finally, the decoder outputs reconstructed motion and still video frames with dimensions corresponding to the input data. Experimental results show that this motion and still video reconstruction network can effectively reconstruct high-quality motion and still video frames and accurately extract dynamic foreground and static background information.

[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0063] like Figure 2 As shown in the figure, this embodiment provides a method for simultaneous reconstruction of dynamic and static scenes based on an event camera. The specific implementation steps are as follows:

[0064] (1) Setting up a hybrid camera system: This embodiment uses a hybrid event-frame camera system, which includes a traditional RGB camera (AlliedVision Alvium1800 U-240c) and an event camera (Sony IMX636). The two cameras are aligned with the field of view through a beam splitter (the light ratio between the event camera and the frame camera is 1:9) to prevent the event pixels from being exposed to excessive light, thereby ensuring that a clear image can be reconstructed from static events.

[0065] (2) Event camera parameter setting and data acquisition: Based on experimental evaluation, the ON and OFF thresholds of the event camera were set to -17 and -50, respectively, to ensure that a sufficient number of static events are generated for image reconstruction.

[0066] (3) Imaging strategies adapted to different brightness scenes: When shooting static scenes with different brightness levels, it is necessary to optimize the static response range of the event camera. For low-light conditions, the aperture should be increased to increase the number of incident photons, so as to avoid insufficient contrast in the number of triggered events due to low scene brightness, which would lead to excessive noise and loss of detail in the reconstructed image. For high-light conditions, the aperture should be reduced to decrease the number of incident photons, so that the scene light intensity falls within the effective range of the static response of the event camera, thereby successfully achieving static image reconstruction.

[0067] (4) Comparison with traditional event-based reconstruction methods: The present invention qualitatively and quantitatively compares the proposed method with mainstream reconstruction methods on the proposed synthetic dataset and real-shot data. Compared with traditional methods that only reconstruct the moving foreground, the method of the present invention reconstructs both the moving foreground and the static background, thus completing the scene information.

[0068] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for simultaneous reconstruction of dynamic and static scenes based on an event camera, characterized in that, Includes the following steps: Step 1: Separate dynamic and static events; The original event stream is divided into a spatiotemporal grid. An event quantity threshold is set. Events exceeding the threshold within the spatiotemporal grid are judged as dynamic events triggered by motion, while events not exceeding the threshold are judged as static events triggered by the background. Step 2: Static background reconstruction; The static event is first processed by convolution integral method to calculate the initial static reconstructed image, and then further optimized by static reconstruction and denoising network to obtain static background image; The specific implementation process of the convolution integral method is as follows: the convolution kernel slides through the entire pixel plane, calculates the number of events in each convolution kernel region, and assigns the statistical value of the number of events to the center pixel to obtain the initial static reconstructed image. The static reconstruction and denoising network includes an input layer, an encoding layer, an intermediate layer, a decoding layer, and a prediction layer. The encoding layer consists of multiple custom SRD-Blocks and adopts a U-Net structure and a non-linear activation design. The encoded features are fed into multiple SRD-Blocks without activation functions. A channel attention mechanism is used to enhance the feature representation ability between channels. During decoding, deconvolution and skip connections are used to restore image details. Step 3: Integration of static and dynamic events; The dynamic event is first divided into a voxel grid through a spatiotemporal window, and then the static background image and the voxel grid are fused together with the event label tensor to obtain a fused tensor. Step 4: Video reconstruction; The fusion tensor is input into the static video reconstruction network to obtain dynamic video with static background.

2. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that, The time span of the spatiotemporal grid is a fixed time step, and the spatial size is a fixed number of pixels.

3. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that, The size of the voxel grid is consistent with that of the static background image, i.e., both are pixel-level resolution.

4. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that, The event label tensor is used to identify the static / dynamic attributes of each event.

5. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that, The video reconstruction network uses a convolutional neural network with a U-Net structure.

6. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 5, characterized in that, In the convolutional neural network, the encoder integrates an LSTM module to extract global temporal features and calculate the feature map of the continuous fused tensor; the intermediate layer keeps the feature map size unchanged and only performs depth feature extraction; the decoder restores high-quality dynamic continuous frames with background, presenting dynamic foreground while retaining static background information.

7. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that, The static reconstruction and denoising network was trained on the real E-Static dataset.

8. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that, The static and dynamic video reconstruction network was trained on the E-StaDyn synthetic dataset.

Images