Method for simultaneously reconstructing dynamic and static scenes based on event camera

Abstract

A method for simultaneously reconstructing a dynamic scene and a static scene based on an event camera belongs to the field of image processing, and comprises the following steps: dividing an original event stream into space-time grids, setting an event number threshold value, judging events exceeding the threshold value as dynamic events triggered by motion, and judging events not exceeding the threshold value as static events triggered by background; calculating an initial static reconstruction image of the static event through a convolution integral method, and further optimizing through a static reconstruction noise reduction network to obtain a static background image; dividing a dynamic event into voxel grids through a space-time window, and fusing the static background image and the voxel grids on the premise of introducing an event tag tensor to obtain a fusion tensor; and inputting the fusion tensor into a dynamic and static video reconstruction network to obtain a dynamic video with a static background. According to the method, the quality of the static reconstructed image is improved, the phenomenon of visual inconsistency caused by fusion after independent reconstruction is eliminated, and model overfitting caused by monotonous data set elements is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for simultaneously reconstructing dynamic and static scenes based on an event camera. Background Art

[0002] An event camera is a new type of neuromorphic vision sensor. Unlike traditional frame cameras, it does not capture global images at fixed time intervals. Instead, it asynchronously triggers event streams based on brightness changes. Specifically, each pixel in an event camera operates independently, generating events only when a change in local brightness is detected. This event stream has a high temporal resolution of microseconds and can effectively capture fast-moving scenes. Therefore, event cameras have significant advantages in high-speed vision tasks such as robotic perception, target tracking, and autonomous driving. However, due to the fundamental difference in the data format output by event cameras compared to traditional frame cameras, their application is limited by the adaptability of existing vision algorithms.

[0003] To fully leverage the advantages of event cameras while maintaining compatibility with traditional vision algorithms, researchers have recently proposed event-based video reconstruction methods. These methods utilize event streams to reconstruct grayscale images, enabling them to achieve visual representation capabilities similar to those of frame-based cameras. However, existing event-based video reconstruction methods primarily focus on recovering moving areas and often neglect the reconstruction of static backgrounds, resulting in a lack of background detail in the final video. In many practical applications, such as autonomous driving, robotic navigation, and industrial inspection, the lack of background information can compromise system stability and robustness.

[0004] Existing research on static scene reconstruction can be broadly divided into two categories: methods that rely on external devices and methods based purely on event data. The former typically utilizes external active illumination or specialized optical devices to enhance the event camera's perception of static scenes. However, the application of these methods is limited by specific hardware, making them difficult to scale in complex environments. Another category of methods attempts to break away from reliance on external devices and directly recover static scenes from event data. By constructing a statistical relationship between noise events and scene brightness, they achieve static scene reconstruction under constant illumination. However, these methods suffer from significant noise interference, resulting in low reconstruction quality. Furthermore, existing event-based video reconstruction methods have significant shortcomings when handling complex scenes with both dynamic foregrounds and static backgrounds. For example, recent research has combined static background information with dynamic video reconstructed using the E2VID (Events-to-Video Network) model, a neural network that reconstructs grayscale images from event data, through mask fusion. However, this approach is limited by the performance of the external neural network and is prone to artifacts at mask boundaries.

[0005] Existing video reconstruction methods based on event data have difficulty in simultaneously restoring static backgrounds and dynamic targets. The main challenges include:

[0006] 1. Limitations of the event triggering mechanism: Event cameras rely on brightness changes to trigger events. Therefore, in static scenes, the event triggering frequency is low, resulting in less static background information.

[0007] 2. Noise interference problem: Due to hardware characteristics, event cameras are prone to generating noise events under low or high illumination conditions, which impairs the restoration quality of static scenes.

[0008] 3. Difficulty in dynamic-static fusion: Most existing methods use separate modeling to handle static and dynamic regions, and fail to achieve end-to-end unified reconstruction under the same framework.

[0009] The prior art closest to the present invention is Cao R, Galor D, Kohli A, et al. Noise2Image: noise-enabled static scene recovery for event cameras [J]. Optica, 2025, 12 (1): 46-55. This document proposes a statistical noise model to describe how noise event generation is associated with scene intensity. Due to the logarithmic sensitivity of the sensor, the number of events triggered by photon noise is mostly negatively correlated with the illumination level. By counting the number of noise events pixel by pixel, it is mapped to the scene intensity. In order to avoid ambiguity in the one-to-many mapping, a Noise2Image network model is proposed to recover the static part of the dynamic scene from the noise events alone. The specific implementation process is as follows:

[0010] (1) First, an event camera is used to shoot a static scene to capture static events triggered by noise. Then, the true noise event count is estimated based on the empirical noise event count measured by the event camera, and the positive and negative polarity of each event is recorded.

[0011] (2) A Noise2Image network model is proposed that receives input with two spatial channels, for positive event counts and negative event counts, and the network is trained to directly map the estimated event counts to the corresponding intensity images.

[0012] (3) The dynamic foreground of the scene is reconstructed into continuous frames through the E2VID network and fused with the background reconstructed by Noise2Image through image segmentation to obtain a dynamic video with a static background.

[0013] The technique disclosed in the aforementioned document fails to address the pixel extremum problem faced by pixel-by-pixel integration methods, resulting in low reconstructed image quality. Furthermore, the technique directly employs the E2VID model to reconstruct the dynamic foreground of a scene. The differences between the two reconstruction methods lead to visual inconsistencies between the static background and the moving foreground. Furthermore, the proposed dataset only contains human faces, lacking diversity. Summary of the Invention

[0014] To address the technical issues of existing methods, this paper proposes a method for simultaneous reconstruction of dynamic and static scenes based on an event camera. By establishing a unified modeling strategy for dynamic and static scenes, this method achieves high-precision reconstruction of event data. This method not only improves the reconstruction quality of static backgrounds but also effectively preserves the details of dynamic objects, making the reconstruction capabilities of event cameras even more practical.

[0015] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0016] The present invention provides a method for simultaneously reconstructing dynamic and static scenes based on an event camera, comprising the following steps:

[0017] Step 1: Separate dynamic and static events;

[0018] The original event stream is divided into a spatiotemporal grid, and a threshold for the number of events is set. Events exceeding the threshold in the spatiotemporal grid are determined as dynamic events triggered by motion, and events below the threshold are determined as static events triggered by background.

[0019] Step 2: Static background reconstruction;

[0020] The static event is first subjected to the convolution integral method to calculate the initial static reconstructed image, and then further optimized through the static reconstruction denoising network to obtain the static background image;

[0021] Step 3: Fusion of dynamic and static events;

[0022] The dynamic event is first divided into voxel grids through the spatiotemporal window, and then the static background image and the voxel grid are fused to obtain the fused tensor under the premise of introducing the event label tensor;

[0023] Step 4: Video reconstruction;

[0024] The fused tensor is input into the dynamic and static video reconstruction network to obtain a dynamic video with a static background.

[0025] Furthermore, the time span of the space-time grid is a fixed time step, and the spatial size is a fixed number of pixels.

[0026] Furthermore, the specific implementation process of the convolution integral method is: use the convolution kernel to slide across the entire pixel plane, calculate the number of events in each convolution kernel area, and assign the central pixel with the statistical value of the number of events to obtain the initial static reconstructed image.

[0027] Furthermore, the static reconstruction denoising network includes an input layer, an encoding layer, an intermediate layer, a decoding layer and a prediction layer; the encoding layer is composed of multiple customized SRD-Blocks, and adopts a U-Net structure and a non-linear activation design. The encoded features enter multiple SRD-Blocks without activation functions, and the channel attention mechanism is used to enhance the feature expression ability between channels. Deconvolution and jump connections are used during decoding to restore image details.

[0028] Furthermore, the size of the voxel grid is consistent with that of the static background image, that is, both have pixel-level resolution.

[0029] Furthermore, the event label tensor is used to identify the dynamic and static attributes of each event.

[0030] Furthermore, the static and dynamic video reconstruction network adopts a convolutional neural network with a U-Net structure.

[0031] Furthermore, in the convolutional neural network, the encoder integrates an LSTM module to extract global time domain features and calculate the feature map of the continuous fusion tensor; the intermediate layer keeps the feature map size unchanged and only performs deep feature extraction; the decoder restores high-quality dynamic continuous frames with background, presenting the dynamic foreground while retaining the static background information.

[0032] Furthermore, the static reconstruction denoising network is trained on the E-Static real dataset.

[0033] Furthermore, the static and dynamic video reconstruction network is trained on the E-StaDyn synthetic dataset.

[0034] The beneficial effects of the present invention are:

[0035] (1) The present invention adopts a two-step reconstruction strategy of convolution integral method + static reconstruction denoising network, which solves the pixel extremum problem and improves the quality of static reconstructed images. Among them, the static scene image reconstructed by the convolution integral method greatly improves the quality and information richness of the initial reconstruction, and provides a learnable sample for subsequent denoising operations. The present invention uses a static reconstruction denoising network to effectively learn the noise response model of the static scene, thereby performing targeted denoising on the static reconstructed image to obtain a high-quality static background image.

[0036] (2) The present invention adopts an end-to-end reconstruction method. After the event data is input into a unified framework, a dynamic video with a static background can be directly obtained, eliminating the visual inconsistency caused by separate reconstruction and then fusion.

[0037] (3) The dataset used in the present invention contains more than 200 different static scenes both indoors and outdoors, all of which are actually captured by cameras, thus avoiding model overfitting caused by the monotony of dataset elements.

[0038] (4) The present invention constructs a unified framework for simultaneously reconstructing static background and moving foreground from event data, which can reconstruct static background and moving foreground at the same time, solves the problem of background information loss in traditional video reconstruction technology, and achieves the goal of capturing real scenes using a single event camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a method for simultaneously reconstructing dynamic and static scenes based on an event camera provided by the present invention.

[0040] Figure 2 A flowchart of a method for simultaneously reconstructing dynamic and static scenes based on an event camera is provided in an embodiment. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below with reference to the accompanying drawings.

[0042] The present invention provides a method for simultaneous reconstruction of static and dynamic scenes based on event cameras. First, a unified framework for simultaneous reconstruction of static background and moving foreground from event data (URSEE) is constructed. Then, four core steps are used to achieve simultaneous reconstruction of static and dynamic scenes. Figure 1 As shown in the figure, the overall technical solution of the present invention mainly includes four steps: dynamic and static event separation, static background reconstruction, dynamic and static event fusion, and video reconstruction. Each step is implemented through different modules, as follows:

[0043] Step 1: Separate dynamic and static events;

[0044] The raw event data is first divided into spatiotemporal grids, each with a fixed time step (e.g., 10ms) and a fixed number of pixels (e.g., 20×20 pixels). Each spatiotemporal grid contains a different number of events. When the number of events exceeds a preset threshold, the events in that spatiotemporal grid are determined to be "dynamic events" triggered by motion; otherwise, the events in that spatiotemporal grid are classified as "static events" triggered by the background. Through the dynamic and static event separation process, the raw event stream is effectively separated into pure static event streams and dynamic event streams, which are input into different reconstruction branches to achieve targeted reconstruction processing.

[0045] Step 2: Static background reconstruction;

[0046] The static event stream extracted in step 1 is first subjected to the convolution integral method to calculate the initial static reconstructed image. Specifically, a 3×3 convolution kernel is slid across the entire pixel plane, the number of events within each convolution kernel region is counted, and this statistical value is assigned to the center pixel (Padding = 1, Step = 1). The resulting initial static reconstructed image is then further optimized using a static reconstruction denoising network to obtain a higher-quality static background image.

[0047] Step 3: Fusion of dynamic and static events;

[0048] The dynamic event flow extracted in step 1 is divided into a voxel grid through a spatiotemporal window. The size of the voxel grid is consistent with the reconstructed static background image, that is, both have pixel-level resolution. Therefore, the two can be directly fused to form a fusion tensor. This fusion tensor not only contains static background information, but also retains the spatiotemporal characteristics of dynamic events due to the characteristics of the voxel grid. In order to further accurately distinguish between the moving foreground and the static background, an event label tensor is additionally introduced in the fusion process to identify the dynamic and static attributes of each event. The final fusion tensor consists of three parts: the static background image, the dynamic event voxel grid and the event label tensor, and serves as the input of the subsequent dynamic and static video reconstruction network.

[0049] Step 4: Video reconstruction;

[0050] A convolutional neural network with a U-Net structure is used as the static and dynamic video reconstruction network. The fused tensor is input into the convolutional neural network with a U-Net structure. In this convolutional neural network, the encoder integrates an LSTM module to extract global time domain features and calculate the feature map of the continuous fusion tensor; the intermediate layer keeps the feature map size unchanged and only performs deep feature extraction; finally, the decoder restores high-quality dynamic continuous frames with background, ensuring that both static background information is retained and dynamic foreground can be accurately presented.

[0051] This invention mainly includes two neural network models: a static reconstruction denoising network and a static-dynamic video reconstruction network, which are trained on the E-Static real dataset and the E-StaDyn synthetic dataset respectively. The specific training process is as follows:

[0052] (1) E-Static real data set;

[0053] E-Static is a dataset collected from real-world static scenes, covering a variety of indoor and outdoor environments. Data acquisition was accomplished using a hybrid event-frame camera system consisting of a traditional RGB camera (Alvium1800 U-240c) and an event camera (Sony IMX636). The two cameras' fields of view were aligned using a beam splitter with a 1:9 (event camera:frame camera) light ratio, effectively avoiding overexposure of event pixels due to excessive illumination and thus improving the reconstructability of static event images. Based on empirical parameter settings, the event camera's ON threshold was set to -17 and its OFF threshold to -50 to ensure a sufficient number of static events for reconstruction. The E-Static real-world dataset contains a total of 200 sets of raw event streams and their corresponding high-quality frame images as ground truth data. For training convenience, the original 1280×720 resolution images were divided into six sub-images of 512×512 each, resulting in 1290 training samples and 96 test samples.

[0054] (2) E-StaDyn synthetic dataset;

[0055] E-StaDyn is a synthetic dataset designed specifically for the task of reconstructing static and dynamic events. It contains a total of 130 scenes, each with a unique static background and dynamic foreground. The data is generated by pairing multiple high-quality static images with multiple 3D models exhibiting random motion and rendering them using Blender software. Each scene renders 600 consecutive frames of images as ground truth frames. Subsequently, the rendered frames are event-processed using the DVS-Voltmeter simulator to generate the corresponding synthetic event stream. The final dataset is divided into 115 scenes for training and 15 scenes for testing, covering a variety of static and dynamic combinations and event characteristics, providing sufficient support for the training of static and dynamic fusion and static and dynamic video reconstruction networks.

[0056] (3) The training process of the neural network;

[0057] S1: For the static reconstruction denoising network, the original event stream of each scene in the E-Static real dataset is reconstructed into an initial image through the convolution integral method. The initial image contains both the scene grayscale information and the noise information, which serves as the input of the static reconstruction denoising network.

[0058] S2: The entire static reconstruction denoising network consists of an input layer, encoding layer, intermediate layer, decoding layer, and prediction layer. The encoding layer consists of multiple custom SRD-Blocks and uses a U-Net (a "U"-shaped encoder-decoder with a cross-connection structure) architecture and a non-nonlinear activation design to reduce computational complexity and improve inference speed. The calculation of each SRD-Block is as follows: in represents the convolution kernel with a window size of w, d represents the number of channels of the output tensor, Denotes a convolution operation, x represents the input tensor or the output of the previous feature convolution layer, b represents the bias term, IN(·) is a batch normalization operation that normalizes the existing tensor, SGU (Simple Gated Unit) is used for feature selection, and Y represents the encoded features. The encoded features are fed into multiple SRD-Blocks without activation functions. A channel-wise attention mechanism is used to enhance the inter-channel feature representation capability. The calculation is as follows: Y = X·σ(W1(W0X)), where X represents the input feature tensor, σ represents the Sigmoid activation function, and W1 and W0 represent fully connected layers. This channel-wise attention mechanism enables the model to adaptively focus on important feature channels.

[0059] S3: During the decoding process, deconvolution and skip connections are used to restore image details. The output of the decoder is a reconstructed image of the same size as the input. The mean square error (MSE) is used as the only loss function during training, and the loss function is defined as follows: where X i represents the reference target of the input image, represents the network output image, and N represents the total number of pixels in the image. During optimization, the Adam optimizer was used to backpropagate the network parameters, with a learning rate set to 0.0001. After multiple rounds of iterations on the training set, the network converged stably and effectively completed the image restoration task.

[0060] S4: The motion / static video reconstruction network has a similar structure to the static reconstruction / denoising network. It is trained using the E-StaDyn synthetic dataset and validated on real data. For each scene in the E-StaDyn synthetic dataset, a synthetic event stream is generated using the DVS-Voltmeter simulator. This event stream is then divided into 40 continuous voxel grids and fed into the motion / static video reconstruction network. The LSTM module in the encoder extracts temporal features from the fused tensor. The output of the LSTM module forms a composite feature tensor, which is then fed into the decoder.

[0061] The decoder uses deconvolution to gradually restore image details and combines it with skip connections to preserve low-level feature information. Ultimately, the decoder outputs reconstructed motion and still video frames, whose dimensions correspond to the input data. Experimental results demonstrate that the proposed motion and still video reconstruction network can effectively reconstruct high-quality motion and still video frames, accurately extracting dynamic foreground and static background information.

[0062] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0063] like Figure 2 As shown, this embodiment provides a method for simultaneously reconstructing dynamic and static scenes based on an event camera, and its specific implementation steps are as follows:

[0064] (1) Building a hybrid camera system: This example uses a hybrid event-frame camera system consisting of a traditional RGB camera (AlliedVision Alvium 1800 U-240c) and an event camera (Sony IMX636). The two cameras are aligned across the field of view via a beam splitter (with a 1:9 light ratio between the event camera and the frame camera) to prevent the event pixels from being overly illuminated, thereby ensuring that a clear image can be reconstructed from static events.

[0065] (2) Event camera parameter setting and data acquisition: Based on experimental evaluation, the ON and OFF thresholds of the event camera are set to -17 and -50, respectively, to ensure that a sufficient number of static events are generated for image reconstruction.

[0066] (3) Imaging strategies adapted to scenes of varying brightness: When capturing static scenes of varying brightness, the static response range of the event camera needs to be optimized. For low-light conditions, the aperture should be increased to increase the number of incident photons. This avoids insufficient contrast in the number of events triggered due to low scene brightness, which in turn results in excessive noise and loss of detail in the reconstructed image. For high-light conditions, the aperture should be narrowed to reduce the number of incident photons, ensuring that the scene light intensity falls within the effective static response range of the event camera, thereby successfully achieving static image reconstruction.

[0067] (4) Comparison with traditional event-based reconstruction methods: The present invention qualitatively and quantitatively compares the proposed method with mainstream reconstruction methods on the proposed synthetic dataset and real-shot data. Compared with the traditional method that only reconstructs the moving foreground, the method of the present invention reconstructs the moving foreground and the static background at the same time, completing the scene information.

[0068] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for simultaneous reconstruction of dynamic and static scenes based on an event camera, characterized in that: The following steps are involved: Step 1: Separate dynamic and static events; The original event stream is divided into a spatiotemporal grid, and a threshold for the number of events is set. Events exceeding the threshold in the spatiotemporal grid are determined as dynamic events triggered by motion, and events below the threshold are determined as static events triggered by background. Step 2: Static background reconstruction; The static event is first subjected to the convolution integral method to calculate the initial static reconstructed image, and then further optimized through the static reconstruction denoising network to obtain the static background image; Step 3: Fusion of dynamic and static events; The dynamic event is first divided into voxel grids through the spatiotemporal window, and then the static background image and the voxel grid are fused to obtain the fused tensor under the premise of introducing the event label tensor; Step 4: Video reconstruction; The fused tensor is input into the dynamic and static video reconstruction network to obtain a dynamic video with a static background.

2. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The time span of the space-time grid is a fixed time step, and the spatial size is a fixed number of pixels.

3. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The specific implementation process of the convolution integral method is: using the convolution kernel to slide across the entire pixel plane, calculating the number of events in each convolution kernel area, and assigning the central pixel with the statistical value of the number of events to obtain the initial static reconstructed image.

4. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The static reconstruction denoising network includes an input layer, an encoding layer, an intermediate layer, a decoding layer and a prediction layer; the encoding layer is composed of multiple customized SRD-Blocks and adopts a U-Net structure and a non-nonlinear activation design. The encoded features enter multiple SRD-Blocks without activation functions, and a channel attention mechanism is used to enhance the feature expression ability between channels. Deconvolution and jump connections are used to restore image details during decoding.

5. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The size of the voxel grid is consistent with that of the static background image, that is, both have pixel-level resolution.

6. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The event label tensor is used to identify the dynamic and static attributes of each event.

7. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The dynamic and static video reconstruction network adopts a convolutional neural network with a U-Net structure.

8. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 7, characterized in that: In the convolutional neural network, the encoder integrates an LSTM module to extract global time-domain features and calculate the feature map of the continuous fusion tensor; the intermediate layer keeps the feature map size unchanged and only performs deep feature extraction; the decoder restores high-quality dynamic continuous frames with background, presenting the dynamic foreground while retaining the static background information.

9. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The static reconstruction denoising network is trained on the E-Static real dataset.

10. The method for simultaneous reconstruction of dynamic and static scenes based on an event camera according to claim 1, characterized in that: The static-motion video reconstruction network is trained on the E-StaDyn synthetic dataset.

Images