An event camera image reconstruction method based on a diffusion model

Abstract

This invention discloses an event camera image reconstruction method based on a diffusion model, belonging to the field of digital image signal processing technology. Currently, deep learning-based dense reconstruction algorithms for event camera images suffer from poor performance and instability due to the motion dependence of event data and the presence of considerable noise and non-ideal results. The method described in this invention introduces a mature diffusion model, using the generative effect of the generative model to guide image reconstruction, thereby compensating for insufficient event information and unstable reconstruction results. Furthermore, this invention bridges the gap between event camera-based image reconstruction and diffusion models, allowing subsequent diffusion-based tasks such as super-resolution and colorization to be easily applied to event camera image reconstruction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of digital image signal processing, and particularly relates to an event camera image reconstruction method based on a diffusion model. BACKGROUND

[0002] An event camera is a new type of visual sensor, also known as a dynamic visual sensor or DAVIS (Dynamic and Active-Pixel Vision Sensor). It is inspired by the biological visual system and takes "only sensing moving objects" as the starting point. Through an asynchronous and independent imaging paradigm, it realizes unique characteristics such as high temporal resolution, high dynamic range and low power consumption, and successfully solves the problems of spatial redundancy and dynamic blur of ordinary cameras. Therefore, it is widely used and performs well in high-speed motion estimation, high dynamic range mapping, feature detection and tracking, etc.

[0003] Unlike traditional cameras that accumulate photons through exposure to form images, each pixel of an event camera is equipped with an independent photoelectric sensing module. When the brightness of the pixel changes beyond a preset threshold, a differential pulse signal (also known as event data) will be output. The event data is encoded as a four-element vector (x i ,y i ,t i ,p i ), where (x i ,y i ) represents the pixel coordinates, t i represents the trigger time, and p i represents the polarity of the brightness change. Due to the independent work of all pixels, the data output of the event camera is time-asynchronous and spatially sparse. This imaging paradigm reduces a large amount of redundant data and eliminates the time-based imaging concept in traditional cameras. However, due to its non-Euclidean data structure, existing image reconstruction algorithms have difficulty in accurately reconstructing event data. Therefore, it is necessary to design new algorithms that are specific to the spatiotemporal characteristics of event data.

[0004] Currently, the main processing methods for event data are mainly divided into two categories: event-by-event processing and event group processing. The event-by-event processing method is commonly used for tasks such as event denoising, feature extraction and image reconstruction filtering, and updates the system state in real time through differential calculation. The event group processing method accumulates the event data within a fixed event window to solve the problem of limited data carried by a single event. Common representation methods include event frames, volume grids and three-dimensional point sets, etc.

[0005] In recent years, deep learning has made significant breakthroughs in the field of image processing, with event camera image reconstruction methods based on deep learning frameworks achieving better performance than traditional methods. However, event data depends on motion and exhibits considerable noise and non-ideal effects, resulting in inconsistent and unsatisfactory performance that requires further improvement and optimization.

[0006] In view of this, the present invention proposes an event camera image reconstruction method based on a diffusion model. Summary of the Invention

[0007] The purpose of this invention is to propose an event camera image reconstruction method based on a diffusion model to solve the problems mentioned in the background art. In view of the defects that event data contains a lot of information and is easily affected by noise, the method uses the generative capability of a pre-trained generative diffusion model, i.e., the prior knowledge of image capture, to guide the reconstruction of event camera images, so that the images have good reconstruction details.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] An event camera image reconstruction method based on a diffusion model specifically includes the following steps:

[0010] S1. Event Stream Acquisition: Obtain event stream data of the target scene by capturing images of the scene to be observed using the event camera and the event camera simulation simulator;

[0011] S2. Event data preprocessing: Event data streams cannot be directly applied to commonly used components in image processing algorithms, such as CNN. This invention preprocesses the event data obtained in S1 by grouping events to represent the event data in the form of event frames.

[0012] S3. Reconstruction Network Design: Design a pre-trained Stable Diffusion model and a pre-trained event data feature extractor; the pre-trained Stable Diffusion model includes a pre-trained Stable Diffusion network part, and the pre-trained event data feature extractor includes a pre-trained event data feature extraction network part and a fusion part;

[0013] S4. DDIM Sampling Process Design: Based on event camera image reconstruction, design DDIM sampling hyperparameters, including sampling steps, step size, and unconditional generation guidance rate, to optimize the imaging results;

[0014] S5. Image Reconstruction Acquisition: The noisy image is input into the pre-trained autoencoder built into the pre-trained Stable Diffusion model for encoding. Then, the encoded latent variable representation, text guidance information, and event frames are input into the reconstruction network constructed in S3. The pre-trained event data feature extractor is used to extract event features and the event features are fitted into the pre-trained Stable Diffusion model through the connection part to generate the denoised image at the current time step.

[0015] S6. Reconstructed Image Generation: Based on the DDIM sampling process in S4, the reconstructed image acquisition process described in S5 is repeated until the number of sampling steps is reached to obtain the latent variable representation of the reconstructed image. Then, the reconstructed image is finally generated by decoding through a pre-trained autoencoder.

[0016] Preferably, S2 specifically includes the following:

[0017] The event interval ΔT = t k Event points within -t0 are encoded into event frames using linear interpolation, with each event point having a polarity p. i It is mapped to the two channels closest to it, as shown in the formula below:

[0018]

[0019] In the formula, E k p represents the encoding result of the event point within the ΔT interval; i Indicates the event point polarity; B represents the number of voxel mesh channels; t i A timestamp representing the point in time that occurred.

[0020] Preferably, the input to the pre-trained Stable Diffusion model is a noisy image after time-embedding and text guidance information, and the input to the pre-trained event data feature extractor is an event frame; the pre-trained event data feature extractor is pre-trained on an event-image reconstruction task, and its encoder part is taken; the pre-trained Stable Diffusion model and the connection part adopt the mature ControlNet structure.

[0021] Compared with existing technologies, this invention provides an event camera image reconstruction method based on a diffusion model, which has the following advantages:

[0022] This invention proposes an event camera image reconstruction method based on a diffusion model. When reconstructing event images, the impact of the information contained in the event being affected by motion and the brightness change attribute on the quality of the reconstructed image will be greatly reduced. The diffusion model applies its powerful generation capability to make up for the original appearance of the generated real image, greatly improving the generation quality. Moreover, replacing the pre-trained diffusion model can be well adapted to super-resolution, colorization and other tasks, truly achieving the adaptation of event processing and traditional image frame processing. Attached Figure Description

[0023] Figure 1 This is an overall flowchart of the event camera image reconstruction method based on the diffusion model proposed in this invention;

[0024] Figure 2 This is the flowchart for reconstructing the image mentioned in Embodiment 1 of the present invention;

[0025] Figure 3 This is an example diagram of the experimental results in Embodiment 1 of the present invention. Detailed Implementation

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0027] Example 1:

[0028] This invention proposes an event camera image reconstruction method based on a diffusion model, which specifically includes the following steps:

[0029] S1. Generating a Simulation Dataset. Considering the difficulty of collecting real-world datasets, this invention employs the ESIM simulator. This simulator combines a rendering engine and an event simulator, enabling dynamic and adaptive sampling of event data. Based on estimates of the real-world dataset, this invention sets the simulator's event triggering threshold between 0.2 and 0.5 to approximate the performance of the real dataset as closely as possible. Through this method, this invention can generate event data that closely resembles real-world scenarios, which can then be used for further research and analysis.

[0030] S2. Event Data Preprocessing. Considering both algorithm execution speed and reconstruction results, a method of grouping events is adopted to represent the event data in S1 as event frames. Specifically, the event interval ΔT = t... k Event points within -t0 are encoded into event frames using linear interpolation, with each event point having a polarity p. i It is mapped to the two channels closest to it, as shown in the formula below:

[0031]

[0032] In the formula, E k p represents the encoding result of the event point within the ΔT interval; i Indicates the event point polarity; B represents the number of voxel mesh channels, here B=5, t i This is the timestamp of the event point.

[0033] S3. Data Augmentation. To generate the simulated dataset, considering the difficulty of collecting real-world datasets, this invention employs the ESIM simulator. This simulator combines the functions of a rendering engine and an event simulator, enabling it to generate event data in a dynamically adaptive manner. To approximate the characteristics of real-world datasets as closely as possible, this invention sets the simulator's event triggering threshold within the range of 0.2 to 0.5, based on estimates of the real-world dataset. Through this method, this invention can generate event data similar to real-world scenarios, providing useful resources for subsequent research and analysis.

[0034] S4, such as Figure 1 As shown in the Event Frame Encoder Blocks section, this invention first feeds the input event frame V (size B×H×W) into a recurrent convolutional backbone network, which consists of a header and three recurrent convolutional blocks. The header is used to convert the input event voxel grid V into a feature map f0 (size C0×H×W) of the first scale. In this invention, C0 is set to 32.

[0035] Leveraging the temporal consistency between consecutive frames improves the video reconstruction of events in this invention. A ConvLSTM layer is used in each loop block, which utilizes previous states to enhance the temporal stability of the reconstruction. Furthermore, in each loop block, a convolutional layer (with a stride of 2) is applied to halve the spatial size of the features. Simultaneously, the number of channels doubles with increasing scale, i.e., C0 l = C0 × 2^I. Therefore, the three stacked cyclic blocks produce feature maps of three scales, which can be represented as:

[0036]

[0037] In the formula, This represents the output of the ConvLstm layer at time t.

[0038] S5, such as Figure 1As shown in the Stable Diffusion and ControlNet section, Stable Diffusion comprises four pre-trained parts: Autoencoder, Time Encoder, Prompt Encoder, and Unet. The mature Stable Diffusion control network ControlNet structure is chosen as the backbone for image reconstruction. The low-dimensional features (256*32*32) obtained from S4 are passed through zero-convolutional layers to obtain a feature vector f1' with the same dimension as the output feature of the first encoding block of Stable Diffusion. The first input is a full-noise image sampled from Gaussian noise, with the same dimension as the low-dimensional representation output of the pre-trained autoencoder. The input at the current time step is passed through the first encoding block of ControlNet to obtain f1. f1' + f1 is then fed into the low-dimensional encoding block. The output feature vector of each encoding block in ControlNet after zero-convolution is fed into Stable Diffusion and added to the corresponding feature vector to provide conditional control. Under this conditional control, as well as semantic and time-step control, the output is the noise prediction for the current time step.

[0039] S6. Image Reconstruction Acquisition. Please refer to [link / reference]. Figure 2 The algorithm framework operation flow for each time step is as shown in S5. In this invention, the standard DDIM generation method is adopted, where the hyperparameters of DDIM are selected as follows: sampling steps are 50, unconditional generation guidance rate is 9.0, and eta = 0.0. The input starts with pure Gaussian noise. At each time step, the current denoised input is used to generate a prediction of the input noise under the conditions of semantics, event frame, and time step. After denoising the input using the standard DDIM method, a new input is obtained by denoising the input with noise. The operation in step five continues. After all time steps are completed, the reconstructed image output is obtained. Stable Diffusion is generated in the latent space. Before the image is sent to the network, it is encoded into a low-dimensional representation by the official pre-trained autoencoder to obtain the reconstructed image. Similarly, the reconstructed image is obtained by decoding by the autoencoder.

[0040] S7. Loss Function: Given an image z0, the diffusion algorithm progressively adds noise to the image, producing a noisy image z. t , where t represents the number of times noise is added. When t is sufficiently large, the image approximates pure noise. Given a set of conditions, including time step t, text prompt c... t and task-specific conditions c f Image diffusion algorithm learns a network ε θ To predict the addition of z to the noisy image t Noise in the middle.

[0041]

[0042] In the formula, L represents the overall learning objective of the entire diffusion model. This learning objective can be directly used to fine-tune the diffusion model.

[0043] S8. Training Process: From the given data, this invention randomly selects a sample. Then, within the time range 1 to T, a time point t is randomly selected. The sample data and time t are passed as input to the Diffusion model. The Diffusion model samples random noise and adds it to the input sample, forming a new noise sample. Next, this noise sample and time t are passed as input to the Unet neural network model.

[0044] The Unet model generates a sinusoidal position code based on time t and combines it with the input sample. The Unet model predicts this added noise and returns it as output. Simultaneously, the Diffusion model calculates the loss between this noise sample and the previously sampled random noise. Then, this invention uses the L2 loss function to calculate the loss between the noise predicted by the Unet model and the previously sampled random noise, calculates the gradient based on the loss, and updates the weights of the Unet model.

[0045] Repeat the above steps until the Unet model is fully trained. Through this training process, the Unet model learns how to predict appropriate noise and how to combine location encoding with input samples to generate more accurate results.

[0046] This invention uses the PyTorch Lightning framework, with training hyperparameters set to a batch size of 4, a learning rate of 0.0005, and a training duration of 50 epochs.

[0047] Example 2:

[0048] Based on Example 1, but with the following differences:

[0049] This invention selects the HQF event domain dataset as the benchmark dataset and chooses the state-of-the-art comparison method E2VID trained on the dataset generated in the same way. E2VID significantly outperforms previous techniques in high-speed event stream video synthesis. In this experiment, both the method of this invention and the E2VID method were used to generate all the data on the HQF dataset. The generation effect of the method of this invention is significantly better than that of E2VID. Two challenging data streams, boxes and reflective materials, were selected in the illustration. Since the beginning of the data stream is generally unstable and produces poor generation results, the first three frames were used as the illustration. Figure 3It is evident that the method of this invention outperforms the E2VID method in terms of global structure, contrast, and image quality. In each set of images, the first row shows the reconstruction result of this invention, the second row shows the actual data, and the third row shows the E2VID reconstruction result. It utilizes powerful pre-trained image priors to compensate for insufficient event information and unstable reconstruction results, thus demonstrating superior performance.

[0050] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for event camera image reconstruction based on a diffusion model, characterized in that, Specifically comprising the following steps: S1, event stream acquisition: obtaining event stream data of a target scene by shooting the scene to be observed through an event camera and an event camera simulation simulator; S2, event data preprocessing: preprocessing the event data obtained in S1, adopting an event grouping processing method to represent the event data in the form of event frames; specifically comprising the following contents: The event interval ΔT= t k - t Event points within 0 are encoded into event frames using linear interpolation, with each event point having a polarity... p i It is mapped to the two channels closest to it, as shown in the formula below: wherein, E k represents the encoding result of the event point in the ΔT interval; p i represents the polarity of the event point; B represents the number of voxel grid channels; t i represents the timestamp of the event point; S3, reconstruction network design: designing a pre-trained Stable Diffusion model and a pre-trained event data feature extractor; the pre-trained Stable Diffusion model comprises a pre-trained Stable Diffusion network part, and the pre-trained event data feature extractor comprises a pre-trained event data feature extraction network part and a fusion part; S4, DDIM sampling process design: designing DDIM sampling hyperparameters based on event camera image reconstruction, the hyperparameters comprising sampling step number, step distance and unconditional generation guide rate; S5, reconstructed image acquisition: inputting a noise image into a pre-trained autoencoder of the pre-trained Stable Diffusion model for encoding, then inputting the encoded hidden variable representation, text guide information and event frame into the reconstruction network constructed in S3, extracting event features by using the pre-trained event data feature extractor and fitting the event features into the pre-trained Stable Diffusion model through the connection part to generate a denoised image at the current time step; S6, reconstructed image generation: based on the DDIM sampling process design in S4, repeating the reconstructed image acquisition process in S5 to reach the sampling step number to obtain the hidden variable representation of the reconstructed image, and finally generating a reconstructed image by decoding through the pre-trained autoencoder.

2. The diffusion model based event camera image reconstruction method of claim 1, wherein, The input of the pre-trained Stable Diffusion model is a noise image after time-embedding and text guide information, and the input of the pre-trained event data feature extractor is an event frame; the pre-trained event data feature extractor is pre-trained on an event-image reconstruction task, and the encoder part thereof is taken; the pre-trained Stable Diffusion model and the connection part adopt a mature Controlnet structure.

Images