Event camera image reconstruction method and system based on forward and backward particle filter
By using the front-back particle filter method, image reconstruction model construction and noise modeling of the intensity frame of the event camera is solved, and the poor image quality problem of the event camera under overexposure and dark light conditions is achieved, and high-quality images are achieved Reconstruct effect.
Patent Information
- Application Number
- CN202411146518.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The intensity image quality captured by the event camera under overexposure and dark light conditions is poor, and there are motion blur and event noise problems with non-standard distribution, which affects the image reconstruction effect.
Using a method based on the front-back particle filter, the image reconstruction model is constructed on the intensity frame of the DAVIS event camera, the event and grayscale image information are fused, and the image reconstruction is carried out through the front-back particle filtering algorithm. Output high time resolution and high quality reconstructed images.
The image reconstruction quality is improved, and the poor image quality problem of event cameras under overexposure and dark light conditions is solved, achieving high temporal resolution and high-quality image reconstruction.
Smart Images

Figure CN118967514A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision image processing, and in particular to an event camera image reconstruction method and system based on forward and backward particle filters. Background Art
[0002] As a low-cost and easy-to-use sensor, vision sensors are widely used in the industry. Traditional vision sensors output absolute pixel intensity values based on a fixed frame rate, but have disadvantages such as motion blur, low dynamic range and latency under certain conditions. In order to solve the above shortcomings of traditional vision sensors, a silicon retinal sensor event camera is proposed. The event camera is different from the traditional CMOS image sensor (CIS), which responds to changes in pixel brightness asynchronously.
[0003] The output of an event camera is a series of discrete events that reflect the changes in the logarithmic intensity of pixel brightness. An event data contains the location of the pixel, the polarity of the pixel brightness change, and the timestamp. Since the event camera does not have a fixed exposure time, there is no motion blur and delay. At the same time, the event camera records the changes in pixel brightness, which is not affected by the absolute brightness of the pixel and has a higher dynamic range.
[0004] The output of the event camera is an asynchronous event stream, and the information contained in a single event is extremely limited. In order to give full play to the advantages of the event camera, decoding the event stream to reconstruct high-quality images is of great significance for subsequent feature detection, motion estimation and other processing. In the image reconstruction based on the dynamic vision sensor (DVS), Kim et al. proposed a high-quality scene reconstruction algorithm based on a two-stage filter. However, in the article, the author made a strict restriction that the camera can only perform pure rotational motion. In subsequent research, in order to design an image reconstruction algorithm for general motion scenes. Reinbacher et al. designed an algorithm to simultaneously restore the motion field and brightness image based on the method of minimizing the cost function. Kim et al. proposed to realize the simultaneous estimation of camera motion, depth and optical flow based on three decoupled probability density filters. Bardow et al. explicitly merged information about the asynchronous nature of events through the event manifold induced by the relative timestamps of events, thereby reconstructing intensity images of arbitrary frame rates.
[0005] DVS only outputs the change of pixel brightness and discards the absolute brightness value of the pixel. This makes the DVS event camera output less events when the scene is static or the camera moves slowly, and it is impossible to effectively reconstruct a good image. To this end, dynamic and active-pixel vision sensors (DAVIS) and asynchronous time-based image sensors (ATIS) are proposed. DAVIS can not only output events, but also output images based on a fixed frame rate like CIS. However, when there is noise in the event stream or frame data, the mismatch between the logarithmic intensity deviation of the pixel and the contrast threshold will make the reconstructed image poor and discontinuous. In order to better fuse the image grayscale value and event data, Scheerlink et al. used an asynchronous complementary filter to reconstruct images with high temporal resolution and high dynamic range. For the problem of event noise, three pixel-by-pixel additive Gaussian noises are considered, namely process noise, isolated pixel noise and refractory period noise. For the mismatch problem of contrast threshold, event pixels are divided into hot pixels, cold pixels, warm pixels and cool pixels, and a threshold calibration algorithm is designed. In view of the problem of motion blur in fixed frame rate output images, some event-based double integral (EDI) models and methods for minimizing cost function optimization are used to deal with motion blur. Recently, deep learning methods with large computational load are used to achieve event-based supervised or unsupervised high-quality image reconstruction. Although it has better reconstruction effect than traditional methods, the algorithm computational efficiency and model generalization ability need to be improved. Summary of the invention
[0006] Purpose of the invention: In order to solve the problem that intensity images taken by event cameras have poor image quality under overexposure and dim light conditions, the present invention provides an event camera image reconstruction method based on forward and backward particle filters. By modeling and analyzing motion blur and non-normally distributed event noise in low-frequency intensity frames, the algorithm has a relatively better image reconstruction effect, solving the problem that intensity images taken by event cameras have poor image quality under overexposure and dim light conditions.
[0007] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is: An event camera image reconstruction method based on a forward and backward particle filter comprises the following steps: Step S1, constructing an image reconstruction model based on the DAVIS event camera for possible motion blur problems of intensity frames obtained by the DAVIS event camera based on a fixed exposure time.
[0008] Step S2, according to the image reconstruction model based on the DAVIS event camera, the information from the event and grayscale images are fused, and the intensity frame is modeled as a continuous time process to obtain a continuous time system equation.
[0009] Step S3, discretizing the continuous-time system equations so as to perform image reconstruction estimation through filters and model non-Gaussian event noise.
[0010] Step S4, combining the obtained discretized image and the event with noise with the image frame through the forward and backward particle filtering algorithm, and outputting a reconstructed image with high time resolution and high quality.
[0011] Preferred: The image reconstruction model based on the DAVIS event camera is: in, express Moment Pixel The logarithmic intensity value, Indicates The time when the picture was obtained. Indicates the exposure time, express Moment in Pixels The pixel intensity value, express Moment Pixel The potential clear image gray value is at, c represents the event threshold, express Moment Pixel The events at the place.
[0012] Preferably: the method of constructing the image reconstruction model based on the DAVIS event camera in step S1 is as follows: Step S11, an event is represented as: in, Indicates an event, , t represents the time when the event is triggered, Indicates the pixel coordinates where the event was triggered. Indicates the polarity of the event.
[0013] If we use a continuous time system to describe the event process, then in pixels An event is defined as: in, Indicates time The polarity of moment events, Represents the increase in pixel logarithmic grayscale value, represents the decrease of logarithmic grayscale value, c represents the event threshold, is the Dirac function.
[0014] Step S12, integrating each event cycle to obtain: in, represents the event period integral, Indicates time The polarity of the event at the moment, c represents the event threshold, is the Dirac function.
[0015] Step S13, for discrete intensity frames based on a fixed frame rate provided by the DAVIS event camera, convert the pixel intensity values into logarithmic intensity values, and convert the pixel grayscale values into logarithmic intensity values. Add a fixed bias on the basis of have to: in, represents the logarithmic intensity value, express Moment in Pixels The pixel intensity value, Indicates fixed paranoia.
[0016] Step S14: According to the imaging principle, the pixel intensity value of motion blur is understood as the average of the real pixel intensity at each moment during the exposure time. Therefore, the pixel intensity of one frame is expressed as: in, Indicates The pixel intensity of the image, Indicates the exposure time, Indicates The time when the image was obtained.
[0017] Step S15, the blurred image is processed, and the relationship between the image frame and the event is established in combination with the imaging mechanism analysis of the above image frame as follows: but: in, express Moment Pixel The logarithmic intensity value, Indicates The time when the picture was obtained. Indicates the exposure time, express Moment in Pixels The pixel intensity value, express Moment Pixel The potential clear image gray value is at, c represents the event threshold, express Moment Pixel The events at the place.
[0018] Preferably: Step S2 fuses the information from the event and the grayscale image, and the specific steps of modeling the intensity frame as a continuous time process are: Step S21, according to the time information, the pixel intensity of the corresponding event is obtained by using an inter-frame interpolation algorithm, and the pixel intensity information of the inter-frame event time information is interpolated: in, express Moment Pixel Logarithmic intensity value.
[0019] Step S22, the events are described using the zero-order holder method: in, express Moment Pixel The logarithmic intensity value, express Moment Pixel Logarithmic intensity value.
[0020] Preferably: the specific steps of step S3 of discretizing the continuous-time system equation so as to be able to perform image reconstruction estimation through a filter and model the non-Gaussian event noise are: Step S31, the continuous-time system model of DAVIS camera image reconstruction is expressed as: in, Represents pixels The observed value of the logarithmic grayscale at . represents event noise. Represents the measurement noise of pixel grayscale.
[0021] Step S32: Since the event is modeled as a continuous Dirac function, integration is performed to obtain: in, express Moment Pixel The grayscale logarithm of the point.
[0022] Step S33, the system model of DAVIS camera image reconstruction is expressed as: in, express Moment Pixel The grayscale logarithm of the point, express Moment Pixel The event cycle integral of a point, express Moment Pixel The event noise of the point.
[0023] Step S34, modeling event noise and observation noise as: in, express Moment Pixel The event noise of the point, Indicates the instability of event thresholds, represents the event threshold, It means it obeys normal distribution. represents the variance, represents the observation noise.
[0024] Preferably, in step S4, the method of combining events and image frames by forward and backward particle filtering algorithms to output a reconstructed image with high temporal resolution and high quality is as follows: a particle smoother is introduced into the image reconstruction of the event camera, and the particle smoother includes a forward particle estimator and a backward smoother. The forward particle estimator is used to calculate the particle weight of . The forward particle estimator is used to calculate the particle weight of .
[0025] Preferably: the method used by the forward particle estimator to calculate the particle weight is: The particle set sampled in the prediction stage is based on the prior probability density is generated: in, express The sampling particle set at time, represents the particle sampling function, represents the prior probability density, Indicates the number of sampled particles.
[0026] Calculate the prior state estimate for each particle based on the state equation: in, express The prior state estimate at time t, represents the forecast covariance, Indicates the event threshold.
[0027] The prior estimates of the observed variables are expressed as: in, represents the prior estimate of the observed variable.
[0028] In the update phase, the weights of the sampled particles are calculated: in, is the weight of the sampled particle, represents the likelihood function, Indicates The sampled particles are in pixels The observed value of the logarithmic grayscale at , express The prior state estimate at time t, is the probability density transfer function, is the importance sampling density.
[0029] Preferably, the method for solving the particle weight of the smoother based on the particle weight calculated by the forward particle estimator of the backward smoother is: The weight of the particles is based on the following probability density function: in, represents the probability density function, Indicates the current state. represents a posteriori knowledge, Indicates the observation time.
[0030] Based on the above formula, the weight formula of particle smoothing is: in, represents the particle weight of the smoothing process, express The weight of the moment, express The sampling particle set at time, express The weight of the moment.
[0031] Normalizing the weights yields: in, represents the normalized weight, Represents the particle weight of the smoothing process.
[0032] Then the FBPF estimate of the pixel logarithm is expressed as: in, represents the FBPF estimate of the pixel logarithm, represents the normalized weight, express The prior state estimate at time t.
[0033] Another object of the present invention is to provide an event camera image reconstruction system based on a forward and backward particle filter, which is used to implement the event camera image reconstruction method based on the forward and backward particle filter, including an image reconstruction model unit based on a DAVIS event camera, a continuous-time system equation unit, a discretized image event noise model unit, and a particle filtering algorithm unit, wherein: The DAVIS event camera-based image reconstruction model unit is used to solve the possible motion blur problem of intensity frames obtained by the DAVIS event camera based on a fixed exposure time, and to construct an image reconstruction model based on the DAVIS event camera.
[0034] The continuous-time system equation unit is used to fuse information from events and grayscale images according to an image reconstruction model based on a DAVIS event camera, and to model the intensity frame as a continuous-time process to obtain a continuous-time system equation.
[0035] The discretized image event noise model unit is used to discretize the continuous-time system equation so that image reconstruction estimation can be performed through a filter and non-Gaussian event noise can be modeled.
[0036] The particle filter algorithm unit is used to combine events and image frames according to the obtained discretized images and events with noise through the forward and backward particle filter algorithms, and output a reconstructed image with high time resolution and high quality.
[0037] Another object of the present invention is to provide a computer system, comprising a memory and a processor, wherein the memory is used to store a computer program / instruction, and the processor is used to execute the computer program / instruction to implement the event camera image reconstruction method based on the forward and backward particle filter.
[0038] Compared with the prior art, the present invention has the following beneficial effects: (1) Events are modeled as continuous-time Dirac functions and intensity frames are modeled as continuous-time processes, which improves the quality of image reconstruction.
[0039] (2) In order to model image frames more accurately, we first assume that the pixel intensity between frames changes linearly. Considering the temporal information of events between frames, we use the inter-frame interpolation algorithm to obtain the pixel intensity of the corresponding event based on the temporal information. Finally, we use the forward and backward particle filter (FBPF) algorithm to combine events and image frames to output a reconstructed image with high temporal resolution and high quality.
[0040] (3) We successfully apply our method to evaluation experiments on a public event camera dataset. Compared with the direct integration (DI) method, Kalman filter (KF) method and complementary filter (CF) method, the proposed method can achieve high-quality image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of the event camera image reconstruction method based on forward and backward particle filter.
[0042] Figure 2 This is the flow chart of the DAVIS event camera image reconstruction algorithm based on FBPT.
[0043] Figure 3 It is a schematic diagram of event generation in event camera.
[0044] Figure 4 It is the information output of the DAVIS event camera.
[0045] Figure 5 It is based on the application of FBPT image reconstruction algorithm in ACD dataset.
[0046] Figure 6 These are the reconstruction results of four typical scenes in the IJRR dataset based on different image reconstruction algorithms. DETAILED DESCRIPTION
[0047] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
[0048] An event camera image reconstruction method based on forward and backward particle filters, such as Figure 1 As shown, the following steps are included: Step S1, constructing an image reconstruction model based on the DAVIS event camera for possible motion blur problems of intensity frames obtained by the DAVIS event camera based on a fixed exposure time.
[0049] like Figure 2 As shown, in step S1, a method for constructing an image reconstruction model based on a DAVIS event camera is used to solve the possible motion blur problem of intensity frames obtained based on a fixed exposure time: Step S11, DAVIS event camera is a new type of visual sensor that combines traditional CIS sensor and DVS in the same pixel. They share the same photodiode, and can generate pixel image information on demand at a fixed frame rate while outputting asynchronous event streams through a distributed switch. The event camera outputs an asynchronous event stream based on the change in the logarithmic brightness of the pixel. The event trigger threshold c is set by the comparison circuit inside the event camera. When the logarithmic brightness change of the pixel reaches +c, a positive event will be triggered, and when the change reaches -c, a negative event will be triggered.
[0050] An event can be represented as: (1) in, Indicates an event, Indicates the event sequence number. , t represents the time when the event is triggered. Indicates the pixel coordinates where the event was triggered. Indicates the polarity of the event.
[0051] The DAVIS event camera can simultaneously output DVS events and grayscale image frames based on a fixed frame rate. When the logarithmic intensity change of the pixel gray value reaches a threshold, an event will be triggered. Since the occurrence time of each event is random, we use a continuous time system to describe the event process. An event can be modeled as a Dirac function We are in Pixel An event is defined as: (2) in, Indicates time The polarity of moment events. Represents the increase in pixel logarithmic grayscale value, represents the logarithmic grayscale value decrease. The constant c represents the event threshold.
[0052] In step S12, we integrate each event cycle to obtain: (3) in, represents the event period integral, Indicates time The polarity of the event at the moment, c represents the event threshold, is the Dirac function.
[0053] Step S13, for the discrete intensity frames based on a fixed frame rate provided by the DAVIS event camera, let express Moment in Pixels Since the event camera generates events through the logarithmic intensity change of pixel brightness, we convert the pixel intensity value into a logarithmic intensity value. The value range of the traditional image frame pixel intensity is .when hour, will not exist. To avoid this problem, we use the pixel gray value Add a fixed bias on the basis of have to: (4) in, represents the logarithmic intensity value, express Moment in Pixels The pixel intensity value, Indicates fixed paranoia.
[0054] The logarithmic function is a nonlinear function. When the logarithmic value of the image intensity reaches the threshold value c, the change in the corresponding pixel grayscale value is not fixed. For the same change in threshold value c, when the initial grayscale value of the pixel is larger, the corresponding intensity change is also larger.
[0055] Step S14: Traditional CIS often uses a fixed exposure time when shooting, which may cause motion blur in the captured image. According to the imaging principle, the pixel intensity value of motion blur can be understood as the average of the real pixel intensity at each moment during the exposure time. Therefore, the pixel intensity of a frame can be expressed as: (5) in, Indicates The pixel intensity of the image, Indicates the exposure time, Indicates The time when the image was obtained.
[0056] Step S15: When an object in the shooting scene moves quickly, an event will also be triggered during the exposure time. A fast-moving object will cause motion blur in the captured image frame, and directly using it for image reconstruction will affect the reconstruction quality.
[0057] To solve this problem, we will process the blurred image. Combining the imaging mechanism analysis of the above image frames, we can establish the relationship between image frames and events as follows: but: (6) in, express Moment Pixel The logarithmic intensity value, Indicates The time when the picture was obtained. Indicates the exposure time, express Moment in Pixels The pixel intensity value, express Moment Pixel The potential clear image gray value is at, c represents the event threshold, express Moment Pixel The events at the place.
[0058] Step S2, according to the image reconstruction model based on the DAVIS event camera, the information from the event and grayscale images are fused, and the intensity frame is modeled as a continuous time process to obtain a continuous time system equation.
[0059] like Figure 3 and 4 As shown in Figure 1, the specific steps of fusing the information from the event and grayscale images and modeling the intensity frame as a continuous time process are as follows: Step S21, since the information frequency of events and grayscale images does not match, in order to fuse the information of the two, we also need to model the intensity frame as a continuous time process. Use the zero-order holder directly to model the inter-frame process. This modeling method completely ignores the changes in pixel grayscale reflected by inter-frame events, resulting in large modeling errors. In order to model image frames more accurately, we first assume that the pixel intensity changes linearly between frames. Considering the time information of inter-frame events, the pixel intensity of the corresponding event is obtained through the inter-frame interpolation algorithm based on the time information. The specific modeling process is: Interpolation of pixel intensity information combined with event time information between frames: (7) in, express Moment Pixel Logarithmic intensity value.
[0060] Step S22, the events are described using the zero-order holder method: (8) in, express Moment Pixel The logarithmic intensity value, express Moment Pixel Logarithmic intensity value.
[0061] Compared with directly using the zero-order holder method to model the inter-frame process, this method refines the changes in pixel intensity between frames and improves the accuracy of the pixel intensity value corresponding to the event. Therefore, using this method to model grayscale images will improve the quality of image reconstruction.
[0062] Step S3, discretizing the continuous-time system equations so as to perform image reconstruction estimation through filters and model non-Gaussian event noise.
[0063] The specific steps for discretizing the continuous-time system equations so that image reconstruction estimation can be performed through filters and modeling non-Gaussian event noise are: Step S31, the continuous-time system model of DAVIS camera image reconstruction can be expressed as: (9) (10) in, Represents pixels The observed value of the logarithmic grayscale at . represents event noise. Represents the measurement noise of pixel grayscale.
[0064] Step S32, since the event is modeled as a continuous Dirac function, we integrate equation (9) to obtain: (11) in, express Moment Pixel The grayscale logarithm of the point.
[0065] Step S33, combined with equation (10), the system model of DAVIS camera image reconstruction can be expressed as: in, express Moment Pixel The grayscale logarithm of the point, express Moment Pixel The event cycle integral of a point, express Moment Pixel The event noise of the point.
[0066] Step S34, in the event camera, event noise may be caused by the following factors: 1) The photons reflected in natural scenes are unstable. When the photons reflected by a pixel in a natural scene are uneven, this will cause the photosensitive device to sense unstable photons, thereby generating unstable events.
[0067] 2) Event Threshold The event in the event camera is generated when the change in pixel brightness reaches a certain percentage. The actual physical circuit design is similar to a comparison circuit. When the current in the circuit is unstable, it will cause the event trigger threshold to be unstable.
[0068] 3) Bus conflict. When there are objects moving too fast in the scene captured by the event camera, the rapid movement of the objects will trigger a large number of events in a short period of time. Due to the event response time limit and the transmission of events, the transmission bus may conflict, which may cause errors in generating events.
[0069] Through the above analysis of event noise, we model event noise and observation noise as follows: (12) (13) in, express Moment Pixel The event noise of the point, Indicates the instability of event thresholds, represents the event threshold, It means it follows normal distribution. represents the variance, represents observation noise, event noise The normal distribution corresponds to the instability of photons and event thresholds, and the uniform distribution corresponds to event errors caused by bus conflicts, etc. Describes the modeling error of the pixel gray value.
[0070] Step S4: The image processed above is combined with events and image frames through a forward and backward particle filter (FBPF) algorithm to output a reconstructed image with high temporal resolution and high quality.
[0071] The images processed above are combined with events and image frames through the forward and backward particle filter (FBPF) algorithm to output high temporal resolution and high-quality reconstructed images. The specific steps are as follows: In an event camera, changes in pixel brightness are recorded through discrete events. This processing method avoids data redundancy and improves data recording efficiency. However, the amount of information contained in a single event is extremely limited, and the state estimation of the particle filter is only related to its state at the previous moment. Such a state estimation algorithm is greatly affected by event noise. In order to improve the correlation between the state estimation results and the information of previous and subsequent events, we introduce a particle smoother into the image reconstruction of the event camera. The goal of the particle smoother is to calculate the edge smoothing distribution in Compared with the particle filter algorithm, the future measurement quantity Also used to estimate the state , so better image reconstruction accuracy can be obtained. The particle smoother is mainly composed of two parts, namely the forward particle estimator and the backward smoother. The forward particle estimator is used to calculate the particle weights. The forward particle estimator is used to calculate the particle weights.
[0072] The method used by the forward particle estimator to calculate particle weights is: The particle set sampled in the prediction stage is based on the prior probability density is generated: (14) in, express The sampling particle set at time, represents the particle sampling function, represents the prior probability density, Indicates the number of sampled particles.
[0073] Then we calculate the prior state estimate of each particle based on the state equation: (15) in, express The prior state estimate at time t, represents the forecast covariance, Indicates the event threshold.
[0074] The prior estimate of the observed variable can be expressed as: (16) in, represents the prior estimate of the observed variable.
[0075] In the update phase, we calculate the weights of the sampled particles: (17) in, is the weight of the sampled particle, represents the likelihood function, Indicates The sampled particles are in pixels The observed value of the logarithmic grayscale at , express The prior state estimate at time t, is the probability density transfer function, is the importance sampling density.
[0076] The method for solving the particle weight of the smoother based on the particle weight calculated by the forward particle estimator is: The main work of the backward smoother is to solve the particle weights of the smoother based on the particle weights calculated by the forward particle estimator. The weights of the particles are based on the following probability density function: in, represents the probability density function, Indicates the current state. represents a posteriori knowledge, Indicates the observation time.
[0077] Based on the above formula, the weight formula of particle smoothing is: in, represents the particle weight of the smoothing process, express The weight of the moment, express The sampling particle set at time, express The weight of the moment.
[0078] Normalizing the weights, we can get: in, represents the normalized weight, Represents the particle weight of the smoothing process.
[0079] Then the FBPF estimation of the pixel logarithm can be expressed as: in, represents the FBPF estimate of the pixel logarithm, represents the normalized weight, express The prior state estimate at time t.
[0080] In another embodiment, an event camera image reconstruction system based on a forward and backward particle filter is provided, which is used to implement the event camera image reconstruction method based on the forward and backward particle filter, including an image reconstruction model unit based on a DAVIS event camera, a continuous-time system equation unit, a discretized image event noise model unit, and a particle filter algorithm unit, wherein: The DAVIS event camera-based image reconstruction model unit is used to solve the possible motion blur problem of intensity frames obtained by the DAVIS event camera based on a fixed exposure time, and to construct an image reconstruction model based on the DAVIS event camera.
[0081] The continuous-time system equation unit is used to fuse information from events and grayscale images according to an image reconstruction model based on a DAVIS event camera, and to model the intensity frame as a continuous-time process to obtain a continuous-time system equation.
[0082] The discretized image event noise model unit is used to discretize the continuous-time system equation so that image reconstruction estimation can be performed through a filter and non-Gaussian event noise can be modeled.
[0083] The particle filter algorithm unit is used to combine events and image frames according to the obtained discretized images and events with noise through the forward and backward particle filter algorithms, and output a reconstructed image with high time resolution and high quality.
[0084] In another embodiment, a computer system is provided, comprising a memory and a processor, wherein the memory is used to store a computer program / instruction, and the processor is used to execute the computer program / instruction to implement the event camera image reconstruction method based on the forward and backward particle filter.
[0085] like Figure 5 The figure shows the application of the FBPT image reconstruction algorithm in the ACD dataset. In order to model the image frames more accurately, we first assume that the pixel intensity between frames changes linearly. Considering the time information of the events between frames, the pixel intensity of the corresponding event is obtained through the inter-frame interpolation algorithm according to the time information. Finally, the forward and backward particle filter (FBPF) algorithm is used to combine the events and image frames to output a reconstructed image with high temporal resolution and high quality.
[0086] like Figure 6The figure shows the reconstruction results of four typical scenes of the IJRR dataset based on different image reconstruction algorithms. We successfully applied our method to the evaluation experiment of the public event camera dataset. Compared with the direct integration (DI) method, Kalman filter (KF) method and complementary filter (CF) method, the proposed method can achieve high-quality image reconstruction.
[0087] The present invention models and analyzes the motion blur and non-normally distributed event noise in low-frequency intensity frames, so that the algorithm has a relatively better image reconstruction effect. The intensity image taken by the event camera will have a problem of poor image quality under overexposure and low light conditions. The event and image frames are combined to fuse the information of the event and grayscale image. The algorithm models and analyzes the motion blur and non-normally distributed event noise in low-frequency intensity frames, so that the algorithm has a relatively better image reconstruction effect. We can set a relatively smaller weight for the intensity map of poor quality to reduce the impact of the wrong pixel intensity value on the image reconstruction.
[0088] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An event camera image reconstruction method based on forward and backward particle filters, It is characterized in that The following steps are involved: Step S1, constructing an image reconstruction model based on the DAVIS event camera for possible motion blur problems of intensity frames obtained by the DAVIS event camera based on a fixed exposure time; Step S2, fusing information from the event and grayscale images according to an image reconstruction model based on a DAVIS event camera, modeling the intensity frame as a continuous time process to obtain a continuous time system equation; Step S3, discretizing the continuous-time system equation so that image reconstruction estimation can be performed through a filter and non-Gaussian event noise is modeled to obtain a discretized image event noise model, and obtaining a discretized image and an event with noise through the discretized image event noise model; Step S4, combining the obtained discretized image and the event with noise with the image frame through the forward and backward particle filtering algorithm, and outputting a reconstructed image with high time resolution and high quality.
2. According to the event camera image reconstruction method based on forward and backward particle filter according to claim 1, Features: The image reconstruction model based on the DAVIS event camera is: in, express Moment Pixel The logarithmic intensity value, Indicates The time when the picture was obtained. Indicates the exposure time, express Moment in Pixels The pixel intensity value, express Moment Pixel The potential clear image gray value is at, c represents the event threshold, express Moment Pixel The events at the place.
3. According to the event camera image reconstruction method based on forward and backward particle filter according to claim 2, Features: Step S1: The method for constructing an image reconstruction model based on the DAVIS event camera is as follows: Step S11, an event is represented as: in, Indicates an event, , t represents the time when the event is triggered, Indicates the pixel coordinates where the event was triggered; Indicates the polarity of an event; If we use a continuous time system to describe the event process, then in pixels An event is defined as: in, Indicates time The polarity of moment events, Represents the increase in pixel logarithmic grayscale value, represents the decrease of logarithmic grayscale value, c represents the event threshold, is the Dirac function; Step S12, integrating each event cycle to obtain: in, represents the event period integral, Indicates time The polarity of the event at the moment, c represents the event threshold, is the Dirac function; Step S13, for discrete intensity frames based on a fixed frame rate provided by the DAVIS event camera, convert the pixel intensity values into logarithmic intensity values, and convert the pixel grayscale values into logarithmic intensity values. Add a fixed bias on the basis of have to: in, represents the logarithmic intensity value, express Moment in Pixels The pixel intensity value, It indicates fixed bigotry; Step S14: According to the imaging principle, the pixel intensity value of motion blur is understood as the average of the real pixel intensity at each moment during the exposure time. Therefore, the pixel intensity of one frame is expressed as: in, Indicates The pixel intensity of the image, Indicates the exposure time, Indicates The time when the image was obtained; Step S15, the blurred image is processed, and the relationship between the image frame and the event is established in combination with the imaging mechanism analysis of the above image frame as follows: but: in, express Moment Pixel The logarithmic intensity value, Indicates The time when the picture was obtained. Indicates the exposure time, express Moment in Pixels The pixel intensity value, express Moment Pixel The potential clear image gray value is at, c represents the event threshold, express Moment Pixel The events at the place.
4. The event camera image reconstruction method based on forward and backward particle filter according to claim 3, Features: Step S2 fuses the information from the event and grayscale images, and the specific steps of modeling the intensity frame as a continuous time process are: Step S21, according to the time information, the pixel intensity of the corresponding event is obtained by using an inter-frame interpolation algorithm, and the pixel intensity information of the inter-frame event time information is interpolated: in, express Moment Pixel Logarithmic intensity value at; Step S22, the events are described using the zero-order holder method: in, express Moment Pixel The logarithmic intensity value, express Moment Pixel Logarithmic intensity value.
5. The event camera image reconstruction method based on forward and backward particle filter according to claim 4, Features: Step S3 discretizes the continuous-time system equation so that the image reconstruction estimation can be performed through the filter and the specific steps of modeling the non-Gaussian event noise are: Step S31, the continuous-time system model of DAVIS camera image reconstruction is expressed as: in, Represents pixels The observed value of the logarithmic grayscale at ; represents event noise; represents the measurement noise of the pixel grayscale; Step S32: Since the event is modeled as a continuous Dirac function, integration is performed to obtain: in, express Moment Pixel Grayscale logarithm of the point; Step S33, the system model of DAVIS camera image reconstruction is expressed as: in, express Moment Pixel The grayscale logarithm of the point, express Moment Pixel The event cycle integral of a point, express Moment Pixel Point event noise; Step S34, modeling event noise and observation noise as: in, express Moment Pixel The event noise of the point, Indicates the instability of event thresholds, represents the event threshold, It means it obeys normal distribution. represents the variance, represents the observation noise.
6. The event camera image reconstruction method based on forward and backward particle filter according to claim 5, Features: In step S4, the method of combining events and image frames through forward and backward particle filtering algorithms to output reconstructed images with high temporal resolution and high quality is as follows: a particle smoother is introduced into the image reconstruction of the event camera, the particle smoother includes a forward particle estimator and a backward smoother; the forward particle estimator is used to calculate the particle weight; the forward particle estimator is used to calculate the particle weight.
7. The event camera image reconstruction method based on forward and backward particle filter according to claim 6, Features: The method used by the forward particle estimator to calculate particle weights is: The particle set sampled in the prediction stage is based on the prior probability density is generated: in, express The sampling particle set at time, represents the particle sampling function, represents the prior probability density, represents the number of sampled particles; Calculate the prior state estimate for each particle based on the state equation: in, express The prior state estimate at time t, represents the forecast covariance, represents the event threshold; The prior estimates of the observed variables are expressed as: in, represents the prior estimate of the observed variable; In the update phase, the weights of the sampled particles are calculated: in, is the weight of the sampled particle, represents the likelihood function, Indicates The sampled particles are in pixels The observed value of the logarithmic grayscale at , express The prior state estimate at time t, is the probability density transfer function, is the importance sampling density.
8. The event camera image reconstruction method based on forward and backward particle filter according to claim 7, Features: The method for solving the particle weight of the smoother based on the particle weight calculated by the forward particle estimator is: The weight of the particles is based on the following probability density function: in, represents the probability density function, Indicates the current state. represents a posteriori knowledge, Indicates the observation time; Based on the above formula, the weight formula of particle smoothing is: in, represents the particle weight of the smoothing process, express The weight of the moment, express The sampling particle set at time, express The weight of the moment; Normalizing the weights yields: in, represents the normalized weight, Represents the particle weight of the smoothing process; Then the FBPF estimate of the pixel logarithm is expressed as: in, represents the FBPF estimate of the pixel logarithm, represents the normalized weight, express The prior state estimate at time t.
9. An event camera image reconstruction system based on forward and backward particle filters, It is characterized in that The method for event camera image reconstruction based on forward and backward particle filter according to any one of claims 1 to 8 comprises an image reconstruction model unit based on a DAVIS event camera, a continuous-time system equation unit, a discretized image event noise model unit, and a particle filter algorithm unit, wherein: The DAVIS event camera-based image reconstruction model unit is used to solve the possible motion blur problem of the intensity frame obtained by the DAVIS event camera based on a fixed exposure time, and to construct an image reconstruction model based on the DAVIS event camera; The continuous-time system equation unit is used to fuse information from events and grayscale images according to an image reconstruction model based on a DAVIS event camera, and to model the intensity frame as a continuous-time process to obtain a continuous-time system equation; The discretized image event noise model unit is used to discretize the continuous-time system equation so that image reconstruction estimation can be performed through a filter and non-Gaussian event noise can be modeled; The particle filter algorithm unit is used to combine events and image frames according to the obtained discretized images and events with noise through the forward and backward particle filter algorithms, and output a reconstructed image with high time resolution and high quality.
10. A computer system, It is characterized in that It comprises a memory and a processor, wherein the memory is used to store computer programs / instructions; and the processor is used to execute the computer programs / instructions to implement the event camera image reconstruction method based on forward and backward particle filters as described in any one of claims 1-8.
Citation Information
Patent Citations
Event camera image reconstruction method and system based on manifold constraint
CN111798370A
Cited By
Spectral line intensity correction method, system and device and storage medium
CN120031760A
