An event stream data denoising method based on spatio-temporal correlation filtering

Abstract

The present invention discloses a denoising method for event stream data based on spatio-temporal correlation filters. The method sequentially reads the current event from the original event stream data obtained by an event camera, obtains its spatio-temporal neighborhood events, analyzes the spatio-temporal correlation of these events, constructs a spatio-temporal correlation filter, and filters the current event until all events in the original event stream are processed. The present invention utilizes the characteristics that noise events occur randomly and independently and conform to the Poisson distribution, while there is spatio-temporal correlation between valid events, and determines the attribute of the current event through spatio-temporal correlation analysis. Since the probability of generating two or more noise events in the adjacent spatial domain of the current event within a given time interval is very low, when it is detected that there are two or more adjacent valid events for the current event, the current event is regarded as a valid event and retained, otherwise it is regarded as a noise event and filtered out. The denoising effect of event stream data is improved, which is beneficial to the expression and detection of valid events.

Description

Technical Field

[0001] The present invention belongs to the technical field of fast-moving object detection, optical flow estimation, high dynamic range image reconstruction, etc. using event stream data acquired by an event camera, and specifically relates to an event stream data denoising method based on spatiotemporal correlation filtering. Background Art

[0002] Dynamic Vision Sensors (DVS) is a new type of bionic vision sensor. The camera using DVS is called an event camera, which is an asynchronous camera based on address event representation (AER). When the event camera is working, it only captures the dynamic information of moving objects in the scene. It can work both in the daytime with good lighting conditions and in the dark night. It uses retina-like dynamic pixel detection technology to perceive and encode visual data. When multiple pixels in the scene request brightening or dimming events at the same time, the event camera will asynchronously output unstructured event stream data with a microsecond delay, which contains events that can characterize the position and time information of dynamic targets in the scene, called valid events. The use of event cameras can effectively eliminate static redundant information, and there will be no dynamic target blurring and tailing phenomena, and it can effectively reduce the amount of data storage and improve the processing speed of subsequent visual tasks. It can be widely used in environmental monitoring, security monitoring, intelligent unmanned systems and other fields.

[0003] However, the circuit structure used by the event camera itself is very sensitive to environmental changes. The background activity noise caused by the thermal noise of the DVS circuit and the junction leakage current will inevitably form events at some pixels in the scene, which are called noise events. Therefore, there are some noise events in the event stream data output by the event camera. Noise events will interfere with the expression and detection of valid events and consume unnecessary communication bandwidth and processing resources. Therefore, it is necessary to denoise the event stream data. However, completely different from the concept of denoising of traditional frame images, the purpose of denoising event stream data is to filter out the noise events and retain as many valid events as possible that reflect the motion of dynamic targets in the scene.

[0004] When using event cameras to collect event stream data, the detection threshold of the camera for scene brightness changes is generally adjusted manually through the camera's supporting software to reduce the camera's detection sensitivity to events and reduce the impact of noise events. However, this method often results in a reduction in the amount of overall event stream data output and the loss of some valid events. Although there are many denoising methods for traditional frame images, their processing methods cannot be directly applied to event stream data. Summary of the invention

[0005] The object of the present invention is to provide a method for denoising event stream data based on spatio-temporal correlation filtering. This method utilizes the characteristics that the noise events in the event stream data output by the event camera occur randomly and independently and conform to the Poisson distribution, while there is spatio-temporal correlation among the effective events reflecting moving targets, to distinguish noise events from effective events and achieve the purpose of filtering out noise events.

[0006] To achieve the above object of the invention, the present invention adopts the following concept:

[0007] Considering that the movement of objects in the scene is completed within a certain time interval and continuous within a certain spatial range, first, the spatial positions and timestamps of each effective event corresponding to the dynamic target in the event stream data are used to form constraints on the spatio-temporal correlation between events, and a spatio-temporal correlation filter is constructed; then, through spatio-temporal correlation analysis, it is determined whether the current event belongs to an effective event. If it is an effective event, it is retained in the event stream data, otherwise it is regarded as a noise event and filtered out; then, spatio-temporal correlation analysis is performed on all events in turn to determine whether they belong to effective events and perform filtering processing.

[0008] According to the above inventive concept, the present invention adopts the following four basic steps to achieve denoising of event stream data:

[0009] Step 1: Input the original event stream data obtained by the event camera, which contains effective events and noise events;

[0010] Step 2: Read the current event, obtain its spatio-temporal neighborhood events, and analyze the spatio-temporal correlation of these events;

[0011] Step 3: Construct a spatio-temporal correlation filter to perform filtering processing on the current event;

[0012] Step 4: Execute Step 2 and Step 3 on each event in turn until all events in the original event stream are processed.

[0013] Compared with the prior art, the present invention has the following outstanding substantial features and remarkable progress:

[0014] The present invention utilizes the characteristics that the noise events occur randomly and independently and conform to the Poisson distribution, while there is spatio-temporal correlation among the effective events reflecting moving targets, to construct a spatio-temporal correlation filter to distinguish noise events from effective events; since the probability of generating two or more noise events in the adjacent spatial domain of the current event within a given time interval is very low, when it is detected that there are two or more adjacent effective events for the current event, the current event is regarded as an effective event and retained, otherwise it is regarded as a noise event and filtered out, thereby improving the denoising effect of the event stream data and being beneficial to the expression and detection of effective events. Description of the Drawings

[0015] Figure 1 This is the implementation step of the event stream data denoising method based on spatio-temporal correlation filtering of the present invention.

[0016] Figure 2 This is the spatio-temporal neighborhood model of a single event of the present invention.

[0017] Figure 3 This is the schematic diagram of the spatio-temporal correlation filter of the present invention. Detailed Embodiment

[0018] The following further describes the specific embodiments of the present invention with reference to the drawings.

[0019] As Figure 1 shown, a method for denoising event stream data based on spatio-temporal correlation filtering comprises the following specific steps:

[0020] Step 1: Input the original event stream data obtained by an event camera, which contains valid events and noise events.

[0021] In the said Step 1, when using the event camera to photograph a moving target in the field of view, each event in the event stream data obtained at the corresponding pixel points contains the position information and time information of the event in the pixel array, expressed as e(x, y, t), where e represents the event generated at the pixel point position (x, y) at the moment t (timestamp). The original event stream data contains both valid events and noise events.

[0022] Although noise events and valid events have the same representation, noise events occur randomly and independently, and their probability distribution conforms to the Poisson distribution, expressed as:[[]]

[0023]

[0024] where N(τ) represents the number of noise events generated by the event camera within the time interval τ, P represents the probability of generating n noise events within the time interval τ, and λ represents the average rate of generating noise events at each pixel point in the event camera, which is one of the factory parameters of the event camera. From the formation mechanism of noise events, it can be known that usually, within a given time interval, the probability of generating two or more noise events in the adjacent spatial domain of a certain pixel point is very low.

[0025] Step 2: Read the current event, obtain the adjacent events in its spatio-temporal neighborhood, and analyze the spatio-temporal correlation of these events.

[0026] 1) In step 2, first set the time interval threshold ΔT, create two empty storage tables Rm and Cm, whose sizes are the same as the number of row and column pixel arrays of the DVS sensor used by the event camera, and use these two tables to store the row and column position information of adjacent events respectively.

[0027] 2) In step 2, according to Figure 2 the spatio-temporal neighborhood model of a single event shown in the figure, for the currently read event e(x, y, t), obtain the position information and timestamp information of the adjacent events in its spatio-temporal neighborhood. In the figure, the red dot represents the current event, the blue dots represent the events e(x, y, t±Δt) generated by the current pixel position (x, y) at different times, where Δt represents the change in time. The green dots represent the events e(x±Δx, y±Δy, t) generated at other positions at the current time t, where Δx and Δy represent the changes in rows and columns in the pixel array respectively, that is, the spatial position changes. The orange dots represent the events e(x±Δx, y±Δy, t±Δt) generated at different positions and different times, where both the row and column information and the timestamp information have changed.

[0028] 3) In step 2, use the following formula (2) to analyze the spatio-temporal correlation between any two events e i (x i , y i , t i ) and e j (x j , y j , t j ):

[0029]

[0030] where ΔT is the set time interval threshold, used to determine whether the difference between the timestamps of two events is within the set range. If the condition of formula (2) is met, it is considered that these two events have spatio-temporal correlation, that is, they are adjacent events.

[0031] 4) In step 2, according to the occurrence position (x, y) of the current event, including the current event, there are at most six possible numbers of events in its neighborhood that satisfy the time interval constraint. Check in turn whether there are two or more adjacent events among these six events according to formula (2): If not, regard the current event as a noise event and remove it from the event stream data, and then directly enter step 4; if so, regard it as two adjacent valid events, obtain their position information and timestamp information, and enter step 3.

[0032] Step 3: Construct a spatio-temporal correlation filter to filter the current event.

[0033] 1) In step 3, construct the current event asFigure 3 The spatio-temporal correlation filter shown in the figure. In the figure, let e 0 (x 0 , y 0 , t 0 ) represent the current event, and e 1 (x 1 , y 1 , t 1 ) and e 2 (x 2 , y 2 , t 2 ) respectively represent two valid events adjacent to the current event.

[0034] 2) In the said step 3, the method for filtering the current event is: according to the constraint condition of formula (3) below, determine whether there are two adjacent valid events in the neighborhood of the current event whose time stamp differences are within the range of ΔT:

[0035]

[0036] If formula (3) is satisfied, regard the current event as a valid event and retain it in the event stream data; otherwise, regard the current event as a noise event and remove it from the event stream data. And update the storage table according to the processing result and update the mark of the current event.

[0037] Step 4: Execute step 2 and step 3 for each event in turn until all events in the original event stream are processed.

[0038] In the said step 4, first judge whether all events in the original event stream data have been processed. If so, the denoising process is completed; otherwise, repeat step 2 and step 3 until all are processed.

[0039] The above-mentioned embodiment is an event stream data denoising method based on a spatio-temporal correlation filter. For the original event stream data obtained by an event camera, read the current event in turn, obtain its spatio-temporal neighborhood events, analyze the spatio-temporal correlation of these events, construct a spatio-temporal correlation filter, and filter the current event until all events in the original event stream are processed. The present invention utilizes the characteristics that noise events appear randomly and independently and conform to the Poisson distribution, while there is spatio-temporal correlation between valid events, and determines the attribute of the current event through spatio-temporal correlation analysis. Since the probability of generating two or more noise events in the adjacent spatial domain of the current event within a given time interval is very low, when it is detected that there are two or more adjacent valid events for the current event, the current event is regarded as a valid event and retained, otherwise it is regarded as a noise event and filtered out. It improves the denoising effect of event stream data and is beneficial to the expression and detection of valid events.

[0040] The above has described the embodiments of the present invention in conjunction with the accompanying drawings. However, the present invention is not limited to the above embodiments, and various changes can be made according to the purpose of the invention of the present invention. Any changes, modifications, substitutions, combinations or simplifications made based on the spirit and principle of the technical solution of the present invention shall be equivalent replacement methods. As long as they meet the invention purpose of the present invention and do not deviate from the technical principle and inventive concept of the present invention, they all fall within the protection scope of the present invention.

Claims

1. A method for denoising event stream data based on spatio-temporal correlation filtering, characterized in that, the specific steps are as follows: Step 1: Input the original event stream data obtained by the event camera, which contains valid events and noise events; Step 2: Read the current event, obtain its spatio-temporal neighborhood events, and analyze the spatio-temporal correlation of these events; Step 3: Construct a spatio-temporal correlation filter to filter the current event; Step 4: Execute Step 2 and Step 3 for each event in turn until all events in the original event stream are processed; In step 3, a spatio-temporal correlation filter is constructed for the current event. Let e 0 (x 0 , y 0 , t 0 ) represent the current event, and e 1 (x 1 , y 1 , t 1 ) and e 2 (x 2 , y 2 , t 2 ) represent two valid events adjacent to the current event respectively; The method for filtering the current event is: According to the constraint condition of formula (3) below, determine whether there are two adjacent valid events in the neighborhood of the current event whose time stamp difference is within the range of ΔT: If formula (3) is satisfied, the current event is regarded as a valid event and retained in the event stream data; otherwise, the current event is regarded as a noise event, removed from the event stream data, and the storage table is updated according to the processing result, and the mark of the current event is updated.

2. The method for denoising event stream data based on spatio-temporal correlation filtering according to claim 1, characterized in that, in Step 1, a moving target in the field of view is photographed by the event camera, and each event in the event stream data obtained by the corresponding pixel points contains the position information and time information of the event in the pixel array, expressed as e(x, y, t), where e represents the event generated at the pixel point position (x, y) at time t; the original event stream data contains both valid events and noise events; Although noise events and valid events have the same representation, noise events occur randomly and independently, and their probability distribution conforms to the Poisson distribution, expressed as: where N(τ) represents the number of noise events generated by the event camera within the time interval τ, P represents the probability of generating n noise events within the time interval τ; λ represents the average rate of generating noise events at each pixel point in the event camera, which is one of the factory parameters of the event camera; from the formation mechanism of noise events, generally, within a given time interval, the probability of generating two or more noise events in the adjacent spatial domain of a certain pixel point is very low.

3. The method for denoising event stream data based on spatio-temporal correlation filtering according to claim 1, characterized in that, 1) In Step 2, first set the time interval threshold ΔT, create two empty storage tables Rm and Cm, the sizes of which are the same as the number of row and column pixel arrays of the DVS sensor used by the event camera, and use these two tables to store the row and column position information of adjacent events respectively; 2) For the read current event e(x, y, t), obtain the position information and time stamp information of its spatio-temporal neighborhood adjacent events; 3) Analyze the spatio-temporal correlation between any two events e i (x i ,y i ,t i ) and e j (x j ,y j ,t j ) using the following equation (2): where ΔT is the set time interval threshold, used to determine whether the time stamp difference between two events is within the set range; if the condition of formula (2) is satisfied, it is considered that these two events have spatio-temporal correlation, that is, they are adjacent events; 4) According to the occurrence location (x, y) of the current event, including the current event, there are at most six event numbers in its neighborhood that may satisfy the time interval constraint; sequentially detect whether there are two or more adjacent events among these six events according to formula (2): if not, regard the current event as a noise event, remove it from the event stream data, and then directly enter step 4; if so, regard it as two adjacent valid events, obtain their location information and timestamp information, and enter step 3.

Images