Denoising method for dynamic visual event stream based on hot pixels and enhanced spatiotemporal correlation

By constructing the dynamic response characteristics of event density map and dynamic vision sensor, the high frequency and background noise in the event stream of dynamic vision sensors are separated and removed, the problem of noise impact in the prior art is solved, and the noise reduction accuracy of event streams is improved and efficiency.

CN115442544BActive Publication Date: 2024-11-12SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211076656.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-05
Publication Date
2024-11-12
Estimated Expiration
2042-09-05

AI Technical Summary

Technical Problem

The event stream output by the dynamic vision sensor contains a large number of noise events, affecting the efficiency and accuracy of the algorithm. The existing spatiotemporal correlation filter fails when the noise is locally aggregated, and traditional denoising methods cannot accurately remove noise in real time.

Method used

By constructing an event density map, combining the high-frequency noise characteristics of thermal pixels and the spatiotemporal continuity of effective events, high-frequency noise and background noise are separated, and dynamic response characteristics of dynamic vision sensors are introduced to enhance the spatial-temporal correlation noise reduction criteria, and filter separately Defective of noise in overlapping and transient events, the event stream after noise reduction is finally compensated.

Benefits of technology

It significantly improves the noise reduction accuracy of event streams, overcomes the problem of filter failure during local noise aggregation, and takes into account the noise removal efficiency and the retention ability of effective events.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115442544B_ABST
    Figure CN115442544B_ABST
Patent Text Reader

Abstract

The present invention provides a method for denoising a dynamic visual event stream based on hot pixels and enhanced spatiotemporal correlation, comprising the following steps: 1. reading the event stream output by a dynamic visual sensor and performing spatiotemporal modeling on the event; 2. dividing the noise event into a high-frequency noise event and a background noise event according to the spatiotemporal characteristics; 3. constructing an event density map, combining the continuous high-frequency characteristics of the hot pixel output noise and the spatiotemporal continuity of the effective event, detecting the hot pixel in real time, and eliminating the high-frequency noise; 4. introducing the dynamic response characteristics of the dynamic visual sensor under different intensity gradients, enhancing the spatiotemporal correlation denoising criteria, and accurately filtering out the background noise; 5. using the denoised event stream to compensate for the output of the hot pixel, and improving the event output accuracy. The method is suitable for denoising the event stream output by a dynamic visual sensor, taking into account both the noise event removal efficiency and the effective event retention capability, and achieving a higher denoising accuracy with smaller computing resources.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of sensor signal processing, and in particular relates to a dynamic visual event stream denoising method based on hot pixels and enhanced spatiotemporal correlation. Background Art

[0002] As a new type of bionic neuromorphic sensor, dynamic vision sensors have great advantages in challenging scenes such as high-speed motion, low light, and high contrast, thanks to their advantages such as low latency, high temporal resolution, and high dynamic range, overcoming the application limitations of traditional cameras. Unlike traditional cameras that synchronously output intensity image frames at a fixed frequency, the pixels of dynamic vision sensors are independent of each other, asynchronously detecting and responding to logarithmic changes in brightness (called events). When these pixel-level brightness changes reach a certain threshold, events are triggered, and the pixel coordinates, timestamps, and polarity of the events are asynchronously output with microsecond resolution. At present, visual algorithms based on dynamic vision sensors have been successfully applied to target tracking, 3D reconstruction, SLAM, optical flow estimation, and other fields.

[0003] However, since dynamic vision sensors are very sensitive to brightness changes in the environment and are affected by external interference such as their own hardware circuits and environmental factors, a large number of noise events are included in their output event streams, which seriously affects the efficiency and accuracy of subsequent algorithms and wastes a lot of computing resources, becoming the main challenge restricting the further development and application of dynamic vision sensors. On the other hand, traditional two-dimensional image and video denoising methods cannot be directly applied to three-dimensional event streams containing position and time information, and it is difficult to denoise event streams with such complex spatiotemporal characteristics. Existing event stream denoising methods cannot accurately remove noise in event streams in real time. According to the event triggering principle, real events are usually generated by intensity gradient regions such as edges in the scene, and are often continuous in time and space, while noise events are usually independent, discrete and discontinuous points in spatiotemporal distribution. Compared with real events, spatiotemporal correlation is poor. Therefore, spatiotemporal correlation has become the basic criterion for event stream denoising. The nearest neighbor (NNb) filter based on spatiotemporal correlation uses the time difference between each event and the latest event on the adjacent pixel to represent its spatiotemporal correlation, and defines events greater than a threshold as noise. However, when multiple noise events are triggered simultaneously in a pixel neighborhood, as the spatiotemporal continuity of the noise increases, filtering algorithms such as NNb based on spatiotemporal correlation will fail. Therefore, it is necessary to develop a high-precision event stream denoising algorithm.

[0004] The existing solutions mainly include the following:

[0005] Application date: 2021 / 8 / 24; Application number: CN202110973961.0, Patent name: Adaptive cutting method of spatiotemporal data stream based on bionic visual sensor. This patent relates to the field of image processing technology. It solves the existing data cutting methods, such as large errors in spatiotemporal data stream cutting, poor robustness to noise, and the unupdated confidence intervals involved in the calculation. The method can reduce noise during the data cutting process, reduce the impact of noise on data stream cutting, and improve the robustness to noise. Moreover, the calculation parameters are continuously updated according to the scene to cope with changes in target speed or quantity, making the method suitable for various complex scenes. And by using the past event elimination mechanism, a virtual frame with clear and sharp edges can be obtained from the spatiotemporal data stream. Finally, the spatiotemporal data stream can be adaptively cut, and the target information contained in the cut spatiotemporal data segment can retain the complete target information without the phenomenon of smearing, thereby improving the accuracy of the subsequent acquisition of target motion information.

[0006] For each event, a 3x3 slider is constructed with it as the center, and whether the event is noise is determined based on whether other events occur in it. However, the present application first eliminates high-frequency noise by constructing an event density map, and then enhances the spatiotemporal correlation noise reduction criterion in combination with the dynamic response characteristics of the dynamic vision sensor, and eliminates background noise in overlapping events and transient events respectively. Although it involves a process of combining adjacent pixels to construct a spatiotemporal neighborhood for noise judgment, the specific judgment conditions for noise are different.

[0007] Application date: 2020 / 7 / 29; Application number: CN202010744062.9, Patent name: A method for denoising the output event stream of a dynamic visual sensor, which relates to the field of image sensors, and solves the defects of existing denoising methods, such as requiring high computing power and complex computing methods, large delays, or fixed parameters that cannot be adjusted. At the same time, the algorithm is not conducive to embedded transplantation, and the events in the spatial field are not fully utilized, resulting in loss of spatial information. This method counts and judges the number and position of events in the spatiotemporal neighborhood of the newly arrived event to determine whether the event is a noise event. Compared with the existing methods, this method has the advantages of simple calculation and adjustable parameters. The method of the present invention does not require future events to participate in calculation and judgment, and reduces delayed events. The newly arrived events are filtered in sequence, and the data is not processed in the form of frames, so as to better retain time information and resolution. By judging the spatial position, the flicker noise that occurs at high frequencies can be eliminated.

[0008] The method first determines whether each event is random noise by whether the number of events in the spatiotemporal neighborhood reaches a judgment threshold, and then determines whether it is flicker noise based on whether there are events in other surrounding pixels, thereby achieving denoising of the event stream; while the present application first constructs an event density map, sets an absolute threshold and an adaptive threshold in combination with the continuous high-frequency characteristics of high-frequency noise and the spatiotemporal continuity of effective events, detects the hot pixels in the dynamic vision sensor, and determines the events it outputs as high-frequency noise events; then, the spatiotemporal correlation denoising criterion is enhanced in combination with the dynamic response characteristics of the dynamic vision sensor, and the background noise in overlapping events and transient events is eliminated respectively. Although this involves the process of constructing a spatiotemporal neighborhood for noise judgment, the specific judgment conditions for noise are completely different; in addition, the patent applied for also uses the denoised event stream to compensate for the output of hot pixels to improve the accuracy of event output.

[0009] Application date: 2017 / 3 / 30; Application number: CN201710203429.4, Patent name: A pixel unit and its denoising method, dynamic visual sensor, imaging device, wherein the pixel unit includes a pixel photosensitive circuit for detecting light signals, and generating and outputting electrical signals when detecting changes in the light signals; and a communication circuit connected to the pixel photosensitive circuit and outputting electrical signals; and also includes: a neighborhood denoising circuit, connected to the pixel photosensitive circuit, the communication circuit and the other pixel units in the four neighborhoods, and used to control whether the communication circuit outputs the electrical signal according to the response state of the adjacent pixel units. In the embodiment of the invention, when the pixel unit responds to the light signal and a triggering event occurs, by obtaining the excitation state signals of the pixel units in the four neighborhoods and judging their states, if at least three of the pixel units in the four neighborhoods are in an unresponsive state, the pixel unit is made not to respond, effectively avoiding the generation of isolated noise, and having the beneficial effects of fast processing speed and high denoising accuracy.

[0010] The invention uses a neighborhood denoising circuit to reduce the noise of dynamic vision sensors at the hardware level. If at least three of the four neighborhood pixels are in an unresponsive state, the central pixel will not respond. However, the present application first eliminates high-frequency noise by constructing an event density map, and then enhances the spatiotemporal correlation noise reduction criterion in combination with the dynamic response characteristics of the dynamic vision sensor, and eliminates the background noise in overlapping events and transient events respectively. Although it is also noise reduction for dynamic vision sensors, the noise reduction process is completely different from that of the invention. Summary of the invention

[0011] In response to the above problems, the present invention provides a dynamic visual event stream denoising method based on hot pixels and enhanced spatiotemporal correlation. This method overcomes the failure problem of existing spatiotemporal correlation filters when noise is locally concentrated, takes into account both the efficiency of noise event removal and the ability to retain effective events, and greatly improves the denoising accuracy of the event stream.

[0012] The present invention provides a dynamic visual event stream denoising method based on hot pixels and enhanced spatiotemporal correlation, and the specific steps are as follows:

[0013] Step 1: Read the event stream output by the dynamic vision sensor and model the event as a pulse function in a three-dimensional space-time system:

[0014] e i (x i ,y i ,t i )=δ(xx i ,yy i ,tt i )

[0015] where e i represents the i-th event in the event stream, x i ,y i Represents the pixel coordinates of the event, t i represents the timestamp of the event, δ(·) represents the impulse function;

[0016] Step 2: Based on the spatiotemporal characteristics, the noise events in the dynamic visual sensor output event stream are divided into high-frequency noise events continuously output by hot pixels and background noise events randomly distributed on the pixel plane;

[0017] Step 3: By constructing an event density map, combining the continuous high-frequency characteristics of hot pixel output noise and the spatiotemporal continuity of effective events, hot pixels are detected in real time to eliminate high-frequency noise;

[0018] Step 4: According to the time interval of the continuous triggering events on each pixel of the dynamic vision sensor, the events are divided into overlapping events that are repeatedly triggered on the same pixel in a short period of time and transient events that are only triggered once. By introducing the dynamic response characteristics of the dynamic vision sensor under different intensity gradients, the spatiotemporal correlation noise reduction criterion is enhanced to filter out the background noise in overlapping events and transient events respectively.

[0019] Step 5: In the output event stream of the hot pixel, there may be valid events submerged in high-frequency noise. In order to accurately describe the scene and target, after filtering out the high-frequency noise output by the hot pixel, the output is compensated in combination with the denoised event stream.

[0020] As a further improvement of the present invention, the construction of the event density map in step 3 includes the following process:

[0021] For each pixel x, y in the dynamic vision sensor, a sliding window with a fixed time length is used to count the cumulative triggering events at the corresponding pixel in the recent period of time, and construct the event density map D(x, y):

[0022]

[0023] where t c The current time, t w is the length of the sliding window.

[0024] As a further improvement of the present invention, the hot pixel detection and high-frequency noise removal in step 3 include the following processes:

[0025] (3-1) Since the hot pixels continuously output high-frequency events with a time interval less than the Refractory Period, we first set the absolute threshold of the number of events by combining the window length and the Refractory Period.

[0026]

[0027] The proportional coefficient s is set to 0.8, τ RP Refractory Period, set to 20ms;

[0028] (3-2) According to the spatiotemporal continuity of effective events, the intensity gradient in the scene will stimulate a similar number of events in adjacent pixel areas, resulting in the window of the hot pixel containing far more events than the adjacent pixels. Therefore, the local maximum pixel on the event density map is detected by adaptive Gaussian threshold, and non-maximum suppression is performed. Together with the absolute threshold, it is used as the judgment condition for hot pixels. The pixels that meet the conditions on the event density map are identified as hot pixels, and their output event e hp Identify it as a noise event to effectively eliminate high-frequency noise in the event stream:

[0029]

[0030]

[0031] in, represents the adaptive Gaussian threshold, Represents the event density map and Gaussian weight G σ The convolution of (x,y) is performed, the window size of the adaptive Gaussian threshold is set to 5x5, and hp represents the dynamic vision sensor pixel plane Z 2 Hot pixels on the

[0032] As a further improvement of the present invention, the event classification process and background noise removal in step 4 include the following process:

[0033] (4-1) First, use the Refractory Period filter to distinguish overlapping events from transient events, and obtain the non-high-frequency noise events e that are repeatedly triggered on each pixel. o ;

[0034] (4-2) Then, according to the dynamic response characteristics of the dynamic vision sensor, the overlapping event e o It is an effective event that is continuously triggered on the same pixel by the edge or other areas with large intensity gradient in the scene, or background noise. Therefore, the spatiotemporal continuity of the scene edge is used to remove the background noise using the nearest neighbor NNb filter to obtain the effective overlapping event. And keep it as real event for output;

[0035] (4-3) Finally, for the transient event e in the event stream i (x i ,y i ,t i ), with the non-hot pixels in the surrounding 8 pixels, combined with the time interval τ th Establish a spatiotemporal neighborhood N i :

[0036] N i ={(x,y,t)||xx i |≤1,|yy i |≤1,|tt i |≤τ th}(x i ,y i )

[0037] And according to the events in the space-time neighborhood and their space-time characteristics, the event e i The spatiotemporal relevance and effectiveness of can be divided into the following three cases:

[0038] Case 1: If N i If there is any valid overlapping event in , then the event is a real event triggered by a low-intensity gradient such as an edge with uneven lighting or a scene texture near an edge, so e i Identify and preserve as a real event;

[0039] Case 2: When e i If there are only transient events in the spatiotemporal neighborhood, then all events ej(xj,yj,t j ) to construct similar sub-spacetime neighborhoods

[0040]

[0041] Afterwards, if In any event, event e i If there is an event in the spatiotemporal neighborhood of a non-empty sub-spatiotemporal neighborhood, it is identified as a valid event and output; and when e i When there is only an event with an empty sub-spacetime neighborhood in the spacetime neighborhood, it is still judged as background noise and removed;

[0042] Case 3: If e i If there is no event in the spatiotemporal neighborhood of , that is, there is no event triggered by the surrounding pixels in the recent moment, the event is judged as background noise and removed.

[0043] As a further improvement of the present invention, the implementation steps of the Refractory Period filter and the nearest neighbor filter in step 4 are as follows:

[0044] The Refractory Period filter distinguishes overlapping events from transient events by setting a minimum time interval between consecutive events on the same pixel, and obtains non-high-frequency noise events that are repeatedly triggered on each pixel. o :

[0045]

[0046] Among them, τ RP is the time interval threshold of the Refractory Period filter, t i (x i ,y i ) represents pixels (x i ,y i )The timestamp of the i-th event on

[0047] The nearest neighbor filter requires that valid events exist in adjacent pixels close in time, and filters out valid overlapping events.

[0048]

[0049] where t m (x i ,y i ) indicates the overlapping event e on the corresponding pixel o The timestamp, τ NNb is the time interval threshold of the NNb filter, and the time interval threshold is set to 20ms.

[0050] As a further improvement of the present invention, the compensation process for hot pixels in step 5 includes:

[0051] For each hot pixel, if its spatiotemporal neighborhood Nhp If there are two consecutive real events triggered with the same polarity on different pixels, the output of the hot pixel is compensated using the events with the same polarity:

[0052]

[0053] if(x i ,y i )≠(x i-1 ,y i-1 )and p i =p i-1

[0054] e hp (x hp ,y hp ,t hp ,p hp )=δ(xx hp ,yy hp ,tt i ,pp i )

[0055] in N hp The i-th real event in hp is the compensated event output at this hot pixel.

[0056] Beneficial effects:

[0057] The present invention divides the noise in the event stream into high-frequency noise and background noise, and eliminates them respectively in combination with their respective spatiotemporal characteristics and the continuity of effective events, thereby improving the efficiency of noise reduction. In addition, the present invention enhances the spatiotemporal correlation noise reduction criterion by introducing the DVS dynamic characteristics under different intensity gradients, effectively solving the failure problem of the existing spatiotemporal correlation filter when the noise is locally concentrated, and taking into account both the noise event removal efficiency and the effective event retention ability, effectively improving the noise reduction accuracy of the event stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of the event stream denoising method provided by the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0060] The present invention discloses a dynamic visual event stream denoising method based on hot pixels and enhanced spatiotemporal correlation. The process of the method of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0061] Step 1: Read the event stream output by the dynamic vision sensor and model the event as a pulse function in a three-dimensional space-time system:

[0062] e i (x i ,y i ,t i )=δ(xx i ,yy i ,tt i )

[0063] where e i represents the i-th event in the event stream, (x i ,y i ) represents the pixel coordinates of the event, t i represents the timestamp of the event, and δ(·) represents the impulse function.

[0064] Step 2: According to the spatiotemporal characteristics, the noise events in the dynamic visual sensor output event stream are divided into high-frequency noise events continuously output by hot pixels and background noise events randomly distributed on the pixel plane.

[0065] Step 3: In order to detect hot pixels and eliminate high-frequency noise, a sliding window with a fixed time length is used for each pixel (x, y) in the dynamic vision sensor, and the cumulative triggering events at the corresponding pixel in the recent period are counted to construct the event density map D(x, y):

[0066]

[0067] where t c The current time, t w is the length of the sliding window.

[0068] Based on the event density map, combined with the continuous high-frequency characteristics of the hot pixel output noise and the spatiotemporal continuity of the effective events, the hot pixels are detected in real time to eliminate the high-frequency noise, including the following steps:

[0069] (3-1) Since the hot pixels continuously output high-frequency events with a time interval less than the Refractory Period, we first set the absolute threshold of the number of events by combining the window length and the Refractory Period.

[0070]

[0071] The proportional coefficient s is set to 0.8, τ RP is the Refractory Period, set to 20ms.

[0072] (3-2) In addition, according to the spatiotemporal continuity of effective events, the intensity gradient in the scene will stimulate a similar number of events in adjacent pixel areas, resulting in the window of the hot pixel containing far more events than the adjacent pixels. Therefore, the local maximum pixel on the event density map is detected by adaptive Gaussian threshold, non-maximum suppression is performed, and the absolute threshold is used together as the judgment condition for hot pixels. The pixels that meet the conditions on the event density map are identified as hot pixels, and their output event e hp Identify it as a noise event and eliminate high-frequency noise in the event stream:

[0073]

[0074]

[0075] in, represents the adaptive Gaussian threshold, Represents the event density map and Gaussian weight G σ The convolution of (x,y) is performed, the window size of the adaptive Gaussian threshold is set to 5x5, and hp represents the dynamic vision sensor pixel plane Z 2 Hot pixels on the

[0076] Step 4: According to the time interval of the continuous triggering events on each pixel of the dynamic vision sensor, the events are divided into overlapping events that are repeatedly triggered on the same pixel in a short period of time and transient events that are triggered only once. By introducing the dynamic response characteristics of the dynamic vision sensor under different intensity gradients, the spatiotemporal correlation noise reduction criterion is enhanced to filter out the background noise in the overlapping events and transient events respectively, including the following steps:

[0077] (4-1) First, the Refractory Period filter is used to distinguish overlapping events from transient events by setting a minimum time interval between consecutive events on the same pixel, and obtain the non-high-frequency noise events that are repeatedly triggered on each pixel. o :

[0078]

[0079] Among them, τ RP is the time interval threshold of the Refractory Period filter, t i (x i ,y i ) represents pixels (x i ,y i ) is the timestamp of the i-th event on .

[0080] (4-2) Then, according to the dynamic response characteristics of the dynamic vision sensor, the overlapping event e oIt may be a valid event that is continuously triggered on the same pixel by an area such as an edge with a large intensity gradient in the scene, or it may be background noise. Therefore, the spatiotemporal continuity of the scene edge is used to remove the background noise using the nearest neighbor (NNb) filter to obtain the valid overlapping events. And keep it as real event for output:

[0081]

[0082] where t m (x i ,y i ) indicates the overlapping event e on the corresponding pixel o The timestamp, τ NNb is the time interval threshold of the NNb filter (set to 20ms), that is, the NNb filter requires that valid events exist in adjacent pixels close in time.

[0083] (4-3) Finally, for the transient event e in the event stream i (x i ,y i ,t i ), with the non-hot pixels in the surrounding 8 pixels, combined with the time interval τ th Establish a spatiotemporal neighborhood N i :

[0084] N i ={(x,y,t)||xx i |≤1,|yy i |≤1,|tt i |≤τ th}(x i ,y i )

[0085] And according to the events in the space-time neighborhood and their space-time characteristics, the event e i The spatiotemporal relevance and effectiveness of can be divided into the following three cases:

[0086] Case 1: If N i If there is any valid overlapping event in , then the event is a real event triggered by a low-intensity gradient such as an edge with uneven lighting or a scene texture near an edge, so e i Identify as a real event and retain it.

[0087] Case 2: When e i If there are only transient events in the spatiotemporal neighborhood, then all events ej(xj,yj,t j ) to construct similar sub-spacetime neighborhoods

[0088]

[0089] Afterwards, if In any event, event e i If there is an event in the spatiotemporal neighborhood of a non-empty sub-spatiotemporal neighborhood, it is identified as a valid event and output; and when e i When there are only events with empty sub-spacetime neighborhoods in the spacetime neighborhood of , they are still judged as background noise and removed.

[0090] Case 3: If e i If there is no event in the spatiotemporal neighborhood of , that is, there is no event triggered by the surrounding pixels in the recent moment, the event is judged as background noise and removed.

[0091] Step 5: In the output event stream of hot pixels, there may be valid events submerged in high-frequency noise. In order to accurately describe the scene and target, after filtering out the high-frequency noise events output by the hot pixels, it is necessary to combine the denoised event stream and perform output compensation on it to improve the output accuracy of the event stream.

[0092] For each hot pixel, if its spatiotemporal neighborhood N hp If there are two consecutive real events triggered with the same polarity on different pixels, the output of the hot pixel is compensated using the events with the same polarity:

[0093]

[0094] if(x i ,y i )≠(x i-1 ,y i-1 )and p i =p i-1

[0095] e hp (x hp ,y hp ,t hp ,p hp )=δ(xx hp ,yy hp ,tt i ,pp i )

[0096] in N hp The i-th real event in hp is the compensated event output at this hot pixel.

[0097] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A dynamic visual event stream denoising method based on hot pixels and enhanced spatiotemporal correlation, the specific steps are as follows, characterized in that: Step 1: Read the event stream output by the dynamic vision sensor and model the event as a pulse function in a three-dimensional space-time system: e i (x i ,y i ,t i )=δ(x-x i ,y-y i ,t-t i ) where e i represents the i-th event in the event stream, x i ,y i Represents the pixel coordinates of the event, t i represents the timestamp of the event, δ(·) represents the impulse function; Step 2: Based on the spatiotemporal characteristics, the noise events in the dynamic visual sensor output event stream are divided into high-frequency noise events continuously output by hot pixels and background noise events randomly distributed on the pixel plane; Step 3: By constructing an event density map, combining the continuous high-frequency characteristics of hot pixel output noise and the spatiotemporal continuity of effective events, hot pixels are detected in real time to eliminate high-frequency noise; The construction of the event density map in step 3 includes the following processes: For each pixel x, y in the dynamic vision sensor, a sliding window with a fixed time length is used to count the cumulative triggering events at the corresponding pixel in the recent period of time, and construct the event density map D(x, y): where t c The current moment, t w is the time length of the sliding window; The hot pixel detection and high-frequency noise removal in step 3 include the following processes: (3-1) Since the hot pixels continuously output high-frequency events with a time interval less than the Refractory Period, we first set the absolute threshold of the number of events by combining the window length and the Refractory Period. The proportional coefficient s is set to 0.8, τ RP Refractory Period, set to 20ms; (3-2) According to the spatiotemporal continuity of effective events, the intensity gradient in the scene will stimulate a similar number of events in adjacent pixel areas, resulting in the window of the hot pixel containing far more events than the adjacent pixels. Therefore, the local maximum pixel on the event density map is detected by adaptive Gaussian threshold, and non-maximum suppression is performed. Together with the absolute threshold, it is used as the judgment condition for hot pixels. The pixels that meet the conditions on the event density map are identified as hot pixels, and their output event e hp Identify it as a noise event to effectively eliminate high-frequency noise in the event stream: in, represents the adaptive Gaussian threshold, Represents the event density map and Gaussian weight G σ The convolution of (x,y) is performed with the adaptive Gaussian threshold window size set to 5x5. hp represents the dynamic vision sensor pixel plane Z. 2 Hot pixels on Step 4: According to the time interval of the continuous triggering events on each pixel of the dynamic vision sensor, the events are divided into overlapping events that are repeatedly triggered on the same pixel in a short period of time and transient events that are only triggered once. By introducing the dynamic response characteristics of the dynamic vision sensor under different intensity gradients, the spatiotemporal correlation noise reduction criterion is enhanced to filter out the background noise in overlapping events and transient events respectively. The event classification process and background noise removal in step 4 include the following processes: (4-1) First, use the Refractory Period filter to distinguish overlapping events from transient events, and obtain the non-high-frequency noise events e that are repeatedly triggered on each pixel. o ; (4-2) Then, according to the dynamic response characteristics of the dynamic vision sensor, the overlapping event e o It is an effective event that is continuously triggered on the same pixel by the edge or other areas with large intensity gradient in the scene, or background noise. Therefore, the spatiotemporal continuity of the scene edge is used to remove the background noise using the nearest neighbor NNb filter to obtain the effective overlapping event. And keep it as real event for output; (4-3) Finally, for the transient event e in the event stream i (x i ,y i ,t i ), with the non-hot pixels in the surrounding 8 pixels, combined with the time interval τ th Establish a spatiotemporal neighborhood N i : N i ={(x,y,t)||x-x i |≤1,|y-y i |≤1,|t-t i |≤τ th }\(x i ,y i ) And according to the events in the space-time neighborhood and their space-time characteristics, the event e i The spatiotemporal relevance and effectiveness of can be divided into the following three cases: Case 1: If N i If there is any valid overlapping event in , then the event is a real event triggered by a low-intensity gradient such as an edge with uneven lighting or a scene texture near an edge, so e i Identify and retain as a real event; Case 2: When there are only transient events in the spatiotemporal neighborhood of ei, all events e in the spatiotemporal neighborhood j (x j ,y j ,t j ) to construct similar sub-spacetime neighborhoods Afterwards, if In any event, event e i If there is an event in the spatiotemporal neighborhood of a non-empty sub-spatiotemporal neighborhood, it is identified as a valid event and output; and when e i When there is only an event with an empty sub-spacetime neighborhood in the spacetime neighborhood, it is still judged as background noise and removed; Case 3: If e i If there is no event in the spatiotemporal neighborhood of , that is, there is no event triggered by the surrounding pixels in the recent moment, then the event is judged as background noise and removed; The implementation steps of the Refractory Period filter and the nearest neighbor filter in step 4 are as follows: The Refractory Period filter distinguishes overlapping events from transient events by setting a minimum time interval between consecutive events on the same pixel, and obtains non-high-frequency noise events that are repeatedly triggered on each pixel. o : Among them, τ RP is the time interval threshold of the Refractory Period filter, t i (x i ,y i ) represents the pixel (x i ,y i )The timestamp of the i-th event on The nearest neighbor filter requires that valid events exist in adjacent pixels close in time, and filters out valid overlapping events. where t m (x i ,y i ) indicates the overlapping event e on the corresponding pixel o The timestamp, τ NNb is the time interval threshold of the NNb filter, and the time interval threshold is set to 20ms; Step 5: In the output event stream of the hot pixel, there may be valid events submerged in high-frequency noise. In order to accurately describe the scene and the target, after filtering out the high-frequency noise output by the hot pixel, the output is compensated in combination with the denoised event stream; The compensation process for hot pixels in step 5 includes: For each hot pixel, if its spatiotemporal neighborhood N hp If there are two consecutive real events triggered with the same polarity on different pixels, the output of the hot pixel is compensated using the events with the same polarity: if(x i ,y i )≠(x i-1 ,y i-1 )and p i =p i-1 e hp (x hp ,y hp ,t hp ,p hp )=δ(x-x hp ,y-y hp ,t-t i ,p-p i ) in N hp The i-th real event in hp is the compensated event output at this hot pixel.