A spatiotemporal correlation filter for denoising dynamic vision sensors

Abstract

The present invention discloses a spatiotemporal correlation filter for dynamic visual sensor noise reduction, which can filter events of DVS output with n rows and m columns. The filter includes: a row noise reduction module, a column noise reduction module and a detection unit; the row noise reduction module includes N RM There are row storage bodies, each row storage body has a storage block, and each storage block stores S RM events, N RM The column noise reduction module contains N CM There are column storage bodies, each column storage body has a storage block, and each storage block stores S CM events, N CM The s×q window is a preset spatial window. The detection unit sums the number of historical events related to the current event output by the row and column noise reduction modules and compares it with a preset threshold to determine whether the current event is actually noise or an event. This invention reduces the spatial complexity to O(m+n), saves storage resources, and provides high-quality noise reduction with low latency.

Description

Technical Field

[0001] The present invention belongs to the technical field of dynamic vision sensor data processing, and in particular relates to a spatiotemporal correlation filter for denoising a dynamic vision sensor. Background Art

[0002] The dynamic vision sensor (DVS) is a novel neuromorphic imaging device whose design is inspired by the biological retina. When the light intensity in a pixel's receiving area changes beyond a positive or negative threshold, the DVS pixel asynchronously generates a positive or negative event. Therefore, the DVS output is typically represented as a four-tuple (x, y, t, p), corresponding to the pixel's row coordinate, column coordinate, timestamp, and event polarity. The DVS imaging process is differential. In contrast, traditional CMOS image sensors (CIS) are based on an integration principle. The capacitance of each pixel accumulates a charge that is positively correlated with the absolute light intensity during the exposure time. This charge is then synchronously converted into a digital signal on a frame-by-frame basis. This charge accumulation period limits the frame rate of a typical CIS, typically to tens of frames. Furthermore, when the light intensity in a particular area is very high or very low, the corresponding pixel in the CIS struggles to generate a valid signal, resulting in a limited dynamic range (<100 dB). In contrast, the DVS, due to its differential imaging and asynchronous readout, offers advantages such as high temporal resolution in the microsecond range, a high dynamic range of 120-160 dB, and low power consumption. Therefore, DVS has received widespread attention and has been widely used in various computer vision tasks, such as object recognition, object tracking, video deblurring, and video interpolation.

[0003] While DVS's differential imaging approach offers advantages, it is also susceptible to background activity (BA) caused by thermal noise and junction leakage current. This background noise significantly degrades the output signal quality, increases communication bandwidth, and increases power consumption. Therefore, noise reduction is particularly important in DVS research. Object motion is continuous in space and time, while noise is random. Consequently, valid events exhibit spatial and temporal correlation, while noise lacks such correlation. Based on this characteristic, various approaches have been developed to address DVS noise reduction, such as spatiotemporal correlation filters. Their basic principle is that if an event is correlated with previous events within a specific spatial and temporal window, it passes the filter and is considered a valid signal. However, previous spatiotemporal correlation filters have not resolved the trade-off between noise reduction effectiveness and storage overhead. Specifically, filters with a spatial complexity of O(mn) offer good performance but suffer from high storage overhead, making them unsuitable for high-resolution DVS cameras. Filters with a spatial complexity of O(m+n) can conserve storage resources, but their noise reduction performance is significantly degraded. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a spatiotemporal correlation filter for denoising of a dynamic vision sensor.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a spatiotemporal correlation filter for denoising a dynamic visual sensor, comprising: a row denoising module, a column denoising module and a detection unit; the row denoising module comprises N RM row banks, each of which has storage blocks, each storing S RM events, N RM for The i-th row memory bank is used to store the column coordinates and N RM The event where the value obtained by rounding up after performing the modulo operation is i, and the value of i ranges from 0 to N RM -1; the column noise reduction module includes N CM column banks, each with storage blocks, each storing S CM events, N CM for The jth column memory is used to store the row coordinates and N CM The event where the value obtained by rounding up after the modulo operation is j, where j is a value between 0 and N CM -1; s×q is the preset spatial window used for filtering;

[0007] The row noise reduction module is used to select a target row storage body and a target storage block of the target row storage body according to the column coordinates, row coordinates and timestamp of the current event output by the DVS with a resolution of m×n, read historical events from the target storage block, and calculate the noise reduction result according to the spatial correlation threshold D. th and time-dependent threshold T th Determine the number of historical events cr_event_row related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, row coordinates, and polarity of the current event into a corresponding row memory bank;

[0008] The column noise reduction module is used to select a target column storage body and a target storage block of the target column storage body according to the row coordinates, column coordinates and timestamp of the current event output by the DVS, read historical events from the target storage block, and calculate the noise reduction result according to the spatial correlation threshold D. th and time-dependent threshold T thDetermine the number of historical events cr_event_col related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, column coordinates, and polarity of the current event in a corresponding column memory bank;

[0009] The detection unit is used to compare the sum of the number of historical events cr_event_row and the number of historical events cr_event_col with the threshold value N of the number of time-space related events. CR Comparison is performed to detect whether the current event is noise.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] The present invention utilizes a cache-like storage structure for event storage by designing the number of storage bodies included in the row and column noise reduction modules in the filter, the number of storage blocks included in each storage body, and the relationship between each storage body and the horizontal and column coordinates of the stored events. For a DVS with a resolution of m×n, the space complexity can be reduced to O(m+n). At the same time, the filter provided by the present invention further reduces the event storage overhead by compressing the timestamp storage bit width of the event, thereby saving storage resources. In addition, the present invention implements noise reduction through the row and column noise reduction modules and the detection unit, which not only has good noise reduction quality but also has the characteristics of low latency, thereby resolving the contradiction between noise reduction effect and storage overhead. In addition, the present invention can be adapted to different usage environments by changing different parameter configurations, thereby having wider applicability.

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 1 is a schematic diagram of an exemplary correspondence relationship between a DVS pixel array and row and column memory banks provided by an embodiment of the present invention;

[0014] Figure 2 This is a diagram showing the working principle of the secondary cache architecture provided by an embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram of the bit width compression process of the timestamp provided by an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of a false alarm interval provided by an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of an exemplary architecture of a CLF provided in an embodiment of the present invention;

[0018] Figure 6The meanings of the different parameters used in the experiments provided in the embodiments of the present invention;

[0019] Figure 7 The resource utilization and power consumption of CLF implemented by FPGA under different parameters provided by the embodiment of the present invention;

[0020] Figure 8 is a schematic diagram of visualization of clean and noisy event data generated using a v2e simulator provided by an embodiment of the present invention;

[0021] Figure 9 The noise reduction result of CLF provided by the embodiment of the present invention is CM and s RM Schematic diagram of the relationship between;

[0022] Figure 10 The noise reduction result of CLF provided by the embodiment of the present invention is compared with T th Schematic diagram of the relationship between;

[0023] Figure 11 The noise reduction result of CLF provided by the embodiment of the present invention is compared with D th and N CR Schematic diagram of the relationship between;

[0024] Figure 12 This is a comparison chart of the noise reduction results of different videos using noise reduction algorithms using different spatiotemporal correlation filters provided by an embodiment of the present invention;

[0025] Figure 13 4 is a visualization diagram of recorded original DVS data and CLF noise reduction results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0027] The present invention provides a spatiotemporal correlation filter for noise reduction in a dynamic vision sensor, comprising a row noise reduction module (RDM), a column noise reduction module (CDM), and a detection unit. The inputs of the row noise reduction module and the column noise reduction module are both connected to the output of a DVS with an m×n resolution, and the outputs of the row noise reduction module and the column noise reduction module are respectively connected to two inputs of the detection unit.

[0028] Here, the row noise reduction module contains N RM row banks, each of which has storage blocks, each of which is used to store S RM events, N RM for The i-th row memory bank is used to store the column coordinates and NRM The value obtained by rounding up after the modulo operation is the event i, and the column coordinates are the same as N RM The value obtained by rounding down after the modulo operation is the index of the storage block used to store the row coordinates and timestamp of the event in the i-th row storage body. The value of i ranges from 0 to N. RM -1. The column noise reduction module contains N CM column banks, each with storage blocks, each storing S CM events, N CM for The jth column memory is used to store the row coordinates and N CM The value obtained by rounding up after the modulo operation is j, and the row coordinate is the same as N CM The value obtained by rounding down after the modulo operation is the index of the storage block used to store the column coordinates and timestamp of the event in the j-th column storage body. The value of j ranges from 0 to N. CM -1; s×q is the preset spatial window used for filtering, Indicates rounding up operation. RM and S CM Both are positive integers greater than 1, and the values ​​of s and q can be the same or different.

[0029] In the present invention, each row memory bank or column memory bank adopts dual-port storage to perform concurrent reading or concurrent reading and writing.

[0030] For example, Figure 1 This is an exemplary correspondence between the DVS pixel array and the row memory bank and column memory bank. Figure 1 As shown, when N RM =N CM=4, the row noise reduction module contains 4 row memory banks, namely RMB0, RMB1, RMB2, and RMB3, and the column noise reduction module contains 4 column memory banks, namely CMB0, CMB1, CMB2, and CMB3. Among them, the events of the pixels in the 0th, 4th, 8th, and 12th rows of the DVS pixel array will be stored in RMB0, the row coordinates and timestamps of the events of the pixels in the 1st, 5th, 9th, and 13th rows will be stored in RMB1, and the row coordinates and timestamps of the events of the pixels in the 2nd, 6th, and 10th rows will be stored in RMB. In B2, the row coordinates and timestamps of events for pixels in rows 3, 7, and 11 will be stored in RMB3. Similarly, the column coordinates and timestamps of events for pixels in columns 0, 4, 8, and 12 from the DVS pixel array will be stored in CMB0, the column coordinates and timestamps of events for pixels in columns 1, 5, 9, and 13 will be stored in CMB1, the column coordinates and timestamps of events for pixels in columns 2, 6, and 10 will be stored in CMB2, and the column coordinates and timestamps of events for pixels in columns 3, 7, and 11 will be stored in CMB3. The architecture of each row bank is similar to that of S RM Way-set associative cache. A memory block is similar to a cache line and can be read and written by address. If the column coordinate of an event is y e , the row coordinates and timestamp of the event will be stored at address The first row in the bank Each memory block in each row memory bank can store S RM Obviously, the mapping relationship between the column coordinates of the event and the row storage and the storage blocks in the row storage is clear, so there is no need to store the column coordinates of the event in the row storage. Similarly, if the row coordinates of an event are x e , the column coordinates and timestamp of the event will be stored at address The first column in the bank Each memory block in each column memory bank can store S CM Obviously, the mapping relationship between the row coordinates of the event and the column memory and the storage blocks in the column memory is clear. Therefore, there is no need to store the row coordinates of the event in the column memory. Therefore, the event information stored in the storage block in RMB is (x e ,t e ,p e ), which are the row coordinate, timestamp and polarity of the event (1 or 0), for example, Figure 1 The x0, t0, p0 in the CMB is the first event stored in a storage block of RMB, and x1, t1, p1 is the second event stored in the storage block of RMB, and so on. The event information stored in the storage block in CMB is (y e ,t e ,p e),For example, Figure 1 In the example, y0, t0, p0 are the first event stored in a storage block of CMB, y1, t1, p1 are the second event stored in the storage block of CMB, and so on. e need bits are stored, y e need The DVS outputs the timestamp of the event t e Generally 32 bits or more, p e Usually only 1 bit is required.

[0031] In the present invention, the row noise reduction module is used to output the column coordinate y of the current event according to the DVS with a resolution of m×n. e , row coordinate x e and timestamp t e , select the target row storage body and the target storage block of the target row storage body, read the historical events from the target storage block, and calculate the spatial correlation threshold D th and time-dependent threshold T th Determine the number of historical events related to the current event cr_event_row, compress the bit width of the timestamp of the current event, and convert the compressed timestamp and row coordinates of the current event into e The column noise reduction module is used to store the row coordinate x of the current event output by DVS. e , column coordinate y e Sum and timestamp, select the target column storage body and the target storage block of the target column storage body, read the historical events from the target storage block, and calculate the spatial correlation threshold D th and time-dependent threshold T th The number of historical events related to the current event, cr_event_col, is determined, the bit width of the timestamp of the current event is compressed, and the compressed timestamp, column coordinates, and polarity of the current event are stored in the corresponding column memory bank. Obviously, the row and column noise reduction modules in the present invention have the same principles, but process different data.

[0032] The detection unit is used to compare the sum of the number of historical events cr_event_row and the number of historical events cr_event_col with the threshold number of spatiotemporal related events N CR Compare to detect whether the current event is noise.

[0033] Here, the threshold number of spatiotemporal related events N CR It can be a preset value, and can be pre-set according to actual needs.

[0034] Here, the detection unit may include an adder module and a comparator module, wherein the two input ends of the adder module are connected to the output ends of the row noise reduction module and the column noise reduction module respectively, the output end of the adder module is connected to an input end of the comparator module, and the other input end of the comparator module is used to input the threshold value N of the number of spatiotemporal correlation events. CR The output end of the comparator module is used as the output end of the spatiotemporal correlation filter provided by the present invention. The adder module is used to sum the number of historical events cr_event_row and the number of historical events cr_event_col to obtain the total number, and the comparator module is used to compare the total number with N CR The size relationship is determined based on the size relationship to determine whether the current event is noise or an event, thereby deciding whether to output the current event, thereby achieving filtering.

[0035] In some embodiments, the row noise reduction module further comprises: a first control unit, a first write address register, a first adder, a first storage control unit, N RM First event decision unit (EDU), wherein the i-th first event decision unit is connected to the i-th row storage bank. The column noise reduction module also includes: a second control unit, a second write address register, a second adder, a second storage control unit, N CM Second event decision units are provided, wherein the jth second event decision unit is correspondingly connected to the jth column memory bank. It should be noted that the terms "first" and "second" are used only to distinguish between identical modules, units, or components in the row and column noise reduction modules. That is, the processing principles of the corresponding units, components, or modules in the row noise reduction module are the same as those in the column noise reduction module, but the data processed and obtained are different.

[0036] Here, the first control unit is used to determine the index of a row storage body corresponding to the current event as the index of the first target row storage body based on the column coordinates and timestamp of the current event output by the DVS, and determine the index of the storage block to be read from the first target row storage body, and determine the indexes of the corresponding two row storage bodies based on the two adjacent column coordinates of the column coordinates of the current event as the index of the second target row storage body and the index of the third target row storage body, and determine the index of the storage block to be read from the second target row storage body and the third target row storage body respectively, so as to obtain the indexes of the three target row storage bodies and the index of the target storage block in each target row storage body; and generate a write enable signal for the first target row storage body, and store the write enable signal of the first target row storage body into the first write signal register. Each of the three target row storage bodies is used to store the S stored in its own target storage block. RMThe information of each historical event is sent to the first event decision unit connected to itself. The first event decision unit connected to each target row storage bank is used to determine the row coordinates and timestamp of the current event and the spatial correlation threshold D th , time-related threshold T th and S RM The first target row storage body is used to store the index of the target storage block of the first target row storage body, and send the index of the target storage block to the first target row storage body in the next time period after the time period in which the first target row storage body reads data. The first target row storage body is also used to compress the bit width of the timestamp of the current event, and use the row coordinates, polarity and compressed timestamp of the current event to replace the row coordinates, polarity and timestamp of the event to be replaced, so as to store the current event in the target storage block. The first adder is used to sum the three sub-quantities obtained by the three first event decision units corresponding to the three target row storage bodies to obtain a historical event number cr_event_row related to the current event.

[0037] Specifically, the first control unit is used to adopt N RM Column coordinate y of the current event e After performing the modulo operation and rounding up, we get the index of a row storage body. The row storage body corresponding to the index is used as the first target row storage body. The column coordinate y of the current event is e With N RM The quotient of is used as the index of the target storage block of the first target row storage bank; and the column coordinate of the current event is added by 1 by the adder to obtain an adjacent column coordinate y of the column coordinate of the current event. e +1, subtract 1 from the column coordinate of the current event through the subtractor to obtain another adjacent column coordinate y of the column coordinate of the current event e -1, and for each adjacent column coordinate of the column coordinate of the current event (i.e. y e +1 or y e -1), using N RM After performing a modulo operation on the adjacent column coordinates and then rounding up, an index of a row memory bank is obtained, and the row memory bank corresponding to the index is used as the second target row memory bank or the third target memory bank, and the adjacent column coordinates are compared with N RM The quotient of is used as the index of the target storage block of the second target row storage bank or the third target row storage bank.

[0038] Specifically, the first storage control unit includes: a storage update unit and an N RMEach scratchpad memory wpt0 contains The zth storage unit of the i-th note memory wpt0 is used to store the index of the latest event to be replaced in the zth storage block of the i-th row storage body, and the value of z ranges from 0 to The storage update unit is used to obtain the index of the latest event to be replaced from the storage unit corresponding to the target storage block of the first target row storage body, and add 1 to the obtained index of the latest event to be replaced to obtain the updated index of the latest event to be replaced, and then determine whether the updated index of the latest event to be replaced is greater than S RM ; If so, replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with the index of the updated latest event to be replaced; if not, set the updated index of the latest event to be replaced to 0, and replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with 0.

[0039] Specifically, the information of an event includes: row coordinates and timestamp. The first event decision unit connected to each target row storage bank is specifically used to compare the row coordinates of the current event with the S RM The row coordinates of each historical time in the historical events are subtracted, and the row coordinate difference is compared with the spatial correlation threshold D th Compare the results to get the first comparison result of 0 or 1, and compare the timestamp of the current event with S RM The timestamp difference of each historical event is calculated and the timestamp difference is compared with the time-related threshold T th Compare and get the second comparison result with the value of 0 or 1, and then add the first comparison result and the second comparison result to get a result value of 0 or 1. RM S that corresponds to each historical event RM After adding the result values, we get the sub-number of historical events related to the current event.

[0040] Exemplarily, the first event decision unit includes 2S RM Subtractors, 2S RM Comparators, S RM AND gates and an adder, wherein the first subtractor is used to make a difference between the row coordinates of the first historical event and the row coordinates of the current event, the second subtractor is used to make a difference between the timestamp of the first historical event and the timestamp of the current event, and the first comparator is used to compare the column coordinate difference output by the first subtractor with the spatial correlation threshold D th The second comparator is used to compare the timestamp output by the second subtractor with the time-related threshold T thThe first AND gate is used to compare the comparison results output by the first comparator and the second comparator to obtain a result value of 0 or 1. The corresponding relationship between the remaining subtractors, comparators and AND gates can be obtained by reasoning in sequence. RM By adding the result values ​​output by the AND gates, a sub-number of historical events related to the current event can be obtained.

[0041] In some embodiments, three first event decision units correspondingly connected to three target row memory banks operate synchronously.

[0042] In some embodiments, the row noise reduction module further includes: a first read address register, a second read address register, and a feedback signal register. The first read address register is used to store the index of the target storage block in the second target row storage bank, and send the index of the target storage block of the second target row storage bank to the second target row storage bank in a time period next to the time period during which the current event is stored in the first target row storage bank, so that the second target row storage bank obtains the address for reading data in the next time period. The second read address register is used to store the index of the target storage block of the third target row storage bank, and send the index of the target storage block of the third target row storage bank to the third target row storage bank in a time period next to the time period during which the current event is stored in the first target row storage bank, so that the third target row storage bank obtains the address for reading data in the next time period. The feedback signal register is used to store a control signal and send the control signal to the second target row memory bank and the third target row memory bank. The control signal is used to control the second target row memory bank and the third target row memory bank to not read data from the target memory block when the value of a sub-number generated by the first event decision unit connected to the first target row memory bank is not zero, and to control the second target row memory bank and the third target row memory bank to read data from the target memory block when the value of a sub-number generated by the first event decision unit connected to the first target row memory bank is zero. In this way, if the output of the EDU corresponding to the first target memory bank is not zero (i.e., indicating the presence of a related event), the corresponding read operation of the second target memory bank and the third target memory bank will be canceled; otherwise, the corresponding read operation will continue. This read cancel technology is very beneficial for reducing energy consumption, especially in ASIC implementation, thereby further reducing power consumption.

[0043] Theoretically, for a 3×3 spatial window (i.e., a filtering window), the probability that an input event is correlated with an event in the same row or column is higher. The present invention selectively uses the three registers mentioned above, namely, the first read address register, the second read address register, and the feedback signal register, to implement a two-stage pipeline structure for storage access, thereby achieving cycle delay and pipelining of data, and further realizing the executable read cancel operation.

[0044] The following Figure 2 , the principle of read cancel operation is further explained. Figure 2 As shown in FIG, it is assumed that four events e1 to e4 enter the CLF (the spatiotemporal correlation filter for DVS noise reduction proposed in this invention is called a cache-like spatiotemporal filter (CLF)) in sequence, and the corresponding memory banks to be accessed are RMB0, RMB0, RMB1, and RMB3 respectively. The memory access process is shown in FIG. Figure 2 As shown, considering the problem of concurrent reading or concurrent reading and writing, the present invention uses dual-port storage to meet the requirements of concurrent access. In the first stage of the pipeline processing e1, RMB0 is read; in the second stage, the storage block of RMB0 read in the first stage is updated, and at the same time, the storage blocks corresponding to RMB1 and RMB3 (y e1 -1 and y e1 +1) is read. At the same time, the first-stage reading of e2 continues, that is, reading RMB0. Subsequent read and write operations of row storage blocks involve a similar process. During this process, if the output generated by the EDU in the first stage is not zero when processing a certain event (for example, e1), this means that there is a historical event related to event e1 in RMB0, then the read operations related to event e1 on the other two storage banks will be canceled, but the read operations or write operations related to other events (for example, event e2) on these two storage banks that occur at the same time will not be canceled; if the output generated by the EDU in the first stage is zero, this means that there is no historical event related to event e1 in RMB0, then the read operations related to event e1 on these two storage banks will continue.

[0045] In some embodiments, the specific process of compressing the bit width of the timestamp of the current event by the first target row memory bank is as follows: th The binary code of the current event is aligned with the binary code of the timestamp of the current event, and the bit width BW to which the binary code of the timestamp of the current event needs to be compressed is calculated based on the preset coefficient a and the bit width BW T , determine the time-related threshold T th The number of reserved bits in the binary code BW TH ,Right now Set the time-dependent threshold T th High BW in binary code TH bit as a time-dependent threshold T th BW TH The reserved bits are used to convert the binary code of the timestamp of the current event into the time-related threshold T th BW THThe lowest bit of the reserved bits is aligned as the starting bit, starting from the starting bit, and selecting consecutive BW T bits, serving as the compressed timestamp of the current event.

[0046] Here, a is a value greater than 0 and less than 1, and the value of a can be set according to actual needs, which is not limited in the present invention.

[0047] For example, Figure 3 This is a schematic diagram of the principle of compressing the bit width of the timestamp. Figure 3 As shown, assuming the time-dependent threshold T th (abbreviated as time threshold T th ) has 8 bits, and the timestamp to be compressed is T s The binary code has 32 bits, and the timestamp to be compressed is T s The bit width of the binary code to be compressed is 8, a is 30%, and the time threshold T th The binary code and the timestamp T to be compressed s After the binary code is aligned, the relevant values ​​are substituted into the above formula to obtain BW TH =3, then Figure 3 As shown, the time threshold T th The upper 3 bits are reserved, and the time threshold T th The [4:0] bits of the 0x0000 are discarded, thus achieving the time threshold T th Afterwards, due to the time threshold T th The reserved bits are from 6th to 8th, and the time threshold T th The binary code and the timestamp T to be compressed s The binary codes are aligned, so the timestamp to be compressed T s The 5th to 13th bits of the data can be used as the timestamp T to be compressed. s The compressed binary code is used to reduce the bit width of the timestamp.

[0048] Due to the limited width of the timestamp, there is a possibility of overflow and zeroing, which may cause two temporally unrelated events to be mistakenly judged as temporally related. Figure 4 As shown, assuming T th is the range represented by the time threshold, T s is the timestamp overflow period, when the timestamp of subsequent input events falls within [n·T S ,n·T s +T th ] time range, the noise is mistakenly judged as a signal. Through experiments, it is found that when the time exceeds a certain limit, different T th The false positive rate tends to 0 under the value of Ts The bit width does not need to be very large. The present invention can reduce the storage overhead of the timestamp and ensure that the judgment result has a high accuracy through such a compression process.

[0049] Similar to the first control unit in the above-mentioned row noise reduction module, the second control unit is used to calculate the row coordinate x of the current event output by the DVS. e , column coordinate y e and timestamp, determine the index of a column storage bank corresponding to the current event as the index of the first target column storage bank, and determine the index of the storage block to be read from the first target column storage bank, as well as the two adjacent row coordinates (i.e., x e +1 and x e -1) determining the indexes of the corresponding two column storage banks, serving as the indexes of the second target column storage bank and the third target column storage bank, respectively, and determining the indexes of the storage blocks to be read from the second target column storage bank and the third target column storage bank, respectively, to obtain the indexes of the three target column storage banks and the index of the target storage block in each target column storage bank; and generating a write enable signal for the first target column storage bank, and storing the write enable signal of the first target column storage bank in the second write signal register.

[0050] Similar to the target row memory bank in the row noise reduction module, each of the three target column memory banks in the column noise reduction module is used to store the S stored in its own target memory block. CM The information of each historical event is sent to the second event decision unit connected to itself.

[0051] Similar to the first event decision unit in the row noise reduction module, the second event decision unit in the column noise reduction module connected to each target column storage bank is used to determine the event based on the column coordinates and timestamp of the current event and the spatial correlation threshold D th , time-related threshold T th and S CM The information of historical events is used to determine the number of historical events related to the current event and obtain a sub-number.

[0052] Similar to the first storage control unit in the row noise reduction module, the second storage control unit in the column noise reduction module is used to generate an index of the event to be replaced in the target storage block of the first target column storage bank for the current event.

[0053] Similar to the first write address register in the above-mentioned row noise reduction module, the second write address register in the column noise reduction module is used to store the index of the target storage block of the first target column storage body, and send the index of the target storage block of the first target column storage body to the first target column storage body in the time period next to the time period in which the first target column storage body reads data.

[0054] Similar to the first target row storage body in the above-mentioned row noise reduction module, the first target column storage body in the column noise reduction module is also used to compress the bit width of the timestamp of the current event, and use the column coordinates, polarity and compressed timestamp of the current event to replace the column coordinates, polarity and timestamp of the event to be replaced, so as to store the current event in the target storage block.

[0055] Similar to the first adder in the row noise reduction module, the second adder in the column noise reduction module is used to sum the three sub-quantities obtained by the three second event decision units corresponding to the three target column storage bodies to obtain a historical event quantity cr_event_col related to the current event.

[0056] Since the above units, devices or modules in the column noise reduction module have the same structure and principle as those in the row noise reduction module, the specific structures and functions of the above units, devices or modules in the column noise reduction module will not be described in detail.

[0057] In some embodiments, N RM =N CM =S RM =S CM =4, and the index of the first target row memory bank is the binary code BW of the column coordinate of the current event y The lowest two bits, the index of the target memory block of the first target row memory bank is the binary code BW of the column coordinate of the current event y The remaining BW y1 -2 bits; the index of the second target row memory bank is the binary code BW of an adjacent column coordinate of the column coordinate of the current event y1 The lowest two bits, the index of the target memory block of the second target row memory bank is the binary code BW y1 The remaining BW y1 -2 bits; the index of the third target row memory bank is the binary code BW of another adjacent column coordinate of the column coordinate of the current event y2 The lowest two bits, the index of the target memory block of the third target row memory bank is the binary code BW y3 The remaining BW y1 -2 bits. Here, when N RM =N CM =4, and directly take y e The lowest two bits of the binary code are the same. and directly take x e The lowest two bits of the binary code are the same, thus avoiding the complexity of the modulo operation.

[0058] For example, Figure 5 It is in NRM =N CM =S RM =S CM =4, a schematic diagram of the CLF architecture provided by the present invention. Since the structures of the row noise reduction module and the column noise reduction module are similar, Figure 5 The noise reduction module is used as an example to demonstrate this. Figure 5 As shown, the CLF includes a row noise reduction module RDM, a column noise reduction module CDM, an adder module 1 and a comparator module 2. The row noise reduction module RDM contains 4 row storage banks RMB0~RMB3, 3 EDUs, a write address register 3, a read address register 4, a read address register 5, a feedback signal register 6, an adder 7, 4 wpt0s, and a storage update unit 8. In addition, the registers are Figure 5 The control unit is not shown in its entirety, and only a subtractor 9 and an adder 10 included in the control unit are shown.

[0059] The following will be Figure 5 As an example, the working principle of the spatiotemporal correlation filter of the present invention is illustrated. Assume that the spatial correlation threshold D th Set to 1, that is, for the input event, it is necessary to check the historical events of its 3×3 spatial neighborhood. The current input event is represented by (x e ,y e ,t e ,p e ), assuming y e The lowest two bits of y are binary 00. First, y is calculated by subtractor 9 and adder 10 respectively. e -1 and y e +1, which are 11 and 01 respectively. Therefore, RMB0, RMB3 and RMB1 need to be read, and the read addresses of RMB0, RMB3 and RMB1 (i.e. the index of the storage block to be read) are y e 、y e -1 and y e +1 for the highest BW y -2 bits, BW y The bit width of the binary code of the column coordinate is then stored in the read address register 4 for RMB1 and the read address register 5 for RMB3. First, according to the read address of RMB0, data rdata is read from RMB0 (i.e., information of four historical events is read (t r0 ,x r0 )、(t r1 ,x r1 )、(t r2 ,x r2 )、(t r3,x r3 ), where (t r0 ,x r0 ) represents the first historical event read, and the rest are similar) and are sent to a corresponding EDU. After rdata is transmitted to the EDU, the EDU first determines whether the current input event is associated with the four historical events by comparing the column coordinates and timestamps of the current input event with the historical events in rdata, and then sums and counts the number of related events. Since the current input event also needs to be stored in RMB0, and it is necessary to read the data rdata from the read address in RMB0 and then store the information of the current input event in the address of RMB0, it is necessary to temporarily store the address of RMB0 through the write address register 3, and after reading the data rdata from the address in RMB0, the write address register 3 sends the address of RMB0 to RMB0, so that RMB0 stores the information of the current input event according to the address. Since four historical events have already been stored in the read address in RMB0, if the information of the current input event is stored in the address, the information of one historical event in the address needs to be replaced. Therefore, the storage update unit 8 can read the index of the latest event to be replaced (for example, 11) from a storage unit corresponding to the address in RMB0 in a wpt0 that corresponds one-to-one with RMB0, and after adding 1 to the index of the latest event to be replaced, store the updated latest event to be replaced (for example, 00) in the storage unit, and send the updated latest event to be replaced (for example, 00) to RMB0, so that RMB0 replaces the row coordinates and timestamp of the first historical event stored in the address with the row coordinates and compressed timestamp of the current input event, so that the data currently stored in the address of RMB0 is (t e ,x e )、(t r1 ,x r1 )、(t r2 ,x r2 )、(t r3 ,x r3). When the output generated by the EDU corresponding to RMB0 in the first stage is not zero, it means that there is a historical event related to the current input event in the address of RMB0, then the feedback signal register 6 sends a control signal to RMB1 and RMB3. After RMB1 and RMB3 respectively receive the control signal and the read address sent by the read address registers 4 and 5, they will not read the data in the read address sent by the read address registers 4 and 5. When the output generated by the EDU corresponding to RMB0 in the first stage is zero, it means that there is no historical event related to the current input event in the address of RMB0, then the feedback signal register 6 sends a control signal to RMB1 and RMB3. Send a control signal. When RMB1 and RMB3 each receive the control signal and the read address sent by the read address registers 4 and 5, they read the data in the read address sent by the read address registers 4 and 5. Specifically, similar to the principle of RMB0 mentioned above, RMB3 and RMB1 each send the read data (rdatap and rdatan) to the corresponding EDU. Each EDU determines whether the adjacent rows of the input event contain related events, and then calculates the sum. The three EDUs each output three quantities. The adder 7 adds the outputs of the three EDUs and outputs the total number of all related events in the three storage blocks, cr_event_row. The working method of CDM is the same as RDM, and it outputs the total number of all related events in the three storage blocks, cr_event_rol. The outputs of RDM and CDM are added by the adder module 1 and sent to the comparator module 2. The comparator module 2 compares the sum output by the adder module 1 with the threshold N CR Compare and output the judgment result of whether the current input event is noise or event.

[0060] The CLF proposed in this paper adopts a multi-level cache-like architecture with a low storage space complexity of O(m+n). Furthermore, the filter further reduces storage overhead by compressing the event timestamp storage bit width. This filter can be implemented on an FPGA, achieving not only excellent noise reduction but also low latency, providing a low-cost solution for acquiring high-quality DVS event data in real time. Furthermore, the filter can be adapted to different usage environments by varying parameter configurations. Under specific parameters, it can degenerate into BAF, STCF, or RCF, encompassing the functionality of previous filters and offering wider applicability.

[0061] The technical effects of the present invention are further illustrated by experiments below.

[0062] This experiment implements CLF on Xilinx's low-cost XC7A35T FPGA (Artix-7 series), with a clock frequency of 100MHz and a delay time of only 50ns. Specifically, the meanings of the different parameters used in the experiment are as follows: Figure 6As shown, it should be noted that although Figure 6 The meanings of some parameters in the text are different from those described above, but their actual meanings are the same. RM / N CM 、s RM 、s CM and BW T , conducted multiple experiments to match different requirements. All experiments support DVS with a resolution of 1280×800. The resource utilization and power consumption of FPGA implementation of CLF under different parameters are listed in Figure 7 In all implementations, the configuration is N RM / N CM =4, S RM =4, S CM =4 and BW T Configurations with 8-bit timestamps and 32-bit timestamps require the most resources, but are still achievable. Furthermore, an 8-bit timestamp reduces resources by over 40% compared to a 32-bit timestamp. Overall, the design is resource-efficient, making it suitable for low-end FPGAs or ASICs. Total power consumption for different configurations ranges from 125mW to 210mW, enabling low-power operation.

[0063] Due to the characteristics of DVS, the number of events output by DVS is much larger than the image frames generated by traditional CIS (image sensors) in the same scene and recording time. In general scenarios, tens of millions of events will be generated in a few seconds using CeleX-V, and it is almost impossible to annotate the event data to provide real labels. In view of this problem, the present invention first experiments with simulated event data, which can be well controlled to be noise-free or noisy. The present invention uses the v2e simulator, which synthesizes realistic DVS data from traditional frame-based videos through an accurate DVS pixel model, including the non-ideal characteristics of DVS. The present invention selected four videos provided by v2e, namely box-moving-2, box-moving-white, pendulum and gemma-hpe, covering scenes from low complexity to high complexity. The resolution of the first two videos is 800×600, and the resolution of the last two videos is 346×260. The present invention adjusted the simulator parameters to obtain clean and noisy data with different noise-to-signal ratios. The visualization of clean and noisy event data generated using the v2e simulator is shown in Figure 2. Figure 8 shown.

[0064] The present invention conducts a comprehensive noise reduction experiment by modifying the design parameters of CLF. In this case, DVS noise reduction is regarded as a classification problem, using precision (P), recall (R) and accuracy (A) as evaluation indicators.

[0065] Figure 9 Shows the different S CM and S RM The noise reduction effect under different values, that is, whether to use RDM and CDM at the same time, and the number of events that can be stored in each storage block. Figure 9 It can be seen that it is impossible to determine which one is more effective when using only RDM or CDM, that is, event denoising has directionality. The present invention believes that this is related to the directionality of the trajectory of the moving object. The present invention also observes that under a certain degree of resource limitation (S CM +S RM =4), using both RDM and CDM (S CM =2, S RM =2) may not guarantee the best results (for example, on the gemma-hpe dataset), but it can achieve balanced performance for various scenarios. CM =4, S RM =4) can improve the performance of CLF. Figure 10 Shows the time threshold T th The impact on the noise reduction results. Figure 10 As can be seen in the figure, with the increase of T th As the T increases, the performance gradually improves and stabilizes, or even decreases slightly. For the dataset used, the T th That's enough. Figure 11 Shows D th and N CR The impact on the denoising results. Generally speaking, a 5×5 spatial window (D th =2) than 3×3(D th =1) can achieve higher accuracy. On the contrary, increasing N CR This usually results in a decrease in accuracy. However, this does not mean that larger N CR Completely useless. In fact, a stricter standard is less likely to misjudge noise as a valid signal. From another perspective, N CR =1 is effective for typical scenarios.

[0066] The present invention selects the existing background activity filter (BAF) and the filter with storage space complexity of O(m+n) (RCF) as the comparison objects, which represent the designs with space complexity of O(mn) and O(m+n) respectively. It should be noted that for fairness, BAF and RCF are both implemented in software, because the hardware implementation of the same design may vary greatly, and the implementation of other solutions by the present invention may not be as efficient as the original version. Even so, the hardware overhead of different designs can be roughly estimated by their space complexity. For all methods, N CRBoth are set to 1, which has been proven to be effective in the previous text. In this case, the existing spatiotemporal correlation filter (STCF) is the same as BAF, so the present invention ignores it. Figure 12 The denoising results of different videos using different spatiotemporal correlation filters are shown. T and S CM / S RM , four different configurations of CLF were used. It can be observed that the performance of RCF is significantly lower than that of BAF and the spatiotemporal correlation filter (CLF) of the present invention, especially in complex scenarios. BAF and the present invention each have their own advantages. For simple scenarios such as box-moving-2 and box-moving-white, BAF obtains better results. In the other two complex cases, the CLF proposed by the present invention performs better in accuracy. The present invention believes that the reason is that compared with BAF, the limited storage capacity of the CLF proposed by the present invention excludes certain noise, thereby avoiding mistaking it for events. For different configurations of RCF, generally speaking, the more bits the timestamp represents and the more events the storage unit stores, the better the performance. However, for most benchmarks, the performance gap between 8-bit and 32-bit timestamps is negligible, and sometimes 8-bit timestamps perform even better, such as in the pendulum scenario. Therefore, it can be seen that reducing the bit width of the timestamp is a good way to achieve a compact design on the chip.

[0067] The present invention further verifies the performance of the proposed CLF using recorded DVS data. The present invention uses a CeleX-V DVS camera for data acquisition. The sensor resolution is 1280×800, which can be supported by the FPGA implementation of the RCF. Figure 13 As shown in Figure 1, the present invention selects four scenes, including vehicles traveling on a highway (Outdoor-1), a pedestrian bridge across the road (Outdoor-2), waving hands (Indoor-1), and an office environment (Indoor-2). The first three scenes were recorded with a stationary camera, and the last one was recorded with a moving camera. 40.62M, 4.38M, 42.10M, and 65.23M events were collected respectively. Then, BAF, RCF, and the CLF proposed by the present invention were applied to these data for noise reduction, and the same time threshold T was used for all of them. th and a 3×3 spatial window. For the CLF proposed in this invention, the configuration is N RM / N CM =4, S RM =4, S CM =4, BW T= 8. The CLF proposed in the present invention removed 36.95%, 76.33%, 63.46% and 55.92% of the events respectively. Figure 13 A visualization of the denoising results is also shown. Figure 13 As can be seen, the noise in the recorded data was effectively removed. However, the quality of the visualized image is difficult to judge. Furthermore, there is no ground truth value as an evaluation criterion to quantitatively assess the performance of different methods. Therefore, the present invention uses a pre-trained YOLOv5 model to perform car detection on Outdoor-1, which can, to some extent, reflect the effectiveness of noise reduction. The mAPs of BAF, RCF, and the proposed CLF are 0.476, 0.453, and 0.486, respectively. These data show that the proposed CLF also outperforms previous spatiotemporal correlation filters on real DVS data.

[0068] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0069] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0070] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0071] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A spatiotemporal correlation filter for denoising a dynamic visual sensor, characterized in that: include: Row noise reduction module, column noise reduction module and detection unit; the row noise reduction module includes N RM row banks, each with storage blocks, each storage block stores S RM events, N RM for The i-th row memory bank is used to store the column coordinates and N RM The event where the value obtained by rounding up after performing the modulo operation is i, and the value of i ranges from 0 to N RM -1; the column noise reduction module includes N CM column banks, each with storage blocks, each storage block stores S CM events, N CM for The jth column memory is used to store the row coordinates and N CM The event where the value obtained by rounding up after the modulo operation is j, where j is a value between 0 and N CM -1; s×q is the preset spatial window used for filtering; The row noise reduction module is used to select a target row storage body and a target storage block of the target row storage body according to the column coordinates, row coordinates and timestamp of the current event output by the DVS with a resolution of m×n, read historical events from the target storage block, and calculate the noise reduction result according to the spatial correlation threshold D. th and time-dependent threshold T th Determine the number of historical events cr_event_row related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, row coordinates, and polarity of the current event into a corresponding row memory bank; The column noise reduction module is used to select a target column storage body and a target storage block of the target column storage body according to the row coordinates, column coordinates and timestamp of the current event output by the DVS, read historical events from the target storage block, and calculate the noise reduction result according to the spatial correlation threshold D. th and time-dependent threshold T th Determine the number of historical events cr_event_col related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, column coordinates, and polarity of the current event in a corresponding column memory bank; The detection unit is used to compare the sum of the number of historical events cr_event_row and the number of historical events cr_event_col with the threshold value N of the number of time-space related events. CR Comparison is performed to detect whether the current event is noise.

2. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 1, characterized in that: The row noise reduction module is specifically configured to set the time-related threshold T th The binary code of the current event timestamp is aligned with the low bit of the binary code, and the bit width BW to be compressed to the preset coefficient a and the binary code of the current event timestamp is T , determine the time-related threshold T th The number of reserved bits in the binary code BW TH , the time-related threshold T th High BT in binary code TH bit as the time-dependent threshold T th BW TH bits to be reserved, and the binary code of the timestamp of the current event is related to the time-related threshold T th BW TH The lowest bit of the reserved bits is used as the starting bit, and starting from the starting bit, continuous BW is selected along the high bit direction. T bits, serving as the compressed timestamp of the current event.

3. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 1, characterized in that: The row noise reduction module further comprises: a first control unit, a first write address register, a first adder, a first storage control unit, N RM first event decision units, wherein the i-th first event decision unit is correspondingly connected to the i-th row memory bank; The first control unit is configured to determine, based on the column coordinates of the current event output by the DVS, an index of a row storage bank corresponding to the current event as the index of a first target row storage bank, and determine an index of a storage block to be read from the first target row storage bank; and determine, based on two adjacent column coordinates of the column coordinates of the current event, indexes of two corresponding row storage banks as the indexes of a second target row storage bank and a third target row storage bank, respectively, and determine the indexes of the storage block to be read from the second target row storage bank and the third target row storage bank, respectively, to obtain indexes of three target row storage banks and an index of a target storage block in each target row storage bank; and store the index of the target storage block of the first target row storage bank into the first write address register; Each of the three target row storage banks is used to store the S stored in its own target storage block. RM The information of each historical event is sent to the first event decision unit connected to itself; The first event decision unit connected to each target row storage body is used to determine the event based on the row coordinates and timestamp of the current event and the spatial correlation threshold D th , the time-related threshold T th and the S RM The information of historical events is used to determine the number of historical events related to the current event, and a sub-number is obtained; The first storage control unit is configured to generate an index of an event to be replaced by the current event in the target storage block of the first target row storage bank; The first write address register is used to store the index of the target storage block of the first target row storage bank, and send the index of the target storage block to the first target row storage bank in a time period next to the time period in which the first target row storage bank reads data; The first target row storage body is further configured to compress the bit width of the timestamp of the current event, and replace the row coordinates, polarity, and timestamp of the event to be replaced with the row coordinates, polarity, and timestamp of the event to be replaced, so as to store the current event in the target storage block; The first adder is used to sum the three sub-quantities obtained by the three first event decision units correspondingly connected to the three target row storage bodies to obtain a historical event quantity cr_event_row related to the current event.

4. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The three first event decision units correspondingly connected to the three target row memory banks operate synchronously.

5. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: Each row memory bank or column memory bank adopts dual-port storage for concurrent reading or concurrent reading and writing; the row noise reduction module further comprises: a first read address register, a second read address register and a feedback signal register; the first read address register being configured to store an index of the target storage block in the second target row storage bank, and to send the index of the target storage block in the second target row storage bank to the second target row storage bank in a time period next to the time period in which the current event is stored in the first target row storage bank, so that the second target row storage bank obtains an address for reading data in the next time period; the second read address register being configured to store an index of the target storage block of the third target row storage bank, and to send the index of the target storage block of the third target row storage bank to the third target row storage bank in a time period next to the time period in which the current event is stored in the first target row storage bank, so that the third target row storage bank obtains an address for reading data in the next time period; The feedback signal register is used to store a control signal and send the control signal to the second target row storage body and the third target row storage body. The control signal is used to control the second target row storage body and the third target row storage body not to read data from the target storage block when the value of a sub-number generated by the first event decision unit corresponding to the first target row storage body is not 0, and to control the second target row storage body and the third target row storage body to read data from the target storage block when the value of a sub-number generated by the first event decision unit corresponding to the first target row storage body is 0.

6. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The first control unit is specifically configured to use N RM Perform a modulo operation on the column coordinates of the current event and then round it up to get an index of a row storage bank. The row storage bank corresponding to the index is used as the first target row storage bank. The column coordinates of the current event are compared with N RM as the index of the target storage block of the first target row storage bank; And, adding 1 to the column coordinate of the current event through an adder to obtain an adjacent column coordinate of the column coordinate of the current event, subtracting 1 from the column coordinate of the current event through a subtractor to obtain another adjacent column coordinate of the column coordinate of the current event, and for each adjacent column coordinate of the column coordinate of the current event, using N RM After performing a modulo operation on the adjacent column coordinates and then rounding up, an index of a row memory bank is obtained, and the row memory bank corresponding to the index is used as the second target row memory bank or the third target row memory bank, and the adjacent column coordinates are compared with N RM The quotient of is used as the index of the target storage block of the second target row storage bank or the third target row storage bank.

7. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 6, characterized in that: N RM =N CM =S RM =S CM =4, the index of the first target row storage bank is the binary code BW of the column coordinate of the current event y The lowest two bits of the first target row memory bank's target memory block index are the binary code BW of the column coordinates of the current event y The remaining BW y1 -2 bits; the index of the second target row memory bank is the binary code BW of an adjacent column coordinate of the column coordinate of the current event y1 The lowest two bits of the second target row memory bank, the index of the target memory block is the binary code BW y1 The remaining BW y1 -2 bits; the index of the third target row memory bank is the binary code BW of another adjacent column coordinate of the column coordinate of the current event y2 The lowest two bits of the target memory block of the third target row memory bank are the binary code BW y3 The remaining BW y1 -2 digits.

8. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The first storage control unit includes: a storage update unit and an N RM Each scratchpad memory wpt0 contains The zth storage unit of the i-th note memory wpt0 is used to store the index of the latest event to be replaced in the zth storage block of the i-th row storage body, and the value of z ranges from 0 to The storage update unit is used to obtain the index of the latest event to be replaced from the storage unit corresponding to the target storage block of the first target row storage body, and add 1 to the obtained index of the latest event to be replaced to obtain the updated index of the latest event to be replaced, and then determine whether the updated index of the latest event to be replaced is greater than S RM ; If so, replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with the index of the latest event to be replaced after the update; if not, set the index of the latest event to be replaced after the update to 0, and replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with 0.

9. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The information of an event includes: row coordinates and timestamp; The first event decision unit connected to each target row storage bank is specifically configured to compare the row coordinates of the current event with the S RM The row coordinates of each historical time in the historical events are subtracted, and the row coordinate difference is compared with the spatial correlation threshold D th Compare the results to obtain a first comparison result with a value of 0 or 1, and compare the timestamp of the current event with the timestamp of the S RM The timestamp difference of each historical time in the historical events is calculated, and the timestamp difference is compared with the time-related threshold T th Compare and obtain a second comparison result with a value of 0 or 1, and then obtain a result value of 0 or 1 after ANDing the first comparison result and the second comparison result. RM S that corresponds to each historical event RM After adding the result values, a sub-number of historical events related to the current event is obtained.

10. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 1, characterized in that: The column noise reduction module further includes: a second control unit, a second write address register, a second adder, a second storage control unit, N CM second event decision units, wherein the j-th second event decision unit is correspondingly connected to the j-th column memory bank; The second control unit is configured to determine, based on the row coordinates of the current event output by the DVS, an index of a column storage bank corresponding to the current event as the index of a first target column storage bank, determine an index of a storage block to be read from the first target column storage bank, determine, based on two adjacent row coordinates of the row coordinates of the current event, indexes of two corresponding column storage banks as the index of a second target column storage bank and the index of a third target column storage bank, respectively, determine indexes of the storage block to be read from the second target column storage bank and the third target column storage bank, respectively, to obtain indexes of three target column storage banks and an index of a target storage block in each target column storage bank; and store the index of the target storage block of the first target column storage bank into the second write address register; Each of the three target column storage banks is used to store the S stored in its own target storage block. CM The information of each historical event is sent to the second event decision unit connected to itself; The second event decision unit connected to each target column storage body is configured to determine the event based on the column coordinates and timestamp of the current event, the spatial correlation threshold D th , the time-related threshold T th and the S CM The information of historical events is used to determine the number of historical events related to the current event, and a sub-number is obtained; The second storage control unit is configured to generate an index of an event to be replaced by the current event in the target storage block of the first target column storage bank; the second write address register is configured to store the index of the target storage block of the first target column storage bank, and send the index of the target storage block to the first target column storage bank in a time period next to a time period in which the first target column storage bank reads data; The first target column storage body is further configured to compress the bit width of the timestamp of the current event, and replace the column coordinates, polarity, and timestamp of the event to be replaced with the column coordinates, polarity, and timestamp of the event to be replaced, so as to store the current event in the target storage block; The second adder is configured to sum the three sub-quantities obtained by the three second event decision units correspondingly connected to the three target column storage bodies to obtain a historical event quantity cr_event_col related to the current event.

Images