Space-time correlation filter for noise reduction of dynamic vision sensor

Abstract

The invention discloses a time-space correlation filter for noise reduction of a dynamic visual sensor, which can filter events output by DVS (Digital Video Standard) in n rows and m columns, and comprises a row noise reduction module, a column noise reduction module and a detection unit, the row noise reduction module comprises NRM row memory banks, each row memory bank is provided with # imgabs 0 # memory blocks, each memory block stores SRM events, the NRM is # imgabs 1 # column noise reduction module comprises NCM column memory banks, each column memory bank is provided with # imgabs 2 # memory blocks, each memory block stores SCM events, and the NCM is # imgabs 3 # s * q is a preset space window; the detection unit sums the number of historical events related to the current event and output by the row noise reduction module and the column noise reduction module and then compares the sum with a preset threshold value so as to detect whether the current event is actually noise or an event. According to the method, the space complexity can be O (m + n), storage resources are saved, the noise reduction quality is good, and the time delay is low.

Description

Technical Field

[0001] The invention belongs to the technical field of dynamic vision sensor data processing, and in particular relates to a spatiotemporal correlation filter for dynamic vision sensor noise reduction. Background Art

[0002] Dynamic vision sensor (DVS) is a new type of neuromorphic imaging device, whose design is inspired by the biological retina. When the light intensity of the pixel receiving area changes beyond the positive or negative threshold, the pixel of DVS will asynchronously generate positive or negative events. Therefore, the output of DVS is usually expressed as a quaternion (x, y, t, p), corresponding to the row coordinates, column coordinates, timestamp and polarity of the pixel, respectively. The imaging process of DVS is carried out in a differential manner. In contrast, the traditional CMOS image sensor (CIS) is based on the principle of integration. The capacitance of each pixel accumulates charges that are positively correlated with the absolute light intensity during the exposure time, and then these charges are synchronously converted into digital signals frame by frame. The period of charge accumulation limits the frame rate of a typical CIS, generally to tens of frames. In addition, when the light intensity in a certain area is very high or very low, it is difficult for the corresponding pixels in the CIS to generate effective signals, resulting in a general dynamic range (<100dB). In contrast, DVS has the advantages of high temporal resolution at the microsecond level, high dynamic range of 120dB-160dB and low power consumption due to the characteristics of differential imaging and asynchronous readout. Therefore, DVS has received widespread attention and has been widely used in various computer vision tasks, such as object recognition, object tracking, video deblurring, video interpolation, etc.

[0003] Although the differential imaging method of DVS has advantages, it is also more susceptible to background activity (BA) caused by thermal noise and junction leakage current. These background noises significantly affect the quality of the output signal and increase the communication bandwidth and power consumption overhead. Therefore, noise reduction is of particular importance in DVS research. Because the movement of objects is continuous in space and time, while noise is random. Therefore, valid events show correlation in space and time, while noise does not have such correlation. Based on this feature, there are many methods to solve the noise reduction problem of DVS, such as spatiotemporal correlation filters. The basic principle is that if an event is associated with the previous event within a specific space and time window, it will pass the filter and be regarded as a valid signal. However, the previous spatiotemporal correlation filters did not solve the contradiction between noise reduction effect and storage overhead, that is, the filter with a spatial complexity of O(mn) has better performance, but the storage overhead is large and is not suitable for high-resolution DVS cameras. Although the filter with a spatial complexity of O(m+n) can save storage resources, the noise reduction performance is greatly reduced. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a spatiotemporal correlation filter for denoising of a dynamic vision sensor.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a spatiotemporal correlation filter for denoising a dynamic visual sensor, comprising: a row denoising module, a column denoising module and a detection unit; the row denoising module comprises N RM row banks, each with storage blocks, each storing S RM events, N RM for The i-th row bank is used to store the column coordinates and N RM The event whose value is i after modulo operation and rounded up, where i ranges from 0 to N RM -1; the column noise reduction module includes N CM column banks, each with storage blocks, each storing S CM events, N CM for The jth column memory bank is used to store the row coordinates and N CM The event whose value is j after modulo operation and rounded up, where j is between 0 and N CM -1; s×q is the preset spatial window used for filtering;

[0007] The row noise reduction module is used to select a target row storage body and a target storage block of the target row storage body according to the column coordinates, row coordinates and timestamp of the current event output by the DVS with a resolution of m×n, read historical events from the target storage block, and calculate the target event according to the spatial correlation threshold D th and time-dependent threshold T th Determine the number of historical events cr_event_row related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, row coordinates and polarity of the current event into a corresponding row storage body;

[0008] The column noise reduction module is used to select a target column storage body and a target storage block of the target column storage body according to the row coordinates, column coordinates and timestamp of the current event output by the DVS, read historical events from the target storage block, and calculate the target event according to the spatial correlation threshold D th and time-dependent threshold T thDetermine the number of historical events cr_event_col related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, column coordinates and polarity of the current event into a corresponding column storage body;

[0009] The detection unit is used to compare the sum of the number of historical events cr_event_row and the number of historical events cr_event_col with the threshold value N of the number of time-space related events. CR A comparison is performed to detect whether the current event is noise.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] The present invention utilizes a cache-like storage structure to store events by designing the number of storage bodies included in the row noise reduction module and the column noise reduction module in the filter, the number of storage blocks included in each storage body, and the relationship between each storage body and the horizontal and column coordinates of the stored events. For a DVS with a resolution of m×n, the space complexity can be made O(m+n). At the same time, the filter provided by the present invention further reduces the event storage overhead by compressing the timestamp storage bit width of the event, thereby saving storage resources. In addition, the present invention realizes noise reduction through the row noise reduction module, the column noise reduction module and the detection unit, which not only has good noise reduction quality, but also has the characteristics of low latency, thereby solving the contradiction between noise reduction effect and storage overhead. In addition, the present invention can be applied to different usage environments by changing different parameter configurations, thereby having a wider applicability.

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of an exemplary correspondence relationship between a DVS pixel array and a row memory bank and a column memory bank provided by an embodiment of the present invention;

[0014] Figure 2 It is a working principle diagram of the secondary cache architecture provided by an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of a bit width compression process of a timestamp provided by an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of a false alarm interval provided by an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of an exemplary architecture of a CLF provided in an embodiment of the present invention;

[0018] Figure 6It is the meaning of different parameters used in the experiments provided in the embodiments of the present invention;

[0019] Figure 7 It is the resource utilization and power consumption of CLF implemented by FPGA under different parameters provided by the embodiment of the present invention;

[0020] Figure 8 is a visualization diagram of clean and noisy event data generated using a v2e simulator provided by an embodiment of the present invention;

[0021] Fig. 9 The noise reduction result of CLF provided by the embodiment of the present invention is CM and RM Schematic diagram of the relationship between;

[0022] Fig.10 The noise reduction result of CLF provided by the embodiment of the present invention and T th Schematic diagram of the relationship between;

[0023] Fig.11 The noise reduction result of CLF provided by the embodiment of the present invention and D th and N CR Schematic diagram of the relationship between;

[0024] Fig.12 is a comparison diagram of denoising results of different videos using denoising algorithms using different spatiotemporal correlation filters provided in an embodiment of the present invention;

[0025] Fig.13 It is a visualization diagram of the recorded original DVS data and the CLF noise reduction result provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0027] The present invention provides a spatiotemporal correlation filter for denoising a dynamic visual sensor, comprising: a row denoising module (RDM), a column denoising module (CDM) and a detection unit. The input ends of the row denoising module and the column denoising module are both connected to the output end of a DVS with a resolution of m×n, and the output end of the row denoising module and the output end of the column denoising module are respectively connected to two input ends of the detection unit.

[0028] Here, the row noise reduction module contains N RM row banks, each with storage blocks, each of which is used to store S RM events, N RM for The i-th row memory bank is used to store the column coordinates and NRM The value obtained by rounding up after the modulo operation is i, and the column coordinates are the same as N RM The value obtained by rounding down after the modulo operation is the index of the storage block in the i-th row storage body used to store the row coordinates and timestamp of the event. The value of i ranges from 0 to N. RM -1. The column noise reduction module contains N CM column banks, each with storage blocks, each storing S CM events, N CM for The jth column memory bank is used to store the row coordinates and N CM The value obtained by rounding up after the modulo operation is j, and the row coordinates are the same as N CM The value obtained by rounding down after the modulo operation is the index of the storage block in the j-th column storage body used to store the column coordinates and timestamp of the event. The value of j ranges from 0 to N. CM -1; s×q is the preset spatial window used for filtering, Indicates a round-up operation. RM and S CM Both are positive integers greater than 1, and the values ​​of s and q can be the same or different.

[0029] In the present invention, each row storage body or column storage body adopts dual-port storage to perform concurrent reading or concurrent reading and writing.

[0030] For example, Figure 1 is an exemplary correspondence between the DVS pixel array and the row memory bank and column memory bank. Figure 1 As shown, when N RM =N CM=4, the row noise reduction module contains 4 row storage bodies, namely RMB0, RMB1, RMB2, and RMB3, and the column noise reduction module contains 4 column storage bodies, namely CMB0, CMB1, CMB2, and CMB3, among which the events of the pixels in the 0th, 4th, 8th, and 12th rows of the DVS pixel array will be stored in RMB0, the row coordinates and timestamps of the events of the pixels in the 1st, 5th, 9th, and 13th rows will be stored in RMB1, and the row coordinates and timestamps of the events of the pixels in the 2nd, 6th, and 10th rows will be stored in RMB. In B2, the row coordinates and timestamps of events of pixels in the 3rd, 7th, and 11th rows will be stored in RMB3. Similarly, the column coordinates and timestamps of events of pixels in the 0th, 4th, 8th, and 12th columns from the DVS pixel array will be stored in CMB0, the column coordinates and timestamps of events of pixels in the 1st, 5th, 9th, and 13th columns will be stored in CMB1, the column coordinates and timestamps of events of pixels in the 2nd, 6th, and 10th columns will be stored in CMB2, and the column coordinates and timestamps of events of pixels in the 3rd, 7th, and 11th columns will be stored in CMB3. The architecture of each row bank is similar to S RM Way-set associative cache. A storage block is like a cache line and can be read and written by address. If the column coordinate of an event is y e , the row coordinates and timestamp of the event will be stored at address The first row in the memory bank Each memory block in each row memory bank can store S RM Obviously, the mapping relationship between the column coordinates of the event and the row storage and the storage blocks in the row storage is clear, so there is no need to store the column coordinates of the event in the row storage. Similarly, if the row coordinates of an event are x e , the column coordinates and timestamp of the event will be stored at address The first column in the memory bank Each memory block in each column memory bank can store S CM Obviously, the mapping relationship between the row coordinates of the event and the column storage and the storage blocks in the column storage is clear. Therefore, there is no need to store the row coordinates of the event in the column storage. Therefore, the event information stored in the storage block in RMB is (x e ,t e ,p e ), which are the row coordinate, timestamp, and polarity of the event (1 or 0), respectively. For example, Figure 1 The x0, t0, p0 in the CMB is the first event stored in a storage block of RMB, and x1, t1, p1 is the second event stored in the storage block of RMB, and so on. The event information stored in the storage block in CMB is (y e ,t e ,p e),For example, Figure 1 The y0, t0, p0 in the table is the first event stored in a storage block of the CMB, and the y1, t1, p1 is the second event stored in the storage block of the CMB, and so on. e need bits are stored, y e need The DVS outputs the event timestamp t e Generally 32 bits or more, p e Usually only 1 bit is required.

[0031] In the present invention, the row noise reduction module is used to output the column coordinate y of the current event according to the DVS with a resolution of m×n. e , row coordinate x e and timestamp t e , select the target row storage body and the target storage block of the target row storage body, read the historical events from the target storage block, and calculate the spatial correlation threshold D th and time-dependent threshold T th Determine the number of historical events related to the current event cr_event_row, compress the bit width of the timestamp of the current event, and convert the compressed timestamp and row coordinates of the current event into e The column noise reduction module is used to store the row coordinate x of the current event output by DVS. e , column coordinate y e Sum and timestamp, select the target column storage body and the target storage block of the target column storage body, read the historical events from the target storage block, and calculate the spatial correlation threshold D th and time-dependent threshold T th Determine the number of historical events cr_event_col related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, column coordinates and polarity of the current event in the corresponding column storage body. Obviously, the row noise reduction module and the column noise reduction module in the present invention have the same principle, but process different data.

[0032] The detection unit is used to compare the sum of the number of historical events cr_event_row and the number of historical events cr_event_col with the threshold number of spatiotemporal related events N CR Compare to detect whether the current event is noise.

[0033] Here, the threshold number of spatiotemporal related events N CR It can be a preset value, and can be pre-set according to actual needs.

[0034] Here, the detection unit may include an adder module and a comparator module, wherein two input ends of the adder module are respectively connected to the output ends of the row noise reduction module and the column noise reduction module, the output end of the adder module is connected to an input end of the comparator module, and the other input end of the comparator module is used to input the threshold value N of the number of spatiotemporal related events. CR The output end of the comparator module is used as the output end of the spatiotemporal correlation filter provided by the present invention. The adder module is used to sum the number of historical events cr_event_row and the number of historical events cr_event_col to obtain the total number, and the comparator module is used to compare the total number with N CR The size relationship is determined based on the size relationship to determine whether the current event is noise or an event, thereby deciding whether to output the current event, thereby achieving filtering.

[0035] In some embodiments, the row noise reduction module further comprises: a first control unit, a first write address register, a first adder, a first storage control unit, N RM First event decision units (EDUs), wherein the i-th first event decision unit is connected to the i-th row storage body. The column noise reduction module also includes: a second control unit, a second write address register, a second adder, a second storage control unit, N CM The j-th second event decision unit is connected to the j-th column storage body. It should be noted that the "first" and "second" here are only used to distinguish the same modules, units or devices in the row noise reduction module and the column noise reduction module. In other words, the processing principle of the corresponding unit, device or module in the row noise reduction module is the same as the processing principle of the corresponding unit, device or module in the column noise reduction module, but the processed data and the obtained data are different.

[0036] Here, the first control unit is used to determine the index of a row storage body corresponding to the current event as the index of the first target row storage body according to the column coordinates and timestamp of the current event output by the DVS, and determine the index of the storage block to be read from the first target row storage body, and determine the indexes of the corresponding two row storage bodies according to the two adjacent column coordinates of the column coordinates of the current event, respectively as the index of the second target row storage body and the index of the third target row storage body, and determine the index of the storage block to be read from the second target row storage body and the third target row storage body, respectively, so as to obtain the indexes of the three target row storage bodies and the index of the target storage block in each target row storage body; and generate a write enable signal for the first target row storage body, and store the write enable signal of the first target row storage body into the first write signal register. Each of the three target row storage bodies is used to store the S stored in its own target storage block. RMThe first event decision unit connected to each target row storage body is used to determine the row coordinates and timestamp of the current event and the spatial correlation threshold D according to the row coordinates and timestamp of the current event. th , time-related threshold T th and S RM The first target row storage body is used to store the index of the target storage block of the first target row storage body, and send the index of the target storage block to the first target row storage body in the next time period of the time period in which the first target row storage body reads data. The first target row storage body is also used to compress the bit width of the timestamp of the current event, and use the row coordinates, polarity and compressed timestamp of the current event to replace the row coordinates, polarity and timestamp of the event to be replaced, so as to store the current event in the target storage block. The first adder is used to sum the three sub-quantities obtained by the three first event decision units corresponding to the three target row storage bodies to obtain a historical event number cr_event_row related to the current event.

[0037] Specifically, the first control unit is used to adopt N RM The column coordinate y of the current event e After performing the modulo operation and rounding up, we get the index of a row storage body. The row storage body corresponding to the index is used as the first target row storage body. The column coordinate y of the current event is e With N RM as the index of the target storage block of the first target row storage body; and, by adding 1 to the column coordinate of the current event through an adder, an adjacent column coordinate y of the column coordinate of the current event is obtained. e +1, subtract 1 from the column coordinate of the current event through the subtractor to obtain another adjacent column coordinate y of the column coordinate of the current event e -1, and for each adjacent column coordinate of the column coordinate of the current event (i.e. y e +1 or y e -1), using N RM After performing a modulo operation on the adjacent column coordinates and then rounding up, an index of a row storage body is obtained, and the row storage body corresponding to the index is used as the second target row storage body or the third target storage body, and the adjacent column coordinates are compared with N RM The quotient of is used as the index of the target storage block of the second target row storage body or the third target row storage body.

[0038] Specifically, the first storage control unit includes: a storage update unit and an N RMEach scratch pad memory wpt0 contains The zth storage unit of the i-th note memory wpt0 is used to store the index of the latest event to be replaced in the zth storage block of the i-th row storage body in a one-to-one correspondence, and the value of z ranges from 0 to The storage update unit is used to obtain the latest index of the event to be replaced from the storage unit corresponding to the target storage block of the first target row storage body, and add 1 to the obtained latest index of the event to be replaced to obtain the updated latest index of the event to be replaced, and then determine whether the updated latest index of the event to be replaced is greater than S RM ; If so, replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with the index of the updated latest event to be replaced; if not, set the updated index of the latest event to be replaced to 0, and replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with 0.

[0039] Specifically, the information of an event includes: row coordinates and timestamp. The first event decision unit connected to each target row storage body is specifically used to compare the row coordinates of the current event with S RM The row coordinates of each historical time in the historical events are subtracted, and the row coordinate difference is compared with the spatial correlation threshold D th The comparison is performed to obtain a first comparison result with a value of 0 or 1, and the timestamp of the current event is compared with S RM The timestamp of each historical event in the historical event is subtracted, and the timestamp difference is compared with the time-related threshold T th Compare and obtain a second comparison result with a value of 0 or 1. After ANDing the first comparison result and the second comparison result, obtain a result value of 0 or 1. RM S RM After adding the result values, a sub-number of historical events related to the current event is obtained.

[0040] Exemplarily, the first event decision unit includes 2S RM Subtractors, 2S RM A comparator, S RM AND gates and an adder, wherein the first subtractor is used to make a difference between the row coordinates of the first historical event and the row coordinates of the current event, the second subtractor is used to make a difference between the timestamp of the first historical event and the timestamp of the current event, and the first comparator is used to compare the column coordinate difference output by the first subtractor with the spatial correlation threshold D th The second comparator is used to compare the timestamp output by the second subtractor with the time-related threshold T thThe first AND gate is used to compare the comparison results output by the first comparator and the second comparator to obtain a result value of 0 or 1. The corresponding relationship between the remaining subtractors, comparators and AND gates can be obtained by reasoning in sequence. RM By adding the result values ​​output by the AND gates, a sub-number of historical events related to the current event can be obtained.

[0041] In some embodiments, three first event decision units correspondingly connected to three target row memory banks operate synchronously.

[0042] In some embodiments, the row noise reduction module further includes: a first read address register, a second read address register, and a feedback signal register. The first read address register is used to store the index of the target storage block in the second target row storage body, and in the next time period after the time period of the current event is stored in the first target row storage body, the index of the target storage block of the second target row storage body is sent to the second target row storage body, so that the second target row storage body obtains the address of the read data in the next time period. The second read address register is used to store the index of the target storage block of the third target row storage body, and in the next time period after the time period of the current event is stored in the first target row storage body, the index of the target storage block of the third target row storage body is sent to the third target row storage body, so that the third target row storage body obtains the address of the read data in the next time period. The feedback signal register is used to store a control signal and send a control signal to the second target row storage body and the third target row storage body. The control signal is used to control the second target row storage body and the third target row storage body not to read data from the target storage block when the value of a sub-number generated by the first event decision unit corresponding to the first target row storage body is not 0, and to control the second target row storage body and the third target row storage body to read data from the target storage block when the value of a sub-number generated by the first event decision unit corresponding to the first target row storage body is 0. In this way, if the output of the EDU corresponding to the first target storage body is not zero (that is, it means that there is a related event), the corresponding read operation of the second target storage body and the third target storage body will be cancelled, otherwise, the corresponding read operation will continue. This read cancellation technology is very beneficial for reducing energy consumption, especially in ASIC implementation, so that power consumption can be further reduced.

[0043] Theoretically, for a 3×3 spatial window (i.e., a filtering window), the probability that an input event is correlated with an event in the same row or column is higher. The present invention realizes a two-stage pipeline structure for storage access by selectively using the three registers, namely, the first read address register, the second read address register, and the feedback signal register, thereby achieving cycle delay and pipelining of data, and further realizing the executable read cancel operation.

[0044] The following Figure 2 , the principle of read cancel operation is further explained. Figure 2 As shown, it is assumed that four events e1 to e4 enter the CLF (the spatiotemporal correlation filter for DVS noise reduction proposed in the present invention is called a cache-like spatiotemporal filter (CLF)) in sequence, and the corresponding storage bodies to be accessed are RMB0, RMB0, RMB1 and RMB3 respectively. The storage access process is shown in the attached figure. Figure 2 As shown, considering the problem of concurrent reading or concurrent reading and writing, the present invention uses dual-port storage to meet the requirements of concurrent access. In the first stage of the pipeline processing e1, RMB0 is read; in the second stage, the storage block of RMB0 read in the first stage is updated, and at the same time, the storage blocks corresponding to RMB1 and RMB3 (respectively y e1 -1 and y e1 +1) is read. At the same time, the first stage reading of e2 continues, that is, reading RMB0. The subsequent read and write operations of the row storage block involve a similar process. In this process, if the output generated by the EDU in the first stage is not zero when processing a certain event (for example, e1), it means that there is a historical event related to event e1 in RMB0, then the read operations related to event e1 on the other two storage bodies will be canceled, but the read operations or write operations related to other events (for example, event e2) on these two storage bodies that occur at the same time will not be canceled; if the output generated by the EDU in the first stage is zero, it means that there is no historical event related to event e1 in RMB0, then the read operations related to event e1 on these two storage bodies will continue.

[0045] In some embodiments, the specific process of compressing the bit width of the timestamp of the current event by the first target row storage bank is as follows: th The binary code of the current event is aligned with the binary code of the timestamp of the current event, and the bit width BW to which the binary code of the timestamp of the current event needs to be compressed is calculated according to the preset coefficient a and the bit width BW T , determine the time-dependent threshold T th The number of reserved bits in the binary code BW TH ,Right now Set the time-dependent threshold T th High BW in binary code TH bit as a time-dependent threshold T th BW TH The reserved bits are used to convert the binary code of the timestamp of the current event into the time-related threshold T th BW THThe lowest bit of the reserved bits is used as the starting bit, and continuous BW is selected along the high bit direction starting from the starting bit. T bits, as the compressed timestamp of the current event.

[0046] Here, a is a value greater than 0 and less than 1, and the value of a can be set according to actual needs, which is not limited in the present invention.

[0047] For example, Figure 3 This is a schematic diagram of the principle of compressing the bit width of the timestamp. Figure 3 As shown, assuming that the time-dependent threshold T th (referred to as time threshold T th ) has 8 bits, and the timestamp to be compressed is T s The binary code has 32 bits, and the timestamp to be compressed is T s The bit width of the binary code to be compressed is 8, a is 30%, and the time threshold T th The binary code and the timestamp T to be compressed s After the binary code is aligned, the relevant values ​​are substituted into the above formula to obtain BW TH =3, then Figure 3 As shown, the time threshold T th The upper 3 bits are reserved, and the time threshold T th The [4:0] bits of the 10000 bits are discarded, thus achieving the time threshold T th After that, due to the time threshold T th The reserved bits are from 6th to 8th, and the time threshold T th The binary code and the timestamp T to be compressed s The binary codes are aligned, so the timestamp to be compressed is T s The 5th to 13th bits of the compressed data can be used as the timestamp T to be compressed. s The compressed binary code is used to reduce the bit width of the timestamp.

[0048] Since the timestamp bit width is limited, it may overflow and return to zero, which may cause two temporally unrelated events to be misjudged as temporally related. Figure 4 As shown, assuming T th is the range represented by the time threshold, T s is the timestamp overflow period, when the timestamp of subsequent input events falls within [n·T S ,n·T s +T th ] time range, the noise is mistakenly judged as a signal. Through experiments, it is found that when the time exceeds a certain limit, different T th The false positive rate tends to 0 under the value of Ts The bit width does not need to be very large. The present invention can reduce the storage overhead of the timestamp and ensure that the judgment result has a high accuracy through such a compression process.

[0049] Similar to the first control unit in the above-mentioned row noise reduction module, the second control unit is used to generate a row coordinate x of the current event output by the DVS. e , column coordinate y e and timestamp, determine the index of a column storage body corresponding to the current event as the index of the first target column storage body, and determine the index of the storage block to be read from the first target column storage body, and the two adjacent row coordinates (i.e., x e +1 and x e -1) determining the indexes of the corresponding two column storage bodies, which are used as the index of the second target column storage body and the index of the third target column storage body respectively, and determining the indexes of the storage blocks to be read from the second target column storage body and the third target column storage body respectively, so as to obtain the indexes of the three target column storage bodies and the index of the target storage blocks in each target column storage body; and generating a write enable signal of the first target column storage body, and storing the write enable signal of the first target column storage body into the second write signal register.

[0050] Similar to the target row storage body in the above row noise reduction module, each of the three target column storage bodies in the column noise reduction module is used to store the S stored in its own target storage block. CM The information of a historical event is sent to the second event decision unit connected to itself.

[0051] Similar to the first event decision unit in the row noise reduction module, the second event decision unit in the column noise reduction module connected to each target column storage body is used to determine the event based on the column coordinates and timestamp of the current event and the spatial correlation threshold D th , time-related threshold T th and S CM The information of historical events is used to determine the number of historical events related to the current event, and a sub-number is obtained.

[0052] Similar to the first storage control unit in the above-mentioned row noise reduction module, the second storage control unit in the column noise reduction module is used to generate the index of the event to be replaced in the target storage block of the first target column storage bank of the current event.

[0053] Similar to the first write address register in the above-mentioned row noise reduction module, the second write address register in the column noise reduction module is used to store the index of the target storage block of the first target column storage body, and send the index of the target storage block of the first target column storage body to the first target column storage body in the next time period of the time period in which the first target column storage body reads data.

[0054] Similar to the first target row storage body in the above-mentioned row noise reduction module, the first target column storage body in the column noise reduction module is also used to compress the bit width of the timestamp of the current event, and use the column coordinates, polarity and compressed timestamp of the current event to replace the column coordinates, polarity and timestamp of the event to be replaced, so as to store the current event in the target storage block.

[0055] Similar to the first adder in the above-mentioned row noise reduction module, the second adder in the column noise reduction module is used to sum the three sub-quantities obtained by the three second event decision units correspondingly connected to the three target column storage bodies to obtain a historical event quantity cr_event_col related to the current event.

[0056] Since the above units, devices or modules in the column noise reduction module have the same structure and principle as those of the row noise reduction module, the specific structures and functions of the above units, devices or modules in the column noise reduction module will not be described in detail.

[0057] In some embodiments, N RM =N CM =S RM =S CM =4, and the index of the first target row storage bank is the binary code BW of the column coordinate of the current event y The lowest two bits, the index of the target memory block of the first target row memory bank is the binary code BW of the column coordinate of the current event y The remaining BW y1 -2 bits; the index of the second target row storage bank is the binary code BW of an adjacent column coordinate of the column coordinate of the current event y1 The lowest two bits, the index of the target memory block of the second target row memory bank is the binary code BW y1 The remaining BW y1 -2 bits; the index of the third target row storage bank is the binary code BW of another adjacent column coordinate of the column coordinate of the current event y2 The lowest two bits, the index of the target memory block of the third target row memory bank is the binary code BW y3 The remaining BW y1 -2 bits. Here, when N RM =N CM =4, With direct access to y e The lowest two bits of the binary code are the same. With direct access to x e The lowest two bits of the binary code are the same, thus avoiding the complexity of modulo operation.

[0058] For example, Figure 5 It is in NRM =N CM =S RM =S CM =4, a schematic diagram of the architecture of the CLF provided by the present invention. Since the structures of the row noise reduction module and the column noise reduction module are similar, Figure 5 The noise reduction module is used as an example for specific demonstration. Figure 5 As shown, CLF includes a row noise reduction module RDM, a column noise reduction module CDM, an adder module 1 and a comparator module 2. The row noise reduction module RDM includes 4 row storage bodies RMB0~RMB3, 3 EDUs, a write address register 3, a read address register 4, a read address register 5, a feedback signal register 6, an adder 7, 4 wpt0s, and a storage update unit 8, and the registers are Figure 5 The control unit is not shown in its entirety, and only a subtractor 9 and an adder 10 included in the control unit are shown.

[0059] The following will be Figure 5 As an example, the working principle of the spatiotemporal correlation filter of the present invention is exemplified. Assume that the spatial correlation threshold D th Set to 1, that is, for the input event, it is necessary to check the historical events in its 3×3 spatial neighborhood. The current input event is represented by (x e ,y e ,t e ,p e ), assuming that y e The lowest two bits of are binary 00. First, y is calculated by subtractor 9 and adder 10 respectively. e -1 and y e +1, which are 11 and 01 respectively. Therefore, RMB0, RMB3 and RMB1 need to be read, and the read addresses of RMB0, RMB3 and RMB1 (i.e. the index of the storage block to be read) are y e ,y e -1 and y e +1 for the highest BW y -2 bit, BW y is the bit width of the binary code of the column coordinates. After that, the read address of RMB1 is stored in the read address register 4, and the read address of RMB3 is stored in the read address register 5. First, according to the read address of RMB0, the data rdata is read from RMB0 (that is, the information of the four historical events is read (t r0 ,x r0 )、(t r1 ,x r1 )、(t r2 ,x r2 )、(t r3,x r3 ), where (t r0 ,x r0 ) represents the first historical event read, and the rest are similar) are sent to a corresponding EDU. After rdata is transmitted to EDU, EDU first determines whether the current input event is associated with the four historical events by comparing the column coordinates and timestamps of the current input event with the historical events in rdata, and then sums and counts the number of related events. Since the current input event also needs to be stored in RMB0, and the information of the current input event needs to be stored in the address of RMB0 after reading the data rdata from the read address in RMB0, it is necessary to temporarily store the address of RMB0 through the write address register 3, and after reading the data rdata from the address in RMB0, the write address register 3 sends the address of RMB0 to RMB0, so that RMB0 stores the information of the current input event according to the address. Since four historical events have been stored in the read address in RMB0, if the information of the current input event is stored in the address, the information of one historical event in the address needs to be replaced. Therefore, the storage update unit 8 can read the index of the latest event to be replaced (for example, 11) from a storage unit corresponding to the address in RMB0 in a wpt0 that corresponds one-to-one with RMB0, and after adding 1 to the index of the latest event to be replaced, store the updated latest event to be replaced (for example, 00) in the storage unit, and send the updated latest event to be replaced (for example, 00) to RMB0, so that RMB0 replaces the row coordinates and timestamp of the first historical event stored in the address with the row coordinates and compressed timestamp of the current input event, so that the data currently stored in the address of RMB0 is (t e ,x e )、(t r1 ,x r1 )、(t r2 ,x r2 )、(t r3 ,x r3). When the output generated by the EDU corresponding to RMB0 in the first stage is not zero, it means that there is a historical event related to the current input event in the address of RMB0, then the feedback signal register 6 sends a control signal to RMB1 and RMB3. When RMB1 and RMB3 receive the control signal and the read address sent by the read address registers 4 and 5 respectively, they will not read the data in the read address sent by the read address registers 4 and 5 respectively. When the output generated by the EDU corresponding to RMB0 in the first stage is zero, it means that there is no historical event related to the current input event in the address of RMB0, then the feedback signal register 6 sends a control signal to RMB1 and RMB3. Send a control signal. When RMB1 and RMB3 each receive the control signal and the read address sent by the read address registers 4 and 5, they read the data in the read address sent by the read address registers 4 and 5. Specifically, similar to the principle of RMB0, RMB3 and RMB1 each send the read data (rdatap and rdatan) to the corresponding EDU. Each EDU determines whether the adjacent rows of the input event contain related events, and then calculates the sum. Thus, the three EDUs each output three quantities. The adder 7 adds the outputs of the three EDUs and outputs the total number cr_event_row of all related events in the three storage blocks. The working mode of CDM is the same as that of RDM, and it outputs the total number cr_event_rol of all related events in the three storage blocks. The outputs of RDM and CDM are added by adder module 1 and sent to comparator module 2. Comparator module 2 compares the sum output by adder module 1 with threshold value N. CR Compare and output the result of judging whether the current input event is noise or an event.

[0060] The CLF proposed in the present invention adopts a multi-level cache-like architecture with a low storage space complexity of O(m+n). At the same time, the filter further reduces the storage overhead by compressing the event timestamp storage bit width. The filter can be implemented on FPGA, not only has good noise reduction quality, but also has the characteristics of low latency, and provides a low-cost solution for real-time acquisition of high-quality DVS event data. In addition, the filter is suitable for different use environments by changing different parameter configurations, and can be degraded to BAF, STCF or RCF under specific parameters, covering the functions of previous filters and having wider applicability.

[0061] The technical effects of the present invention are further illustrated by experiments below.

[0062] This experiment implements CLF on Xilinx's low-cost XC7A35T FPGA (Artix-7 series), with a clock frequency of 100MHz and a delay time of only 50ns. Specifically, the meanings of the different parameters used in the experiment are as follows: Figure 6As shown, it should be noted that although Figure 6 The meanings of some parameters in the text are different from those described above, but their actual meanings are the same. RM / N CM 、s RM 、s CM and BW T , and conducted multiple experiments to match different requirements. All experiments support DVS with a resolution of 1280×800. The resource utilization and power consumption of FPGA implementation of CLF under different parameters are listed in Figure 7 In all implementations, the configuration is N RM / N CM =4, S RM =4, S CM =4 and BW T =32 configurations require the most resources, but are still achievable. In addition, 8-bit timestamps save more than 40% of resources compared to 32-bit timestamps. Overall, the design of the present invention is resource efficient, making it suitable for low-end FPGAs or ASICs. The total power consumption of different configurations ranges from 125mW to 210mW, achieving low-power operation.

[0063] Due to the characteristics of DVS, the number of events output by DVS is much larger than the image frames generated by traditional CIS (image sensors) in the same scene and recording time. In general scenarios, tens of millions of events are generated in a few seconds using CeleX-V, and it is almost impossible to annotate the event data to provide realistic labels. In view of this problem, the present invention first experiments with simulated event data, which can be well controlled to be noise-free or noisy. The present invention uses the v2e simulator, which synthesizes realistic DVS data from traditional frame-based videos through an accurate DVS pixel model, including the non-ideal characteristics of DVS. The present invention selects four videos provided by v2e, namely box-moving-2, box-moving-white, pendulum and gemma-hpe, covering scenes from low complexity to high complexity. The resolution of the first two videos is 800×600, and the resolution of the latter two videos is 346×260. The present invention adjusts the simulator parameters to obtain clean and noisy data with different noise-to-signal ratios. The visualization of clean and noisy event data generated using the v2e simulator is shown in the figure. Figure 8 shown.

[0064] The present invention conducts a comprehensive denoising experiment by modifying the design parameters of CLF. At this time, DVS denoising is regarded as a classification problem, and precision (P), recall (R) and accuracy (A) are used as evaluation indicators.

[0065] Fig. 9 Shows the different S CM and S RM The noise reduction effect under different values, that is, whether RDM and CDM are used at the same time, and the number of events that can be stored in each storage block. Fig. 9 It can be seen that it is impossible to determine which one is more effective when using only RDM or CDM, that is, event denoising has directionality. The present invention believes that this is related to the directionality of the trajectory of the moving object. The present invention also observes that under a certain degree of resource limitation (S CM +S RM =4), using both RDM and CDM (S CM =2, S RM =2) may not guarantee the best results (for example, on the gemma-hpe dataset), but it can achieve balanced performance for many scenarios. Increase the capacity of the storage block (S CM =4, S RM =4) can improve the performance of CLF. Fig.10 Shows the time threshold T th The impact on the noise reduction results. Fig.10 As can be seen in the figure, with the increase of T th As the T increases, the performance gradually improves and stabilizes, or even decreases slightly. For the dataset used, the T range of 200μs to 400μs th That's enough. Fig.11 Shows D th and N CR The impact on the denoising results. Generally speaking, a 5×5 spatial window (D th =2) than 3×3(D th =1) can achieve higher accuracy. On the contrary, increasing N CR This usually results in a decrease in accuracy. However, this does not mean that larger N CR Completely useless. In fact, a stricter standard is less likely to misjudge noise as a valid signal. From another perspective, N CR =1 is valid for typical scenarios.

[0066] The present invention selects the existing background activity filter (BAF) and the filter with storage space complexity of O(m+n) (RCF) as the comparison objects, which represent the designs with space complexity of O(mn) and O(m+n), respectively. It should be noted that for fairness, BAF and RCF are both implemented in software, because the hardware implementation of the same design may vary greatly, and the implementation of other schemes by the present invention may not be as efficient as the original one. Even so, the hardware overhead of different designs can be roughly estimated by their space complexity. For all methods, N CRAll are set to 1, which has been proven to be effective in the previous text. In this case, the existing spatiotemporal correlation filter (STCF) is the same as BAF, so the present invention ignores it. Fig.12 The denoising results of different videos using different spatiotemporal correlation filters are shown. T and S CM / S RM , four CLFs with different configurations were used. It can be observed that the performance of RCF is significantly lower than that of BAF and the spatiotemporal correlation filter (CLF) of the present invention, especially in complex scenarios. BAF and the present invention each have their own advantages. For simple scenarios such as box-moving-2 and box-moving-white, BAF obtains better results. In the other two complex cases, the CLF proposed by the present invention performs better in accuracy. The present invention believes that the reason is that the limited storage capacity of the CLF proposed by the present invention excludes certain noises compared with BAF, thereby avoiding mistaking them for events. For different configurations of RCF, in general, the more bits the timestamp represents and the more events the storage unit stores, the better the performance. However, for most benchmarks, the performance gap between 8-bit and 32-bit timestamps is minimal, and sometimes 8-bit timestamps perform even better, such as in the pendulum scenario. Therefore, it can be seen that reducing the bit width of the timestamp is a good way to achieve a compact design on the chip.

[0067] The present invention further verifies the performance of the CLF proposed in the present invention using recorded DVS data. The present invention uses a CeleX-V DVS camera for data acquisition. The resolution of the sensor is 1280×800, which can be supported by the FPGA implementation of the RCF. Fig.13 As shown in the figure, the present invention selects four scenes, including vehicles traveling on a highway (Outdoor-1), a pedestrian bridge across the road (Outdoor-2), waving (Indoor-1), and an office environment (Indoor-2). The first three scenes were recorded with a stationary camera, and the last one was recorded with a moving camera. 40.62M, 4.38M, 42.10M, and 65.23M events were collected respectively. Then, BAF, RCF, and the CLF proposed by the present invention were applied to these data for noise reduction, and the same time threshold T was used for all of them. th and a 3×3 spatial window. For the CLF proposed in the present invention, the configuration is N RM / N CM =4, S RM =4, S CM =4, BW T= 8. The CLF proposed in the present invention removes 36.95%, 76.33%, 63.46% and 55.92% of the events respectively. Fig.13 A visualization of the denoising results is also shown. Fig.13 It can be seen that the noise in the recorded data has been effectively removed. However, the quality of the visualized image is difficult to judge. In addition, there is no ground truth as an evaluation criterion to quantitatively evaluate the performance of different methods. Therefore, the present invention uses the pre-trained YOLOv5 model to perform car detection on Outdoor-1, which can reflect the effectiveness of noise reduction to some extent. The mAPs of BAF, RCF, and the CLF proposed in the present invention are 0.476, 0.453, and 0.486, respectively. These data show that the CLF proposed in the present invention is also superior to the previous spatiotemporal correlation filters on real DVS data.

[0068] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0069] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0070] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0071] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A spatiotemporal correlation filter for denoising a dynamic visual sensor, characterized in that: include: A row noise reduction module, a column noise reduction module and a detection unit; the row noise reduction module includes N RM row banks, each with storage blocks, each storing S RM events, N RM for The i-th row memory bank is used to store the column coordinates and N RM The event whose value is i after modulo operation and rounded up, where i ranges from 0 to N RM -1; the column noise reduction module includes N CM column banks, each with storage blocks, each storing S CM events, N CM for The jth column memory bank is used to store the row coordinates and N CM The event whose value is j after modulo operation and rounded up, where j is between 0 and N CM -1; s×q is the preset spatial window used for filtering; The row noise reduction module is used to select a target row storage body and a target storage block of the target row storage body according to the column coordinates, row coordinates and timestamp of the current event output by the DVS with a resolution of m×n, read historical events from the target storage block, and calculate the target event according to the spatial correlation threshold D th and time-dependent threshold T th Determine the number of historical events cr_event_row related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, row coordinates and polarity of the current event into a corresponding row storage body; The column noise reduction module is used to select a target column storage body and a target storage block of the target column storage body according to the row coordinates, column coordinates and timestamp of the current event output by the DVS, read historical events from the target storage block, and calculate the target event according to the spatial correlation threshold D th and time-dependent threshold T th Determine the number of historical events cr_event_col related to the current event, compress the bit width of the timestamp of the current event, and store the compressed timestamp, column coordinates and polarity of the current event into a corresponding column storage body; The detection unit is used to compare the sum of the number of historical events cr_event_row and the number of historical events cr_event_col with the threshold value N of the number of time-space related events. CR A comparison is performed to detect whether the current event is noise.

2. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 1, characterized in that: The row noise reduction module is specifically used to set the time-related threshold T th The binary code of the current event is aligned with the binary code of the timestamp of the current event in the low bit, and the binary code of the timestamp of the current event needs to be compressed to the bit width BW according to the preset coefficient a and the bit width BW T , determine the time-related threshold T th The number of reserved bits in the binary code BW TH , the time-related threshold T th High BT in binary code TH bit as the time-dependent threshold T th BW TH bits to be reserved, and the binary code of the timestamp of the current event is th BW TH The lowest bit of the reserved bits is used as the starting bit, and starting from the starting bit, continuous BW is selected along the high-bit direction. T bits, serving as the compressed timestamp of the current event.

3. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 1, characterized in that: The row noise reduction module further comprises: a first control unit, a first write address register, a first adder, a first storage control unit, RM first event decision units, wherein the i-th first event decision unit is connected to the i-th row storage body correspondingly; The first control unit is used to determine, according to the column coordinates of the current event output by the DVS, an index of a row storage body corresponding to the current event as the index of the first target row storage body, and determine the index of the storage block to be read from the first target row storage body, and determine the indexes of two corresponding row storage bodies according to two adjacent column coordinates of the column coordinates of the current event as the index of the second target row storage body and the index of the third target row storage body, respectively, and determine the indexes of the storage block to be read from the second target row storage body and the third target row storage body, respectively, so as to obtain the indexes of the three target row storage bodies and the index of the target storage block in each target row storage body; and store the index of the target storage block of the first target row storage body into the first write address register; Each of the three target row storage bodies is used to store the S stored in its own target storage block. RM The information of each historical event is sent to the first event decision unit corresponding to itself; A first event decision unit connected to each target row storage body is used to determine the event based on the row coordinates and timestamp of the current event and the spatial correlation threshold D th , the time-related threshold T th and the S RM information of historical events, determining the number of historical events related to the current event, and obtaining a sub-number; The first storage control unit is used to generate an index of an event to be replaced by the current event in the target storage block of the first target row storage body; The first write address register is used to store the index of the target storage block of the first target row storage body, and send the index of the target storage block to the first target row storage body in a time period next to the time period in which the first target row storage body reads data; The first target row storage body is further used to compress the bit width of the timestamp of the current event, and replace the row coordinates, polarity and timestamp of the event to be replaced with the row coordinates, polarity and timestamp of the event to be replaced, so as to store the current event in the target storage block; The first adder is used to sum the three sub-quantities obtained by the three first event decision units correspondingly connected to the three target row storage bodies to obtain a historical event quantity cr_event_row related to the current event.

4. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The three first event decision units correspondingly connected to the three target row storage bodies operate synchronously.

5. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: Each row storage body or column storage body adopts dual-port storage for concurrent reading or concurrent reading and writing; the row noise reduction module also includes: a first read address register, a second read address register and a feedback signal register; The first read address register is used to store the index of the target storage block in the second target row storage body, and send the index of the target storage block in the second target row storage body to the second target row storage body in a time period next to the time period in which the first target row storage body stores the current event, so that the second target row storage body obtains the address of the read data in the next time period; The second read address register is used to store the index of the target storage block of the third target row storage body, and send the index of the target storage block of the third target row storage body to the third target row storage body in a time period next to the time period in which the first target row storage body stores the current event, so that the third target row storage body obtains the address of the read data in the next time period; The feedback signal register is used to store a control signal and send the control signal to the second target row storage body and the third target row storage body. The control signal is used to control the second target row storage body and the third target row storage body not to read data from the target storage block when the value of a sub-number generated by the first event decision unit corresponding to the first target row storage body is not 0, and to control the second target row storage body and the third target row storage body to read data from the target storage block when the value of a sub-number generated by the first event decision unit corresponding to the first target row storage body is 0.

6. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The first control unit is specifically configured to use N RM The column coordinates of the current event are modulo-operated and then rounded up to obtain an index of a row storage body, and the row storage body corresponding to the index is used as the first target row storage body. The column coordinates of the current event are compared with N RM as the index of the target storage block of the first target row storage body; And, by adding 1 to the column coordinate of the current event through an adder, an adjacent column coordinate of the column coordinate of the current event is obtained, and by subtracting 1 from the column coordinate of the current event through a subtractor, another adjacent column coordinate of the column coordinate of the current event is obtained, and for each adjacent column coordinate of the column coordinate of the current event, N RM After performing a modulo operation on the adjacent column coordinates and then rounding up, an index of a row storage bank is obtained, and the row storage bank corresponding to the index is used as the second target row storage bank or the third target storage bank, and the adjacent column coordinates are compared with N RM The quotient of is used as the index of the target storage block of the second target row storage body or the third target row storage body.

7. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 6, characterized in that: N RM =N CM =S RM =S CM =4, the index of the first target row storage bank is the binary code BW of the column coordinate of the current event y The lowest two bits of the first target row memory bank's target memory block index are the binary code BW of the column coordinates of the current event. y The remaining BW y1 -2 bits; the index of the second target row storage bank is the binary code BW of an adjacent column coordinate of the column coordinate of the current event y1 The lowest two bits of the second target row memory bank, the index of the target memory block is the binary code BW y1 The remaining BW y1 -2 bits; the index of the third target row storage bank is the binary code BW of another adjacent column coordinate of the column coordinate of the current event y2 The lowest two bits of the target memory block of the third target row memory bank are the binary code BW y3 The remaining BW y1 -2 digits.

8. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The first storage control unit includes: a storage update unit and an N RM Each scratch pad memory wpt0 contains The zth storage unit of the i-th note memory wpt0 is used to store the index of the latest event to be replaced in the zth storage block of the i-th row storage body in a one-to-one correspondence, and the value of z ranges from 0 to The storage update unit is used to obtain the latest index of the event to be replaced from the storage unit corresponding to the target storage block of the first target row storage body, and add 1 to the obtained index of the latest event to be replaced to obtain the updated index of the latest event to be replaced, and then determine whether the updated index of the latest event to be replaced is greater than S RM ; If so, replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with the index of the updated latest event to be replaced; if not, set the index of the updated latest event to be replaced to 0, and replace the index of the latest event to be replaced in the storage unit corresponding to the target storage block of the first target row storage body with 0.

9. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 3, characterized in that: The information of an event includes: row coordinates and timestamp; A first event decision unit connected to each target row storage body is specifically used to compare the row coordinates of the current event with the S RM The row coordinates of each historical time in the historical events are subtracted, and the row coordinate difference is compared with the spatial correlation threshold D th The timestamp of the current event is compared with the S RM The timestamp of each historical time in the historical events is subtracted, and the timestamp difference is compared with the time-related threshold T th The first comparison result and the second comparison result are compared to obtain a second comparison result with a value of 0 or 1, and the first comparison result and the second comparison result are ANDed to obtain a result value of 0 or 1, and the result is ANDed to obtain a result value of 0 or 1. RM S RM After the result values ​​are added together, a sub-number of historical events related to the current event is obtained.

10. The spatiotemporal correlation filter for dynamic visual sensor noise reduction according to claim 1, characterized in that: The column noise reduction module further includes: a second control unit, a second write address register, a second adder, a second storage control unit, CM second event decision units, wherein the j-th second event decision unit is correspondingly connected to the j-th column storage body; The second control unit is used to determine, according to the row coordinates of the current event output by the DVS, an index of a column storage body corresponding to the current event as the index of the first target column storage body, and determine the index of the storage block to be read from the first target column storage body, and determine the indexes of two corresponding column storage bodies according to two adjacent row coordinates of the row coordinates of the current event as the index of the second target column storage body and the index of the third target column storage body respectively, and determine the indexes of the storage block to be read from the second target column storage body and the third target column storage body respectively, so as to obtain the indexes of the three target column storage bodies and the index of the target storage block in each target column storage body; and store the index of the target storage block of the first target column storage body into the second write address register; Each of the three target column storage bodies is used to store the S stored in the target storage block of the target column storage body. CM The information of each historical event is sent to the second event decision unit connected to itself; A second event decision unit connected to each target column storage body is used to determine the event based on the column coordinates and timestamp of the current event and the spatial correlation threshold D th , the time-related threshold T th and the S CM information of historical events, determining the number of historical events related to the current event, and obtaining a sub-number; The second storage control unit is used to generate an index of an event to be replaced by the current event in the target storage block of the first target column storage body; The second write address register is used to store the index of the target storage block of the first target column storage body, and send the index of the target storage block to the first target column storage body in a time period next to the time period in which the first target column storage body reads data; The first target column storage body is further used to compress the bit width of the timestamp of the current event, and replace the column coordinates, polarity and timestamp of the event to be replaced with the column coordinates, polarity and timestamp of the event to be replaced, so as to store the current event in the target storage block; The second adder is used to sum the three sub-quantities obtained by the three second event decision units correspondingly connected to the three target column storage bodies to obtain a historical event quantity cr_event_col related to the current event.

Images