Configurable dynamic vision camera event real-time preprocessing chip

Abstract

The invention relates to the technical field of dynamic vision camera event processing, in particular to a configurable dynamic vision camera event real-time preprocessing chip which comprises a scaling module used for executing horizontal and vertical coordinate scaling of AER data. The denoising module is used for removing output noise of the dynamic vision camera; the framing module is used for realizing statistical framing of events in a timestamp; the pulse issuing module is used for judging whether the number of events of each pixel point after the statistical framing is greater than a set threshold value or not; the ping-pong control module is used for controlling data streams, solving the difference of instantaneous data rates and realizing real-time performance; and the configuration module is used for realizing the configuration of the denoising precision in the scaling module and the denoising module, the configuration of the time step length in the framing module and the configuration of the pulse sending threshold value of the pulse sending module. The processing chip can realize real-time scaling, denoising and pulse distribution of the output event of the dynamic vision camera under the same chip architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic vision camera event processing, and in particular to a configurable dynamic vision camera event real-time preprocessing chip. Background Art

[0002] Dynamic Vision Camera: Dynamic Vision Sensor camera, referred to as DVS, has the biggest working feature compared to traditional digital cameras. The asynchronous sensing of each pixel in VS is independent and does not interfere with each other. That is to say, each photosensitive pixel only senses the part of the area it is responsible for. At the same time, each photosensitive pixel no longer directly samples the real value, but compares the relative change value of the photocurrent at all times to see if it exceeds a threshold. Correspondingly, the change in light intensity actually corresponds to the change in the size of the photocurrent. In fact, each pixel position on the DVS is sensing whether the change in relative light intensity is obvious. When enough light intensity changes are sensed at the corresponding pixel, the sensor will send out an "event signal". Generally speaking, this signal is a data packet (AER data) containing the following information

[0003] The pixel coordinate address (x, y);

[0004] The polarity of changes in light intensity;

[0005] The timestamp of the event;

[0006] The dynamic vision camera preprocessing chip is a chip specifically used to preprocess AER data. The preprocessing here includes scaling, denoising, pulse emission and other operations that need to be performed before inputting into the AI ​​chip. Compared with general-purpose processors, it has the advantages of higher efficiency and lower power consumption.

[0007] There is very little research in the academic community on the implementation of dynamic vision camera preprocessing. Most preprocessing implementations are performed on the PC software side, and the few hardware implementations also have the problem of excessive hardware resource usage, high power consumption, and no consideration of noise removal. The current research on denoising algorithms mainly uses the age mechanism to search for related events in the surrounding square area, which actually takes advantage of the time and space limitations of events. The research on scaling and pulse emission is based on shifting and threshold activation.

[0008] However, there is currently no more efficient, real-time algorithm or corresponding hardware implementation solution for real-time preprocessing of event streams for dynamic vision cameras, which also makes it impossible to complete real-time end-to-end system implementation.

[0009] Therefore, a new technical solution is urgently needed to solve the above technical problems. Summary of the invention

[0010] The purpose of the present invention is to overcome the above-mentioned problems of the prior art and provide a configurable real-time preprocessing chip for dynamic vision camera events to solve the technical problem that the traditional technical solution fails to achieve more efficient and low-power event denoising and a complete preprocessing process, resulting in the inability to complete the implementation of a real-time end-to-end reasoning system.

[0011] The above objectives are achieved through the following technical solutions:

[0012] A configurable real-time pre-processing chip for dynamic vision camera events, including:

[0013] A scaling module, the scaling module is used to perform horizontal and vertical coordinate scaling of the AER data;

[0014] A denoising module, wherein the denoising module is used to remove output noise of a dynamic vision camera;

[0015] A framing module, the framing module is used to realize framing of event statistics within a timestamp;

[0016] A pulse issuing module, which is used to determine whether the number of events of each pixel point after statistical framing is greater than a set threshold, and if so, to issue a pulse;

[0017] A ping-pong control module, which is used to control data flow, solve the difference in instantaneous data rate, and achieve real-time performance;

[0018] A configuration module, wherein the configuration module is used to configure the denoising accuracy in the scaling module and the denoising module, the time step in the framing module, and the pulse emission threshold of the pulse sending module.

[0019] Furthermore, the pulse issuing module includes a first pulse issuing module and a second pulse issuing module, and the frame module includes a first frame module and a second frame module; the first frame module and the first pulse issuing module constitute an upper half area, and the second frame module and the second pulse issuing module constitute a lower half area, and the operation of the upper half area and the lower half area is switched by the ping-pong control module.

[0020] Furthermore, the configuration module implements parameter configuration of the scaling module, the denoising module, the framing module and the pulse sending module through a configuration module command code. The configuration module command code uses a UART serial port to implement transmission of control words, and uses bits 32 to 35 as configuration options, including a decision time step, a pulse statistics time step, an event number valid threshold and an event number cumulative threshold.

[0021] Furthermore, the denoising module pairs events in pairs using spatial correlation and temporal correlation of events, and paired events are regarded as valid events, while unpaired events are regarded as noise and removed; the denoising process of the denoising module includes:

[0022] In the configuration stage, m adjacent events are arranged in the order of occurrence, and the entire area is divided into n domains. If two or more of the m events hit the same area, the two or more events are paired;

[0023] The out-of-order output stage implements out-of-order output of events that have been paired in the buffer.

[0024] Furthermore, the configuration stage is specifically as follows:

[0025] Open an independent channel for each area, record it as pipe, and set a flag for the pipe;

[0026] Cache the first event that enters the pipe among m events;

[0027] When an event in a pipe is paired, the flag of the pipe is set to valid;

[0028] After that, when one of these m events enters the pipe with a valid flag, it must be a valid event.

[0029] Furthermore, the out-of-order output stage is specifically as follows:

[0030] When the first A event enters a pipe, it will wait for the next event to enter the pipe to complete the pairing.

[0031] During the waiting process of event A, if an event C enters a pipe with a valid flag, this event C will issue a highest priority request to the output channel and output C directly;

[0032] During the waiting process of event A, if event B enters the pipe, the event A and B that has just completed pairing and has stayed in the pipe the longest will have the second highest priority when outputting. A will be directly outputted and B will enter the output cache of the pipe.

[0033] When there is no request on the output channel, the output buffer of each pipe is polled in one cycle and the events in the output buffer are output;

[0034] The output events are in a different order than they were in the input.

[0035] Furthermore, the scaling module can scale the dynamic visual camera image to a predefined size using a pipelined shift operation.

[0036] Furthermore, the framing module is used to implement framing of event statistics within a timestamp, specifically: the positive / negative events occurring at each pixel point within the same time step are counted separately by a pixel event counter.

[0037] The configurable real-time preprocessing chip for dynamic vision camera events provided by the present invention can realize real-time scaling, denoising and pulse emission of the output events of the dynamic vision camera under the same chip architecture, and has the characteristics of low power consumption, low area and real-time performance, paving the way for the subsequent real-time end-to-end reasoning system implementation. The denoising algorithm using out-of-order output can not only meet the real-time requirements in the event denoising processing, but also reduce the use and power consumption of hardware resources at this stage; the threshold-based pulse emission module can be used to filter less important information in the event for a second time, reducing the complexity of the subsequent AI processing chip; allowing users to configure the architecture according to different needs; in addition, the input interface uses universal AER data and data valid interface, which is highly universal. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A block diagram of a configurable real-time preprocessing chip for dynamic visual camera events according to the present invention;

[0039] Figure 2 The present invention provides a configuration module command code structure and a semantic comparison table of each part in a configurable dynamic vision camera event real-time preprocessing chip;

[0040] Figure 3 This is a conceptual diagram of a denoising algorithm of a configurable real-time preprocessing chip for dynamic visual camera events according to the present invention;

[0041] Figure 4 This is an architecture diagram of the frame division and pulse emission module in a configurable real-time preprocessing chip for dynamic vision camera events described in the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below based on the accompanying drawings and embodiments. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, this solution provides a configurable real-time pre-processing chip for dynamic visual camera events, including:

[0044] A scaling module, which is used to perform horizontal and vertical coordinate scaling of the AER data. The scaling module uses a pipelined shift operation to scale the dynamic visual camera image to a predefined size and simultaneously detects in parallel whether a new time step has been reached through a time step detection module;

[0045] A denoising module, wherein the denoising module is used to remove output noise of a dynamic vision camera;

[0046] A framing module, the framing module is used to realize framing of event statistics within a timestamp;

[0047] A pulse issuing module, which is used to determine whether the number of events of each pixel point after statistical framing is greater than a set threshold, and if so, to issue a pulse;

[0048] A ping-pong control module, which is used to control data flow, solve the difference in instantaneous data rate, and achieve real-time performance;

[0049] A configuration module, wherein the configuration module is used to configure the denoising accuracy in the scaling module and the denoising module, the time step in the framing module, and the pulse emission threshold of the pulse sending module.

[0050] Before the chip performs preprocessing, the key parameters required in the algorithm need to be configured through the configuration module. Figure 2 As shown, the configuration module realizes parameter configuration of the scaling module, the denoising module, the framing module and the pulse sending module through the configuration module command code. The configuration module command code uses the UART serial port to realize the transmission of the control word, and uses the 32nd to 35th bits as configuration options, including the decision time step (the time step of the event required for each decision or reasoning), the pulse statistics time step (a reasoning is performed only after a number of frame pulse statistics time steps), the event number valid threshold (setting a specific value of the event number size determines that there is no valid event in this time step) and the event number accumulation threshold (the event number accumulation threshold, the input pulse is issued when it is greater than the threshold).

[0051] In this chip, one preprocessing step is to perform scaling, denoising, framing and pulse emission in sequence. Among them, the framing module and the pulse emission module are logically replicated twice, divided into upper and lower halves, and switched by the ping-pong control module. That is, whenever the time item in the AER time changes to a new time step, the ping-pong module switches to another half-area for processing, and the half-area before the switch starts to emit the previously collected pulses.

[0052] Specifically, the pulse issuing module in this embodiment includes a first pulse issuing module and a second pulse issuing module, and the frame dividing module includes a first frame dividing module and a second frame dividing module;

[0053] The first frame division module and the first pulse issuing module constitute an upper half zone, the second frame division module and the second pulse issuing module constitute a lower half zone, and the operations of the upper half zone and the lower half zone are switched by the ping-pong control module.

[0054] The framing module in this embodiment is used to implement event statistics framing within a timestamp, specifically: the positive / negative events occurring at each pixel point within the same time step are counted separately by a pixel event counter.

[0055] like Figure 4 As shown, in the framing module and the pulse emission module, the regional comparator first classifies and diverts the events in different regions; then, the pixel event counter in the framing module counts the positive / negative events occurring at each pixel point within the same time step, and the upper half-area pulse emission compares the count value of the framing module with the set threshold. If it is greater than the threshold, a pulse is emitted. When the ping-pong control module does not allow the output of the half area, the pulse is first stored in the upper half area FIFO. When a new time step is detected, the ping-pong control module switches the state, reads the pulse from the upper half area FIFO, and repeats the above process in the lower half area.

[0056] The denoising module in this embodiment uses the spatial correlation and temporal correlation of events to pair events in pairs, and paired events are regarded as valid events, while unpaired events are regarded as noise and removed;

[0057] The denoising process of the denoising module includes:

[0058] In the configuration stage, m adjacent events are arranged in the order of occurrence, and the entire area is divided into n domains. If two or more of the m events hit the same area, the two or more events are paired;

[0059] The out-of-order output stage implements out-of-order output of events that have been paired in the buffer.

[0060] It should be noted that the configuration phase is used to perform parallel pairing of the n regions, specifically:

[0061] Open an independent channel for each area, record it as pipe, and set a flag for the pipe;

[0062] Cache the first event that enters the pipe among m events;

[0063] When an event in a pipe is paired, the flag of the pipe is set to valid;

[0064] After that, when one of these m events enters the pipe with a valid flag, it must be a valid event.

[0065] The out-of-order output stage refers to the fact that the order in which valid events enter the denoising module may be different from the order in which they are output, specifically:

[0066] When the first A event enters a pipe, it will wait for the next event to enter the pipe to complete the pairing.

[0067] During the waiting process of event A, if an event C enters a pipe with a valid flag, this event C will issue a highest priority request to the output channel and output C directly;

[0068] During the waiting process of event A, if event B enters the pipe, the event that has stayed in the pipe the longest (i.e. event A) will have the second highest priority when outputting. A request will be made to the output channel, and A will be directly output, while B will enter the output cache of the pipe.

[0069] When there is no request on the output channel, the output buffer of each pipe is polled in one cycle and the events in the output buffer are output;

[0070] The output events are in a different order than they were in the input.

[0071] like Figure 3 As shown in the figure, as a specific embodiment of the denoising process, the example in the figure shows four areas (A, B, C, D), and the corresponding events in each area are A x , B x , C x , D x .

[0072] First, an independent channel (pipe) is opened for each area. Each channel is responsible for capturing and identifying events in its own area. A flag is set for the pipe. The initial state of each pipe is idle.

[0073] After that, the first event that enters the pipe among the m events is cached, and the state of the pipe changes from idle to waiting;

[0074] When an event in a pipe is paired (for example, there is a cached event A in channel A), 0 In this case, another event A of this channel is captured 1), the state of the pipe changes from waiting to passing; after that, when the pipe with a valid flag is entered again in these m events, it must be a valid event.

[0075] The out-of-order output stage means that the order in which valid events enter the denoising module may be different from the order in which they are output. Specifically:

[0076] As shown in event group 1, in cycle 1, the first A in region A 0 When an event enters the pipe in area A, it will wait for the next event to enter the pipe to complete the pairing. In cycles 2 and 4, two events in area B have completed the pairing. At this time, these two events are considered valid events, an output request is issued, and the state of channel B is switched to pass. In cycle 5, while event A is waiting, event B 2 Entering channel B which is already in the passing state, this event B 2 The highest priority request will be issued to the output channel. 2 Direct output; in cycle 6, another event A arrives 1 , channel A changes from waiting to passing. At this time, two events in channel A need to be output, but due to B 2 The output channel is occupied, so event A in channel A needs to be 1 The cache is called the reservation station. When there is no request for the output channel, the reservation station is queried within a cycle and the events in the reservation station are output. This causes the order of output events to be different from that of input. When the statistics of an event group are completed, all channel states return to the initial state.

[0077] As shown in event group 2, after the last event of cycle 9 is input, B 3 and D 2 If they are not paired, the two events are considered as noise and removed.

[0078] The above description is only for illustrating the implementation mode of the present invention and is not intended to limit the present invention. For those skilled in the art, any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A configurable real-time preprocessing chip for dynamic visual camera events, characterized in that: include: A scaling module, the scaling module is used to perform horizontal and vertical coordinate scaling of the AER data; A denoising module, wherein the denoising module is used to remove output noise of a dynamic vision camera; A framing module, the framing module is used to realize framing of event statistics within a timestamp; A pulse issuing module, which is used to determine whether the number of events of each pixel point after statistical framing is greater than a set threshold, and if so, to issue a pulse; A ping-pong control module, which is used to control data flow, solve the difference in instantaneous data rate, and achieve real-time performance; A configuration module, wherein the configuration module is used to configure the denoising accuracy in the scaling module and the denoising module, the time step in the framing module, and the pulse emission threshold of the pulse sending module.

2. A configurable dynamic visual camera event real-time preprocessing chip according to claim 2, characterized in that: The pulse issuing module includes a first pulse issuing module and a second pulse issuing module, and the frame dividing module includes a first frame dividing module and a second frame dividing module; The first frame division module and the first pulse issuing module constitute an upper half zone, the second frame division module and the second pulse issuing module constitute a lower half zone, and the operations of the upper half zone and the lower half zone are switched by the ping-pong control module.

3. The configurable real-time preprocessing chip for dynamic visual camera events according to claim 1, characterized in that: The configuration module implements parameter configuration of the scaling module, the denoising module, the framing module and the pulse sending module through a configuration module command code. The configuration module command code uses a UART serial port to implement transmission of control words, and uses bits 32 to 35 as configuration options, including a decision time step, a pulse statistics time step, an event number valid threshold and an event number cumulative threshold.

4. The configurable dynamic vision camera event real-time preprocessing chip according to claim 1, characterized in that: The denoising module pairs events in pairs using spatial correlation and temporal correlation of events, and paired events are regarded as valid events, while unpaired events are regarded as noise and removed; The denoising process of the denoising module includes: In the configuration stage, m adjacent events are arranged in the order of occurrence, and the entire area is divided into n domains. If two or more of the m events hit the same area, the two or more events are paired; The out-of-order output stage implements out-of-order output of events that have been paired in the buffer.

5. A configurable dynamic vision camera event real-time preprocessing chip according to claim 4, characterized in that: The configuration phase is used to perform parallel pairing of the n regions, specifically: Open an independent channel for each area, record it as pipe, and set a flag for the pipe; Cache the first event that enters the pipe among m events; When an event in a pipe is paired, the flag of the pipe is set to valid; After that, when one of these m events enters the pipe with a valid flag, it must be a valid event.

6. A configurable dynamic vision camera event real-time preprocessing chip according to claim 5, characterized in that: The out-of-order output stage means that the order in which valid events enter the denoising module may be different from the order in which they are output, specifically: When the first A event enters a pipe, it will wait for the next event to enter the pipe to complete the pairing. During the waiting process of event A, if an event C enters a pipe with a valid flag, this event C will issue a highest priority request to the output channel and output C directly; During the waiting process of event A, if event B enters the pipe, the event that has stayed in the pipe the longest (i.e. event A) will have the second highest priority when outputting. A request will be made to the output channel, and A will be directly output, while B will enter the output cache of the pipe. When there is no request on the output channel, the output buffer of each pipe is polled in one cycle and the events in the output buffer are output; The output events are in a different order than they were in the input.

7. The configurable dynamic vision camera event real-time preprocessing chip according to claim 1, characterized in that: The scaling module uses pipelined shift operations to scale the dynamic visual camera image to a predefined size.

8. The configurable real-time preprocessing chip for dynamic visual camera events according to claim 1, characterized in that: The framing module is used to implement event statistics framing within a timestamp, specifically: the positive / negative events occurring at each pixel point within the same time step are counted separately by a pixel event counter.

Images