A configurable dynamic vision camera event real-time preprocessing chip

Abstract

The application relates to the technical field of dynamic vision camera event processing, and particularly relates to a configurable dynamic vision camera event real-time preprocessing chip, which comprises a scaling module for performing horizontal and vertical coordinate scaling of AER data; a denoising module for removing output noise of a dynamic vision camera; a frame dividing module for realizing event statistics within one timestamp; a pulse sending module for judging whether the event number of each pixel point after statistics frame dividing is greater than a set threshold value; a ping-pong control module for controlling data flow, solving the difference of instantaneous data rate and realizing real-time performance; and a configuration module for realizing the configuration of denoising precision in the scaling module and the denoising module, the configuration of time step in the frame dividing module and the configuration of the pulse sending threshold value of the pulse sending module. The processing chip can realize real-time scaling, denoising and pulse sending of the output events of the dynamic vision camera under the same chip architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dynamic vision camera event processing, and particularly relates to a configurable dynamic vision camera event real-time preprocessing chip. BACKGROUND

[0002] Dynamic vision camera: Dynamic Vision Sensor camera, abbreviated as DVS, compared with traditional digital cameras, the biggest working feature is asynchronous perception. The work of each pixel in VS is independent and does not interfere with each other. That is to say, each photosensitive pixel only performs independent photosensing on the part of the area it is responsible for. At the same time, each photosensitive pixel no longer directly samples the real value, but compares the relative change value of the photocurrent at all times to see if it exceeds a threshold. Correspondingly, the change of light intensity actually corresponds to the change of photocurrent. In fact, each pixel position on the DVS is sensing the relative light intensity change. When enough light intensity change is sensed on the corresponding pixel, the sensor will send out an "event signal", which is usually a data packet (AER data) containing the following information

[0003] The coordinate address (x, y) of the pixel;

[0004] The polarity of the change of light intensity;

[0005] The timestamp of the event occurrence;

[0006] The dynamic vision camera preprocessing chip is a chip specially for preprocessing AER data. The preprocessing mentioned here includes scaling, denoising, pulse emission and other operations that need to be performed before inputting the AI chip. Compared with general-purpose processors, it has the advantages of higher efficiency and lower power consumption.

[0007] For the implementation of dynamic vision camera preprocessing, there is very little related research in the academic field. Most of the preprocessing implementations are done on the PC software side, and the few hardware implementations also have the problems of excessive hardware resource occupation, high power consumption and not considering noise removal. The current research on denoising algorithms mainly searches for related events in the surrounding square field by means of age mechanism, which actually also utilizes the temporal and spatial limitations of events. The research on scaling and pulse emission is based on shifting and threshold activation.

[0008] However, there is no more efficient and real-time algorithm and corresponding hardware implementation solution for the real-time preprocessing of dynamic vision camera event streams, which also leads to the inability to complete real-time end-to-end system implementation.

[0009] Therefore, there is an urgent need for a new technical solution to solve the above technical problems. SUMMARY

[0010] The present application aims to overcome the problems of the prior art, and provides a configurable dynamic visual camera event real-time preprocessing chip to solve the technical problem that the traditional technical solution does not realize efficient and low-power event denoising and complete preprocessing process, resulting in the inability to complete the real-time end-to-end inference system implementation.

[0011] The above object is achieved by the following technical solution:

[0012] A configurable dynamic visual camera event real-time preprocessing chip comprises:

[0013] A scaling module for performing horizontal and vertical coordinate scaling of AER data;

[0014] A denoising module for removing output noise of a dynamic visual camera;

[0015] A frame dividing module for implementing event statistics within one timestamp;

[0016] A pulse emitting module for judging whether the number of events of each pixel point after statistical frame dividing is greater than a set threshold, and if so, emitting a pulse;

[0017] A ping-pong control module for controlling data flow, solving the difference in instantaneous data rate, and realizing real-time performance;

[0018] A configuration module for configuring the denoising precision in the scaling module and the denoising module, configuring the time step in the frame dividing module, and configuring the pulse emitting threshold of the pulse emitting module.

[0019] Further, the pulse emitting module comprises a first pulse emitting module and a second pulse emitting module, and the frame dividing module comprises a first frame dividing module and a second frame dividing module; the first frame dividing module and the first pulse emitting module constitute an upper half area, the second frame dividing module and the second pulse emitting module constitute a lower half area, and the operation of the upper half area and the lower half area is switched through the ping-pong control module.

[0020] Further, the configuration module realizes parameter configuration of the scaling module, the denoising module, the frame dividing module, and the pulse emitting module through a configuration module command code, the configuration module command code realizes transmission of a control word through a UART serial port, and the 32nd to 35th bits are used as configuration selection, including decision time step, pulse statistical time step, event number effective threshold, and event number cumulative threshold.

[0021] Further, the denoising module pairs events two by two according to spatial correlation and temporal correlation of the events, and events that have been paired are regarded as valid events, while events that have not been paired are regarded as noise and removed; the denoising process of the denoising module includes:

[0022] The configuration stage arranges adjacent m events in the order of event occurrence, and then divides the entire region into n domains; if two or more events in the m events hit the same domain, the two or more events are paired;

[0023] The out-of-order output stage realizes out-of-order output of events that have been paired in the buffer.

[0024] Further, the configuration stage specifically includes:

[0025] An independent channel is opened for each domain, denoted as pipe, and a flag bit is set for the pipe;

[0026] The first event in the m events that enters the pipe is cached;

[0027] When events in a pipe are paired, the flag bit of the pipe is set as valid;

[0028] Thereafter, when events in the m events enter the pipe with the valid flag bit, the events are valid events.

[0029] Further, the out-of-order output stage specifically includes:

[0030] When the first A event enters a pipe, it waits for the next event that enters the pipe to pair with it;

[0031] During the waiting process of the A event, if an event C enters a pipe with a valid flag bit, the event C sends a highest priority request to the output channel, and the C is directly outputted;

[0032] During the waiting process of the A event, if an event B enters the pipe, the event that stays in the pipe for the longest time among the two events A and B that have just been paired has a second highest priority when outputted, sends a request to the output channel, and the A is directly outputted, while the B enters the output cache of the pipe;

[0033] When there is no request for the output channel, the output cache of each pipe is polled in a cycle, and the events in the output cache are outputted;

[0034] The order of outputted events is different from that of inputted events.

[0035] Further, the scaling module utilizes pipelined shift operations to scale the dynamic vision camera image to a predefined size.

[0036] Further, the frame dividing module is used to divide the event statistics in one time stamp into frames, specifically, by using a pixel event counter to count the positive / negative events occurring on each pixel point in the same time step.

[0037] The configurable dynamic vision camera event real-time preprocessing chip provided by the application can realize real-time scaling, denoising and pulse emission of the output events of the dynamic vision camera under the same chip architecture, has the characteristics of low power consumption, low area and real-time, and paves the way for the subsequent real-time end-to-end inference system implementation. The denoising algorithm using out-of-order output can meet the real-time requirement in event denoising processing, and can reduce the use of hardware resources and power consumption in this stage; the pulse emission module based on threshold can filter the less important information in the events twice, and reduce the complexity of the subsequent AI processing chip; the user can select the configuration of the architecture according to different requirements; in addition, the input interface uses general AER data and data valid interface, and has high universality. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A block diagram of the configurable dynamic vision camera event real-time preprocessing chip provided by the application;

[0039] Figure 2 A configuration module command code structure and part semantic comparison table of the configurable dynamic vision camera event real-time preprocessing chip provided by the application;

[0040] Figure 3 A denoising algorithm conceptual diagram of the configurable dynamic vision camera event real-time preprocessing chip provided by the application;

[0041] Figure 4 A frame dividing and pulse emission module architecture diagram of the configurable dynamic vision camera event real-time preprocessing chip provided by the application. DETAILED DESCRIPTION

[0042] The application will be described in further detail below with reference to the drawings and embodiments. The described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0043] As Figure 1 shown, the application provides a configurable dynamic vision camera event real-time preprocessing chip, which comprises:

[0044] a scaling module for performing scaling of horizontal and vertical coordinates of AER data, which utilizes pipelined shift operation to scale dynamic vision camera image to predefined size while detecting whether a new time step is reached by time step detection module in parallel;

[0045] a denoising module for removing output noise of dynamic vision camera;

[0046] a frame dividing module for dividing events statistics in a time stamp into frames;

[0047] a pulse emitting module for judging whether the number of events of each pixel point after frame division is greater than a set threshold value, and emitting pulse if yes;

[0048] a ping-pong control module for controlling data flow, solving difference of instantaneous data rate and realizing real-time performance;

[0049] a configuration module for configuring denoising precision in the scaling module and the denoising module, time step length in the frame dividing module and pulse emitting threshold value in the pulse emitting module.

[0050] Before the chip performs preprocessing work, the key parameters required in the algorithm need to be configured by the configuration module first. As shown in Figure 2 The configuration module realizes parameter configuration of the scaling module, the denoising module, the frame dividing module and the pulse emitting module through configuration module command code. The configuration module command code realizes transmission of control word by UART serial port, and takes the 32nd to 35th bits as configuration selection, including decision time step length (time step length of events required for each decision or inference), pulse statistics time step length (time step length after which a plurality of frame pulses are counted before inference is performed), event number effective threshold value (set event number size specific value to determine that there is no effective event in this time step) and event number accumulation threshold value (event number accumulation threshold value, greater than the threshold value to emit input pulse).

[0051] In the present chip, one preprocessing performs the steps of scaling, denoising, frame dividing and pulse emitting in sequence. Among them, the frame dividing module and the pulse emitting module are logically copied twice and divided into upper and lower half areas, which are switched by the ping-pong control module. That is, whenever the time item in AER time changes to a new time step length, the ping-pong module switches another half area for processing, and the half area before switching starts to emit the collected pulses before.

[0052] Specifically, in this embodiment, the pulse issuing module includes a first pulse issuing module and a second pulse issuing module, and the framing module includes a first framing module and a second framing module;

[0053] The first framing module and the first pulse issuing module constitute an upper half zone, the second framing module and the second pulse issuing module constitute a lower half zone, and operations of the upper half zone and the lower half zone are switched by the ping-pong control module.

[0054] The framing module in this embodiment is used to implement event statistics framing within a timestamp, specifically by separately counting the positive / negative events occurring at each pixel point within the same time step through a pixel event counter.

[0055] like Figure 4 As shown, in the framing module and pulse emission module, a regional comparator first classifies and diverts events in different regions. Next, the pixel event counter in the framing module counts the positive and negative events occurring at each pixel within the same time step. The upper-half-region pulse emission compares the framing module's count value with a set threshold. If the value exceeds the threshold, a pulse is emitted. If the ping-pong control module does not allow output for that half-region, the pulse is first stored in the upper-half-region FIFO. When a new time step is detected, the ping-pong control module switches state, reads the pulse from the upper-half-region FIFO, and repeats the above process for the lower-half-region.

[0056] The denoising module in this embodiment uses the spatial correlation and temporal correlation of events to pair events. Paired events are considered valid events, while unpaired events are considered noise and removed.

[0057] The denoising process of the denoising module includes:

[0058] In the configuration phase, m adjacent events are arranged in order of occurrence, and the entire area is divided into n domains. If two or more of the m events hit the same domain, the two or more events are paired;

[0059] The out-of-order output stage is used to output the events that have been paired in the buffer in an out-of-order manner.

[0060] It should be noted that the configuration phase is used to perform parallel pairing of the n regions, specifically:

[0061] Open an independent channel for each area, record it as pipe, and set a flag for the pipe;

[0062] Cache the first event that enters the pipe among m events;

[0063] When the event in a pipe is completed pairing, the flag of the pipe is set to valid;

[0064] After this, when the m events enter the pipe with valid flag, it must be a valid event.

[0065] The out-of-order output stage refers to the order of the valid events entering the denoising module and the order of the output may be different, specifically:

[0066] When the first A event enters a pipe, it will wait for the next event entering the pipe to complete pairing with it;

[0067] During the waiting process of the A event, if an event C enters a pipe with a valid flag, the event C will issue a highest priority request to the output channel and directly output C;

[0068] During the waiting process of the A event, if an event B enters the pipe, the two events A and B that have just completed pairing will stay in the pipe for the longest time, i.e., event A, which has the second highest priority, will issue a request to the output channel and directly output A, while B will enter the output buffer of the pipe;

[0069] When there is no request for the output channel, the output buffer of each pipe is polled in a cycle, and the events in the output buffer are output;

[0070] The order of output events is different from that of input.

[0071] As shown in Figure 3 , as a specific embodiment of the denoising process, the figure shows four regions (A, B, C, D), and the corresponding events in each region are A x , B x , C x , and D x .

[0072] First, an independent channel (pipe) is opened for each region, and each channel is responsible for capturing and identifying events in the respective region. A flag is set for the pipe, and the initial state of each pipe is idle;

[0073] Then, the first event of the m events entering the pipe is cached, and the state of the pipe changes from idle to waiting;

[0074] When a certain pipe has a pair of events completed (for example, when there is a buffered event A0 in channel A and another event A1 of the channel is captured), the state of the pipe changes from waiting to passing; thereafter, when a pipe with the entered flag valid among the m events enters the denoising module, it is always a valid event.

[0075] The out-of-order output stage refers to the order of the valid events entering the denoising module and the order of the output, which can be different, and specifically:

[0076] As shown in event group 1, in period 1, the first A0 event of the A region entering the pipe of the A region will wait for the next event entering the pipe to complete pairing; in period 2 and period 4, two events of the B region have completed pairing, and the two events are regarded as valid events, an output request is sent, and the state of channel B is switched to passing; in period 5, during the waiting process of the A event, an event B2 enters channel B which is in the passing state, the event B2 will send a highest priority request to the output channel, and B2 is directly output; in period 6, an event A1 arrives, the state of channel A changes from waiting to passing, at this time, two events in channel A need to be output, but since B2 occupies the output channel, the event A1 in channel A needs to be buffered, and the position of the buffering is called a reservation station; when the output channel has no request, the reservation station is queried in a period, and the event in the reservation station is output, which leads to the difference between the order of the output event and the input event. After the statistics of an event group is completed, the states of all channels return to the initial state.

[0077] As shown in event group 2, after the last event is input in period 9, B3 and D2 are not paired, and at this time, the two events are regarded as noise and removed.

[0078] The above only describes the embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A configurable real-time pre-processing chip for dynamic visual camera events, characterized in that: include: A scaling module, the scaling module is used to perform horizontal and vertical coordinate scaling of the AER data; A denoising module, wherein the denoising module is used to remove output noise of a dynamic vision camera; A framing module, which is used to realize framing of event statistics within a timestamp; A pulse issuing module is used to determine whether the number of events at each pixel point after statistical framing is greater than a set threshold, and if so, to issue a pulse; A ping-pong control module, which is used to control data flow, resolve instantaneous data rate differences, and achieve real-time performance; a configuration module, the configuration module being used to configure the denoising accuracy in the scaling module and the denoising module, the time step in the framing module, and the pulse emission threshold in the pulse emission module; The denoising module pairs events using their spatial and temporal correlations. Paired events are considered valid events, while unpaired events are considered noise and removed. The denoising process of the denoising module includes: In the configuration phase, m adjacent events are arranged in order of occurrence, and the entire area is divided into n domains. If two or more of the m events hit the same domain, the two or more events are paired; The out-of-order output stage is used to output the paired events in the buffer in an out-of-order manner. The out-of-order output stage means that the order in which valid events enter the denoising module may be different from the order in which they are output. Specifically: When the first A event enters a pipe, it will wait for the next event to enter the pipe to complete the pairing. During the waiting process of event A, if an event C enters a pipe with a valid flag, this event C will issue a highest priority request to the output channel and output C directly; At this time, if event B enters the pipe, the event that has stayed in the pipe the longest (i.e., event A) will have the second highest priority when outputting. A request is made to the output channel, and event A is directly output, while event B will enter the output buffer of the pipe. When there is no request on the output channel, the output buffer of each pipe is polled in one cycle and the events in the output buffer are output; The output events are in a different order than they were in the input.

2. A configurable dynamic visual camera event real-time preprocessing chip according to claim 1, characterized in that: The pulse issuing module includes a first pulse issuing module and a second pulse issuing module, and the framing module includes a first framing module and a second framing module; The first framing module and the first pulse issuing module constitute an upper half zone, the second framing module and the second pulse issuing module constitute a lower half zone, and operations of the upper half zone and the lower half zone are switched by the ping-pong control module.

3. The configurable dynamic vision camera event real-time pre-processing chip according to claim 1, characterized in that: The configuration module implements parameter configuration of the scaling module, the denoising module, the framing module and the pulse emission module through the configuration module command code. The configuration module command code uses the UART serial port to realize the transmission of the control word, and uses the 32nd to 35th bits as configuration options, including the decision time step, the pulse statistics time step, the event number valid threshold and the event number cumulative threshold.

4. The configurable real-time pre-processing chip for dynamic visual camera events according to claim 1, characterized in that: The configuration phase is used to pair the n regions in parallel, specifically: Open an independent channel for each area, record it as pipe, and set a flag for the pipe; Cache the first event that enters the pipe among m events; When an event in a pipe is paired, the flag position of the pipe is set to valid; After this, when any of these m events enters the pipe with a valid flag, it will be a valid event.

5. The configurable dynamic vision camera event real-time pre-processing chip according to claim 1, characterized in that: The scaling module scales the dynamic vision camera image to a predefined size using pipelined shift operations.

6. The configurable dynamic vision camera event real-time pre-processing chip according to claim 1, characterized in that: The framing module is used to implement event statistics framing within a timestamp, specifically by separately counting the positive / negative events occurring at each pixel point within the same time step through a pixel event counter.

Images