Monitoring event queue in event-based vision sensor to take event bursts into account

The event burstiness indicator and queue management system in event-based visual sensors mitigate event bursts, improving timestamp accuracy and reducing processing loads by selectively handling event bursts.

JP2024163052APending Publication Date: 2024-11-21PROPHESEE SOLUTIONS PVT LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024074583
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2024-05-02
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Event-based visual sensors experience discrepancies between assigned timestamps and actual event occurrences due to event bursts, leading to latency and high processing loads, particularly during sudden scene brightness changes.

Method used

Implementing an event burstiness indicator based on queue length to manage event bursts, using a queue monitor to switch between row and column arbiters for efficient processing and applying burst mitigation measures such as subsampling to reduce event load and timestamp errors.

Benefits of technology

Reduces latency and processing load by accurately tracking event bursts, ensuring reliable timestamping and optimizing resource consumption in downstream processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163052000001_ABST
    Figure 2024163052000001_ABST
Patent Text Reader

Abstract

To properly perform processing of an event queue in situations where events occur in bursts in an event-based vision sensor.SOLUTION: An event-based vision sensor comprises: an array (10) of event-based pixels; a request arbiter (12, 20) for enqueuing corresponding line readout requests in a request queue (Y-queue) so as to serve the request queue sequentially; an event readout circuit (16, 24) to read out, in parallel, events of a line of pixels corresponding to a current line readout request, and to transmit readout events with a timestamp indicative of the time of the readout in an event stream (ESTRM); a queue length counter (YQLEN) to produce a request queue length as a number of enabled request lines; a queue monitor (26) to generate an event burstiness indicator based on a current request queue length; and means to carry out a burstiness compensation action based on the event burstiness indicator.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to event-based vision sensors, and more particularly to processing an event queue in situations where events occur in bursts. [Background technology]

[0002] FIG. 1 shows a block diagram of a typical event-based vision sensor. The event-based vision sensor includes an array of pixels 10, each configured to generate an event asynchronously when a contrast change is detected at the pixel. Each event represents a relative increase or decrease in light intensity above a threshold. An arbiter 12 assigned to the array detects the events and, with the help of a control circuit 14, synchronously organizes their readout via a readout circuit 16, paced by a system clock CK. Upon readout, each event is assigned a timestamp TS, its coordinates (X,Y) in the array, its signed threshold level Δ, and is placed and transmitted in an event stream ESTRM for further processing in the application program.

[0003] In such an arrangement, situations arise that can cause discrepancies between the assigned timestamp and the actual occurrence of an event. Summary of the Invention

[0004] An event-based visual sensor generally comprises an array of event-based pixels, each pixel configured to record an event based on variations in sensed photocurrent and to enable a request line common to pixels aligned along an axis of the array upon occurrence of an event; a request arbiter configured to asynchronously respond to the enabled request lines by enqueuing corresponding line read requests in a request queue and to process the request queue sequentially; an event read circuit connected to the request arbiter, the event read circuit configured to read in parallel events for a line of pixels corresponding to a current line read request serviced by the arbiter and to transmit the read event together with a timestamp indicating the time of the read in the event stream; a queue length counter connected to the request lines and configured to generate a request queue length as a number of enabled request lines; a queue monitor connected to the queue length counter and configured to generate an event burstiness indicator based on the current request queue length; and means configured to perform a burstiness compensation operation based on the event burstiness indicator.

[0005] The readout circuit may be configured to insert successive event burstiness indicator values ​​into the event stream, and the sensor may further comprise a processor programmed to receive the event stream and to apply processing actions to the event stream in response to the event burstiness indicator values.

[0006] The burstiness indicator may be a queue length, and the arbiter may be controlled to subsample the request queue when the queue length exceeds a threshold, such that only requests corresponding to a region of interest of the pixel array are processed, and to reset the resulting unprocessed requests in the request queue.

[0007] The sensor may further comprise a row request arbiter configured to process row read requests enqueued in the row request queue, a row read circuit associated with the row request arbiter, a column request arbiter configured to process column read requests enqueued in the column request queue, a column read circuit associated with the column request arbiter, row queue length counters and column queue length counters assigned to the row and column request queues, respectively, and a request queue monitor associated with the row and column request arbiters, the request queue monitor configured to activate the row request arbiter to process the row request queue when the row request queue length exceeds the column request queue length by a threshold amount, and to activate the column request arbiter to process the column request queue when the column request queue length exceeds the row request queue length by a threshold amount.

[0008] The row and column request arbiters may be configured to operate simultaneously to enqueue respective row and column read requests, and the request queue monitor may be configured to reset the request queue of an arbiter that is switched from an active state to an inactive state. [Brief description of the drawings]

[0009] Embodiments are set forth in the following description, given by way of example only, in connection with the accompanying drawings, in which: [Figure 1] FIG. 1 is a block diagram of an event-based vision sensor organized to process events row by row in a pixel array. [Diagram 2] FIG. 1 is a block diagram of an event-based visual sensor implementing an embodiment of event burst mitigation based on a burstiness indicator. [Diagram 3] FIG. 1 is a block diagram of an event-based visual sensor illustrating another embodiment of event burst mitigation based on a burstiness indicator. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In an event-based vision sensor, bursts of events occurring, for example upon sudden scene luminance changes, can result in significant latency between the occurrence of an event and its actual readout, which can introduce errors into the timestamps typically assigned to the readouts.

[0011] Furthermore, the transmitted event stream experiences bursts of event data, which can cause large traffic loads on the chip-to-chip interface and high event processing loads on the host processor.

[0012] The present disclosure provides for monitoring event generation and sending event burst information for downstream processing to apply corrective action where possible.

[0013] 1, an exemplary situation that can cause an event burst is shown: sensor array 10 observes a sudden brightness change along a line spanning eight pixels, generating eight events EV1-EV8. In this example, the pixels happen to be in three columns and eight rows.

[0014] In a typical event-based sensor, each row of the pixel array is assigned a request line REQ that is monitored by the arbiter 12. A pixel that detects an event immediately activates the request line of its row by pulling the line to an active level. If the request line is already active, subsequent events in the same row do not affect the request line. The arbiter finally serves the request by activating an acknowledge line ACK common to the row of pixels, which transfers the event data for the entire row to the latches of the readout circuit 16 and clears the request line. Only the data of the active events of a row are analyzed by the readout circuit and concatenated into an event stream ESTRM, together with the current timestamp TS, the X,Y coordinates provided by the position of the active event in the row being read, and the rank of the current request line being processed by the arbiter.

[0015] If the arbiter does not have time to process the request of the first row before an event occurs in another row generating subsequent requests, these subsequent requests are enqueued in the request queue 18 to be processed later. The enqueuing of requests is performed asynchronously by combinatorial logic in the arbiter. This may be the situation of the illustrated events EV1-EV8, which asynchronously generate eight separate requests. The arbiter and control circuitry can synchronously process only one request at a time, at the pace of the system clock CK. If events occur faster than the system clock, the arbiter processes the first request and queues the others. The remaining requests are processed according to a specified priority rule, preferably a first-in-first-out rule. In particular, in cases where it is difficult to keep track of the asynchronous order of occurrence of requests, other priority rules may be implemented, such as a fair arbitration rule based on the position of the first row to be processed, such that a particular row is processed twice only if the other rows have been processed at least once.

[0016] It is clear that when the arbiter finally processes a request in the queue, the timestamp assigned to the corresponding event, which is the time of its read, will lag behind the time of the event. The lag, which translates to a timestamp error in the event stream, is variable and depends on the position of the request in the queue, which can vary between 1 and a variable length reached by the queue. The length of the queue depends on the number of rows affected by a burst of events occurring within one period of the system clock.

[0017] The position of a request in the queue is generally not a direct indication of delay. Indeed, when events occur simultaneously, such as when a flash is observed in a scene, requests are queued at different positions that all represent the same time of occurrence. Thus, a position N in the queue can represent any time lag between one system clock period and N clock periods in processing by the arbiter.

[0018] Additionally, as a pending request waits in the queue, a newer event may occur on the corresponding line. Because the pending request line is still active, the new event does not cause any change. Thus, the same pending request may represent different event occurrence times within the time it takes to process the request.

[0019] It is provided herein to use a queue length based event burstiness indicator to apply burst mitigation measures. The burstiness indicator may simply be the current queue length or it may be a statistical indicator representing the evolution of the queue length.

[0020] In a simple embodiment, the burstiness indicator may be inserted with the event data in the event stream, and the indicator sent with the current event represents the burstiness considered for the group of previously sent events. The indicator may also be communicated over an available hardware interface. A downstream processor of the sensor or application program may use the burstiness indicator to adapt the event data processing to take into account possible timestamp errors, or apply processing that does not require accurate timestamps. In either case, the burstiness indicator may be used by downstream processing to detect a return-to-normal condition after an event burst condition, in which case the timestamps may be trusted again. Indeed, in some situations, event bursts may generally be ignored, whereby the timestamp information is irrelevant, but it is useful to know when the burst condition begins and ends.

[0021] FIG. 2 is a block diagram of an event-based vision sensor implementing another embodiment of event burst mitigation based on request queue length.

[0022] The sensor is a supplemented version of the sensor of FIG. 1 and comprises the same elements such as pixel array 10, row request arbiter 12, now labeled Y-ARBIT, row readout circuit 16, now labeled X-READ, and row request queue 18, now labeled Y-QUEUE. These elements retain their previous function, i.e., organizing the readout of rows of pixels of the array based on pixels that activate a row request Y-REQ. In other words, these elements are configured to process horizontal lines (rows) of pixels aligned along the Y axis.

[0023] In addition, the sensor comprises elements for processing the array vertically along the X-axis, i.e. for organizing the readout of vertical lines (columns) of pixels based on the pixels activating a column request X-REQ. These elements include a column request queue (X-queue) and a column request arbiter (X-arbiter 20) associated with a column readout circuit (Y-readout 24). The array 10 is thus completed by a vertical column request line X-REQ connected to the x-arbiter 20 for each column of pixels, and a number of horizontal lines (not shown) connected to the y-readout circuit 24 for reading out in parallel the events of the columns processed by the y-arbiter 20. The event stream output by the sensor is generated by merging the outputs of the x and y readout circuits, which may operate one at a time, as will be described below.

[0024] Essentially, these additional elements operate on columns of pixels in the same way that conventional elements operate on rows of pixels.

[0025] In addition, the sensor's control circuitry includes a queue monitor QMON 26 that monitors both the row request queue Y-queue and the column request queue X-queue, and each arbiter is assigned a queue length counter YQLEN, XQLEN. Each queue length counter may have a parallel counter structure with one input for each request line and an output that encodes the number of inputs that are active. For example, for a sensor with a 1280x728 pixel array, counter YQLEN has 728 inputs and a 10-bit output to encode any number between 0 and 727, and counter XQLEN has 1280 inputs and an 11-bit output.

[0026] Exemplary occurrences of events EV1-EV8 are again shown in array 10. These events are assumed to occur simultaneously or within at least one read cycle (or system clock period CK). According to a conventional row approach, the events occur in eight separate rows, thus activating eight separate row request lines Y-REQ that cannot be immediately serviced. The y-arbiter places the requests in the y-queue, which achieves a size of eight (assuming the first of the requests is not immediately serviced).

[0027] In parallel, events EV1-EV8 occurring in three separate columns activate three separate column request lines X-REQ. The x-arbiters connected to these request lines queue these requests in the x-queues.

[0028] Enqueuing the requests into the x and y queues is done asynchronously and quasi-synchronously.

[0029] When it is time to service the queues, the queue monitor 26 compares the queue lengths of the x and y queues and enables the arbiter that enables the shortest queue, say the x arbiter, to service that queue, and the other arbiter, say the y arbiter, is disabled and its request queue is reset. The x arbiter then services its three requests, representing the eight pending events in the array, saving five cycles compared to if the y arbiter had been alone to process the events.

[0030] The following table further illustrates the above operations on successive cycles of the system clock CK.

[0031] [Table 1]

[0032] Just before cycle time t0, events EV1-EV8 have triggered corresponding x and y requests, the requests have been queued in the x and y queues, and the queue lengths have been compared by the queue monitor QMON, which now enables the x arbiter. These operations are performed asynchronously via combinational logic.

[0033] At time t0, the first row of the x queue, with events EV1 and EV2, is processed, the x queue length is decremented to 2, and the y queue is reset.

[0034] At time t1, the next row in the x queue, having events EV3-EV5, is processed and the x queue length is decremented to one.

[0035] At time t2, the last row of the x queue, with events EV6-EV8, is processed and the x queue length is decremented to 0. At this point, all requests included in the previous comparison have been processed, thereby initiating a new comparison. In this example, since no further requests are enqueued, the system is idle and the queue monitor performs a comparison every new cycle to detect any new queue length imbalances.

[0036] While the x arbiter is processing its request, a new event may occur, for example, events EV9-EV11 occur in row 2, column 3, as shown in Figure 2. For clarity, assume that these rows and columns are different from the rows and columns of events EV1-EV8. As mentioned above, both the x arbiter and the y arbiter are configured to enqueue new requests, but only one of them is active at a time to process the request. In other words, the disabled arbiter continues to detect and queue requests in preparation for the next comparison.

[0037] The table below shows the additional effects of events EV9 through EV11, which are all assumed to occur slightly before time t1.

[0038] [Table 2]

[0039] Thus, shortly before time t1, when events EV9-EV11 occur, the arbiter increments the y queue by 2 and the x queue by 3, bringing the lengths of the y and x queues to 2 and 5, respectively.

[0040] At time t1, the column with events EV3 through EV5 is processed and the x queue is decremented to 4.

[0041] At time t2, the row with events EV6 through EV8 is processed and the x-queue is decremented to 3. All x-requests in the previous comparison are processed, thereby making the new comparison valid.

[0042] At time t3, queue lengths 2 and 3 are compared asynchronously, which leads to the activation of the y arbiter to service its y queue and the resetting of the x queue. In the same cycle, synchronously, the row with event EV9 is processed and the y queue is decremented to 1.

[0043] At time t4, the last row with events EV10 and EV11 is processed and the y queue is decremented to 0. A new comparison becomes valid.

[0044] In the preceding discussion, switching between row and column modes of operation was organized by directly comparing the x and y queue lengths. In some cases, switching between modes may incur some overhead that adds delay in processing the queues. In such situations, it may be preferable to switch modes when one queue length exceeds the other by a threshold that is greater than the net overhead in number of cycles.

[0045] 3 is a block diagram of an event-based vision sensor illustrating another embodiment of event burst mitigation based on queue length. The sensor may be based on a conventional structure organized to process events row by row, and includes the queue monitor 26 and row queue length counter YQLEN of FIG. 2. The arbiter 12 and pixel array 10 are further configured to selectively implement a row subsampling mode.

[0046] If the current queue length exceeds a threshold, the queue monitor 26 switches the arbiter from normal mode to sub-sampling mode, where the arbiter processes only a limited number of requests corresponding to a region of interest ROI in the pixel array. The region of interest may be, for example, one row out of every N rows, such as one row out of every three rows as shown. Any unprocessed requests in the queue are reset, i.e. cleared without being read out, and removed from the queue, thereby reducing the queue length. As soon as the queue length falls below the threshold, the normal mode may be resumed. The resetting of requests may be performed gradually, i.e. by resetting unprocessed requests preceding the currently processed request in the queue, or all at once, i.e. by resetting all requests in the queue that are not processed immediately upon entering the sub-sampling mode.

[0047] Such an arrangement responds to bursts of events by ignoring some events within the burst, thus reducing the sensor resolution in favor of finer accuracy of the timestamp.

[0048] A region of interest may be a contiguous group of rows interspersed with a group of rows to be ignored, or a sub-matrix separated by a group of rows and a group of columns to be ignored, in which case the columns to be ignored are masked out as each row is read out, potentially reducing the number of active events to send for each row and reducing downstream processing resource consumption when bursts of events occur.

[0049] Such subsampling techniques may be used along both axes of the sensor of FIG. 2, ie, for both rows and columns.

Claims

1. An array (10) of event-based pixels, each pixel comprising: Recording events based on fluctuations in the sensed photocurrent; an array (10) configured to activate a request line common to pixels aligned along an axis of said array upon the occurrence of an event; A request arbiter (12, 20), asynchronously responding to the enabled request line by enqueuing a corresponding line read request in a request queue (Y-queue); a request arbiter (12, 20) configured to process the request queue sequentially; an event read circuit (16, 24) connected to the request arbiter, reading in parallel the events for a line of pixels corresponding to a current line read request being processed by the arbiter; an event read circuit (16, 24) configured to transmit said read event together with a timestamp indicating the time of the read in an event stream (ESTRM); a queue length counter (YQLEN) connected to said request line and generating said request queue length as a number of enabled request lines; a queue monitor (26) coupled to the queue length counter and configured to generate an event burstiness indicator based on the current request queue length; means configured to perform burstiness compensation operations based on the event burstiness indicator.

2. 2. The sensor of claim 1, wherein the readout circuitry is configured to insert successive event burstiness indicator values ​​into the event stream, and the sensor further comprises a processor programmed to receive the event stream and apply processing actions to the event stream in response to the event burstiness indicator values.

3. 2. The sensor of claim 1, wherein the burstiness indicator is a queue length, and wherein the arbiter is controlled to subsample the request queue when the queue length exceeds a threshold such that only requests corresponding to a region of interest of the pixel array are processed and to reset any resulting unprocessed requests in the request queue.

4. a row request arbiter (12) configured to process row read requests enqueued in a row request queue (Y-queue); a row read circuit (16) associated with said row request arbiter; a column request arbiter (20) configured to process column read requests enqueued in a column request queue (X-queue); a column read circuit (24) associated with said column request arbiter; a row queue length counter (YQLEN) and a column queue length counter (XQLEN) assigned to the row and column request queues, respectively; the request queue monitor (26) associated with the row and column request arbiter, activating the row request arbiter to service the row request queue when the row request queue length exceeds the column request queue length by a threshold; a request queue monitor (26) configured to activate the column request arbiter to service the column request queue when the column request queue length exceeds the row request queue length by a threshold amount.

5. 5. The sensor of claim 4, wherein the row and column request arbiters are configured to operate simultaneously to enqueue the respective row and column read requests, and the request queue monitor is configured to reset the request queue of an arbiter that is switched from an active state to an inactive state.