Monitoring event queues in event-based visual sensors to account for event bursts
By using event burst indicators and subsampling modes of queue length in event-based vision sensors, the timestamp inconsistency and high load problems caused by event bursts are solved, and the accuracy and efficiency of event processing are improved.
Patent Information
- Application Number
- CN202410549039.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-05-06
- Publication Date
- 2024-11-12
Smart Images

Figure CN118945496A_ABST
Abstract
Description
Technical Field
[0001] The present invention generally relates to event-based vision sensors, and more particularly to the processing of event queues in the presence of event bursts. Background Art
[0002] Figure 1 A block diagram of a typical event-based vision sensor is shown. It comprises an array of pixels 10, each of which is configured to asynchronously generate an event when a contrast change is detected in the pixel. Each event represents a relative increase or decrease in light intensity exceeding a threshold. An arbiter 12 assigned to the array detects the events and, with the aid of a control circuit 14, organizes their readout synchronously at the speed of a system clock CK by a readout circuit 16. Upon readout, each event is assigned a timestamp TS, its coordinates (X, Y) in the array, its signed threshold level Δ, and is placed in an event stream ESTRM for transmission and further processing in the application.
[0003] With this configuration, inconsistencies may occur between the assigned timestamp and the actual occurrence of the event. Summary of the invention
[0004] An event-based vision sensor generally includes: an array of event-based pixels, each pixel being configured to record an event based on a change in a sensed photocurrent, and enabling a request line common to pixels aligned along an axis of the array when an event occurs; a request arbitrator configured to asynchronously respond to enabled request lines by enqueuing corresponding line readout requests into a request queue, and to service the request queue in sequence; an event readout circuit connected to the request arbitrator and configured to read out events of pixel lines corresponding to a current line readout request serviced by the arbitrator in parallel, and to transmit readout events with a timestamp indicating the readout time in an event stream; a queue length counter connected to the request line to generate a request queue length as the number of enabled request lines; a queue monitor connected to the queue length counter and configured to generate an event burstiness indicator based on the current request queue length; and a device configured to perform a burstiness compensation action based on the event burstiness indicator.
[0005] The readout circuitry may be configured to insert consecutive event burstiness indicator values in the event stream, and the sensor may further include a processor that receives the event stream and is programmed to apply a processing action to the event stream in response to the event burstiness indicator values.
[0006] The burstiness indicator may be a queue length, and when the queue length exceeds a threshold the arbiter may be controlled to subsample the request queue so that only requests corresponding to the region of interest of the pixel array are served, and to reset the resulting unserved requests in the request queue.
[0007] The sensor may further include: a row request arbitrator configured to service row read requests queued into a row request queue; a row read circuit associated with the row request arbitrator; a column request arbitrator configured to service column read requests queued into a column request queue; a column read circuit associated with the column request arbitrator; a row queue length counter and a column queue length counter, the row queue length counter and the column queue length counter being assigned to the row request queue and the column request queue, respectively; a request queue monitor associated with the row request arbitrator and the column request arbitrator to: activate the row request arbitrator to service the row request queue when the row request queue length exceeds the column request queue length by a threshold, and activate the column request arbitrator to service the column request queue when the column request queue length exceeds the row request queue length by a threshold.
[0008] The row request arbiter and the column request arbiter may be configured to operate simultaneously to queue corresponding row read requests and column read requests, and the request queue monitor may be configured to reset the request queues of the arbiters switching from an active state to an inactive state. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Embodiments will be presented in the following description provided for exemplary purposes only, in conjunction with the accompanying drawings, in which:
[0010] Figure 1 is a block diagram of an event-based vision sensor organized to process events by rows in a pixel array;
[0011] Figure 2 is a block diagram of an event-based vision sensor for an embodiment of implementing event burst mitigation based on a burstiness indicator; and
[0012] Figure 3 is a block diagram of an event-based vision sensor illustrating another embodiment of event burst mitigation based on burstiness indicators. DETAILED DESCRIPTION
[0013] In event-based vision sensors, when events occur in bursts, such as sudden changes in scene brightness, significant delays can occur between the occurrence of an event and its actual readout, introducing errors in the timestamps typically assigned at the time of the readout.
[0014] Furthermore, the transmitted event stream experiences bursts of event data, which may result in a large traffic load on the chip-to-chip interface and a higher event processing load on the host processor.
[0015] Provided in the present disclosure is monitoring event generation and transmitting event burstiness information for use by downstream processing to apply compensatory measures where possible.
[0016] See again Figure 1 , shows an exemplary situation that may lead to event bursts. The sensor array 10 observes a sudden brightness change along a line spanning eight pixels, resulting in eight events EV1-EV8. In this example, the pixels belong to exactly three columns and eight rows.
[0017] In a typical event-based sensor, a request line REQ monitored by an arbiter 12 is assigned to each row of the pixel array. The pixel that detects the event immediately enables the request line by pulling the request line of the pixel row to the activity level. If the request line is already active, any subsequent event in the same row has no effect on the request line. The arbiter eventually services the request by enabling the confirmation line ACK shared by the pixel row, which causes the event data of the entire row to be transferred to the latch of the readout circuit 16 and clears the request line. Only the data of the active event of the row is parsed by the readout circuit and cascaded into the event stream ESTRM with the current timestamp TS and the X, Y coordinates provided by the position of the active event in the readout row and the level of the current request line served by the arbitrator.
[0018] If the arbitrator does not have time to service the request of the first row before the event occurs in other rows, thereby generating subsequent requests, these subsequent requests are placed in the request queue 18 and serviced later. The entry of the request is performed asynchronously by the combinational logic in the arbitrator. The described events EV1-EV8 may be such a case, thereby generating eight different requests asynchronously. The arbitrator and control circuit can only synchronously service one such request at a time at the speed of the system clock CK. If the event occurs faster than the system clock, the arbitrator serves the first request and puts the other requests in the queue. According to a given priority rule, preferably a first-in-first-out rule, the remaining requests are served. Other priority rules can be implemented, especially in the case of an asynchronous order in which it is difficult to track the occurrence of requests, such as a fair arbitration rule based on the position of the first row served, so that only when other rows have been served at least once, a given row is served a second time.
[0019] When the arbiter finally services a request in the queue, it is apparent that the timestamp assigned to the corresponding event (which is the read time) lags behind the time of the event. This lag, which translates into timestamp errors in the event stream, is variable and depends on the position of the request in the queue, which can vary between 1 and a variable length that the queue has reached. The length of the queue depends on the number of rows affected by a burst of events that occurs within one cycle of the system clock.
[0020] The position of a request in the queue does not generally provide a direct indication of lag. In fact, if events occur simultaneously, such as when a flash of light is observed in a scene, requests will be enqueued at different positions that all represent the same time of occurrence. Thus, position N in the queue may represent any time lag between one system clock cycle and N clock cycles when serviced by the arbitrator.
[0021] Furthermore, while a pending request is waiting in the queue, more recent events may have occurred in the corresponding row. Since the pending request line is still active, the new event does not change anything. Therefore, the same pending request can represent different event occurrence times within the time it takes to service the request.
[0022] This article provides the use of an event burstiness indicator based on queue length to apply burst mitigation measures. The burstiness indicator can be just the current queue length, or it can be a statistical indicator representing the evolution of the queue length.
[0023] In a simple embodiment, a burstiness indicator can be inserted into the event stream along with the event data, wherein the indicator transmitted with the current event indicates the burstiness to be considered for a set of previously transmitted events. The indicator can also be transmitted over an available hardware interface. A downstream processor of a sensor or application can use the burstiness indicator to adjust the event data processing to account for possible timestamp errors or to apply processing that does not require accurate timestamps. In any case, the downstream processing can use the burstiness indicator to detect the return to normal state after an event burst, where the timestamp can be trusted again. In fact, in some cases, the burst of events is usually ignored, so the timestamp information is irrelevant, but it is useful to know when the burst begins and ends.
[0024] Figure 2 is a block diagram of an event-based vision sensor implementing another embodiment of event burst mitigation based on request queue length.
[0025] The sensor is Figure 1The present invention is a complementary version of the sensor of FIG. 1 and includes the same elements, such as the pixel array 10, the row request arbiter 12 (here labeled Y-arbiter), the row readout circuit 16 (here labeled X-readout) and the row request queue 18 (here labeled Y-queue). These elements retain their previous functionality, namely, organizing the readout of the pixel rows of the array based on the pixels activating the row request Y-REQ. In other words, these elements are configured to process horizontal pixel lines (rows) aligned along the Y axis.
[0026] In addition, the sensor includes elements for processing the array vertically along the X-axis, i.e., organizing the readout of vertical lines of pixels (columns) based on pixels activating column requests X-REQ. These elements include a column request arbiter X-arbiter 20 associated with a column request queue X-queue and a column readout circuit Y-readout 24. Thus, the array 10 is completed by a vertical column request line X-REQ connected to the x-arbiter 20 for each column of pixels and a plurality of horizontal lines (not shown) connected to the y-readout circuit 24 for parallel readout of events for the columns served by the y-arbiter 20. The event stream output by the sensor is generated by merging the outputs of the x-readout circuit and the y-readout circuit, which circuits can be operated one at a time, as described later.
[0027] Basically, these additional elements operate with columns of pixels similar to how traditional elements operate with rows of pixels.
[0028] In addition, the control circuit of the sensor includes a queue monitor QMON 26, which monitors the row request queue Y-queue and the column request queue X-queue, and each arbiter is assigned a queue length counter YQLEN, XQLEN. Each queue length counter may have a parallel counter structure with one input for each request line and an output encoding the number of inputs in the active state. For example, for a sensor with a 1280x728 pixel array, the counter YQLEN has 728 inputs and a 10-bit output for encoding any number between 0 and 727, and the counter XQLEN has 1280 inputs and an 11-bit output.
[0029] An exemplary occurrence of events EV1-EV8 is again shown in array 10. It is assumed that these events occur simultaneously or at least within one read cycle (or system clock cycle CK). According to the conventional row method, the events occur in eight different rows and thus enable eight different row request lines Y-REQ that cannot be immediately serviced. The y-arbiter places the request in the y-queue, which reaches a size of 8 (assuming that the first of the requests is not immediately serviced).
[0030] In parallel, events EV1-EV8 occurring in three different ranks enable three different rank request lines X-REQ. The x-arbiters connected to these request lines place these requests in the x-queues.
[0031] Enqueuing requests into the x-queue and y-queue occurs asynchronously and quasi-simultaneously.
[0032] When it is time to service a queue, the queue monitor 26 compares the queue lengths of the x-queue and y-queue, and enables the arbitrator with the shortest queue (here, the x-arbitrator) to service its queue, while the other arbitrator (here, the y-arbitrator) is disabled and its request queue is reset. The x-arbitrator then services its three requests, representing eight pending events in the array, saving five cycles compared to the case where the y-arbitrator processes the events alone.
[0033] The following table further illustrates the above operations in successive cycles of the system clock CK.
[0034]
[0035] Just before cycle time t0, events EV1-EV8 have triggered corresponding x-requests and y-requests, which have been placed in the x-queue and y-queue, and the queue lengths have been compared by the queue monitor QMON, where the x-arbiter is enabled. These operations occur asynchronously through combinatorial logic.
[0036] At time t0, the first column in the x-queue with events EV1 and EV2 is served, the x-queue length is decremented to 2, and the y-queue is reset.
[0037] At time t1 , the next column in the x-queue with events EV3 - EV5 is served and the x-queue length is decremented to 1 .
[0038] At time t2, the last queue in the x-queue with events EV6-EV8 is serviced, and the x-queue length is decremented to 0. At this point, all requests involved in the previous comparison have been serviced, thereby initiating a new comparison. In this example, no additional requests have been queued, so the system is idle, and the queue monitor performs comparisons at each new cycle to detect new queue length imbalances.
[0039] When the x-arbitrator serves its request, new events may occur, such as events E9-E11 occurring in two rows and three columns, as shown in Figure 2. For clarity, assume that these rows and columns are different from those of events E1-E8. As previously described, both the x-arbitrator and the y-arbitrator are configured to enqueue new requests, but only one arbitrator is active at a time to service its requests. In other words, the disabled arbitrator continues to detect requests and enqueue them in preparation for the next comparison.
[0040] The following table shows the additional impact of events EV9-EV11, assuming that all events occur slightly earlier than time t1.
[0041]
[0042]
[0043] Thus, at slightly earlier than time t1, the occurrence of events EV9-EV11 causes the arbiter to increment the y-queue by 2 and the x-queue by 3, resulting in lengths of 2 and 5 for the y-queue and x-queue, respectively.
[0044] At time t1, the column with events EV3-EV5 is serviced and the x-queue is decremented to 4.
[0045] At time t2, the column with events EV6-EV8 is serviced, and the x-queue is decremented to 3. All x-requests involved in the previous comparison have been serviced, thereby enabling a new comparison.
[0046] At time t3, queue lengths 2 and 3 are compared asynchronously, resulting in the activation of the y-arbiter for servicing its y-queue, and resetting the x-queue. In the same cycle, synchronously, the row with event EV9 is served and the y-queue is decremented to 1.
[0047] At time t4, the last row with events EV10 and EV11 is serviced and the y-queue is decremented to 0. A new comparison is enabled.
[0048] In the foregoing description, the x-queue length and the y-queue length are directly compared to organize the switching between the row operation mode and the column operation mode. In some cases, switching between modes may cause some overhead, thereby increasing the delay of processing the queue. In such cases, it may be preferred to switch modes when one queue length exceeds the other queue length by a threshold value greater than the resulting overhead in the number of cycles.
[0049] Figure 3 is a block diagram of an event-based visual sensor, which shows another embodiment of queue length-based event burst mitigation. The sensor can be based on a traditional structure organized to process events in rows and include Figure 2The queue monitor 26 and the row queue length counter YQLEN are configured to selectively implement a row sub-sampling mode.
[0050] When the current queue length exceeds a threshold, the queue monitor 26 switches the arbiter from normal mode to a sub-sampling mode in which the arbiter only services a limited number of requests corresponding to a region of interest ROI in the pixel array. The region of interest may be, for example, one row in every N rows, such as one row in every three rows, as shown. Unserved requests in the queue are reset, i.e., cleared without being read out, and removed from the queue, thereby reducing the queue length. As soon as the queue length drops below a threshold, normal mode may be restored. The reset request may be performed stepwise, i.e., by resetting unserved requests preceding the currently serviced request in the queue, or the reset request may be performed globally, i.e., by resetting all requests at once that will not be serviced once the sub-sampling mode is entered.
[0051] Such a configuration responds to bursts of events by ignoring multiple events in the event burst, thereby reducing sensor resolution and improving timestamp accuracy.
[0052] The region of interest can be a set of consecutive rows alternating with groups of rows to be ignored, or a sub-matrix separated by groups of rows to be ignored and groups of columns to be ignored. In the latter case, the columns to be ignored are masked when each row is read out, thereby potentially reducing the number of active events to be transmitted for each row, which will reduce downstream processing resource consumption when event bursts occur.
[0053] Can be along Figure 2 This subsampling technique is used for both axes of the sensor (ie, for both rows and columns).
Claims
1. An event-based visual sensor comprising: an array of event-based pixels (10), each pixel being configured to record an event based on a change in sensed photocurrent, and Upon occurrence of an event, enabling a request line common to pixels aligned along an axis of the array; a request arbiter (12, 20) configured to asynchronously respond to enabled request lines by enqueuing corresponding line read requests into a request queue (Y-queue), and Servicing the request queue in order; An event readout circuit (16, 24) is connected to the request arbiter and is configured to events for reading out in parallel the lines of pixels corresponding to the current line readout request serviced by the arbiter, and transmitting said read events in an event stream (ESTRM) with a timestamp indicating the time of the read; a queue length counter (YQLEN) coupled to the request line to generate a request queue length as the number of enabled request lines; a queue monitor (26) connected to the queue length counter and configured to generate an event burstiness indicator based on a current request queue length; as well as Means are configured to perform a burstiness compensation action based on the event burstiness indicator.
2. The sensor of claim 1 , wherein the readout circuit is configured to insert consecutive event burst indicator values in the event stream, and the sensor further comprises a processor that receives the event stream and is programmed to apply processing measures to the event stream in response to the event burst indicator values.
3. The sensor of claim 1 , wherein the burstiness indicator is a queue length, and when the queue length exceeds a threshold, the arbiter is controlled to subsample the request queue so that only requests corresponding to a region of interest of the pixel array are served, and the resulting unserved requests in the request queue are reset.
4. The sensor according to claim 1, comprising: a row request arbiter (12) configured to service row read requests queued in a row request queue (Y-queue); a row readout circuit (16), the row readout circuit being associated with the row request arbiter; a column request arbiter (20) configured to service column read requests enqueued in a column request queue (X-queue); a column readout circuit (24), the column readout circuit being associated with the column request arbiter; a row queue length counter (YQLEN) and a column queue length counter (XQLEN), the row queue length counter and the column queue length counter being assigned to the row request queue and the column request queue, respectively; A request queue monitor (26) associated with the row request arbitrator and the column request arbitrator to: When the row request queue length exceeds the column request queue length by a threshold, activating the row request arbiter to serve the row request queue; and When the column request queue length exceeds the row request queue length by a threshold, the column request arbiter is activated for servicing the column request queue.
5. The sensor of claim 4 , wherein the row request arbiter and the column request arbiter are configured to operate simultaneously to queue corresponding row readout requests and column readout requests, and the request queue monitor is configured to reset the request queue of the arbiter switching from an active state to an inactive state.
Citation Information
Cited By
Complementary metal oxide semiconductor (CMOS) image sensor of mixed event and frame pixel and reading method of CMOS image sensor
CN119383483A
