Method for processing a sequence of events received asynchronously from a pixel array of an event-based light sensor - Patent 7732637

By grouping events into time slots and using maps to update data structures in batches, the method optimizes memory access for event-based sensors, enhancing computational efficiency and maintaining temporal accuracy.

JP7584425B2Active Publication Date: 2024-11-15プロフジー
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2021546751
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-02-11
Filing Date
2020-02-10
Publication Date
2024-11-15
Estimated Expiration
2040-02-10

AI Technical Summary

Technical Problem

Existing event-based sensors consume excessive computing resources due to the need for frequent memory access to process asynchronous event data chronologically, limiting their efficiency.

Method used

A method for processing event data by grouping events into time slots, using a map to update data structures in batches, reducing memory access by leveraging faster local memory for processing.

Benefits of technology

This approach significantly reduces memory access frequency, optimizing computational resources and maintaining temporal accuracy within acceptable limits, enabling efficient use of event-based sensors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584425000001
    Figure 0007584425000001
  • Figure 0007584425000002
    Figure 0007584425000002
  • Figure 0007584425000003
    Figure 0007584425000003
Patent Text Reader

Abstract

The method processes a series of events asynchronously received from a pixel array of an event-based light sensor, each event in the series including an address of a pixel in the array from which the event is received and an attribute value dependent on incident light sensed by that pixel. The method includes storing a data structure in a first memory, the data structure including event data for at least some of the pixels of the array, the event data for a pixel associated with at least one event most recently received from the pixel; building a map for the pixels of the array during a time slot as the series of events is received in a second memory having faster access than the first memory, the map including an information element for each pixel of the array, the information element having one of a plurality of values ​​including a zero value indicating an absence of an event during the time slot; and updating the data structure after the time slot using the map.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to machine vision, and more particularly to a method for processing a sequence of events received asynchronously from a pixel array of an event-based optical sensor. [Background technology]

[0002] Machine vision is a field that includes methods for collecting, processing, analyzing, and understanding images for use in many types of applications, such as, for example, security applications (e.g., surveillance, intrusion detection, object detection, facial recognition, etc.), environmental use applications (e.g., lighting control), object detection and tracking applications, automated inspection, process control, and robotic guidance. Machine vision can therefore be integrated with many different systems.

[0003] Technically, the above applications are achieved by using frame-based image sensors, which collect visual information from a scene, either sequentially for each sensor element (pixel) or for each row / column of pixels, or in various patterns, pixel-parallel, but always time-sampled at some limited frame rate. Therefore, frame-based image sensors have limitations in many applications, such as in fast motion rendering and stereo matching.

[0004] Event-based sensors were developed to overcome the limitations of frame-based image sensors. Instead of capturing an image with all pixels simultaneously and trying to compensate for artifacts and suppress information, event-based image sensors collect the temporal intensity profile sensed by each pixel from the scene seen by the sensor. Event-based sensors are able to capture fast-changing information in a visual scene due to the high temporal resolution of the information collected by pixels individually. A more detailed description of such event-based machine vision techniques is given in Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B., and Delbruck, T., et al. (2014), "Retinomorphic event-based vision sensors: bioinspired cameras with spiking output," Proceedings of the IEEE, 102(10), 1470-1484.

[0005] Since event-based sensors output asynchronous events, some representation of the stream of events is necessary to enable various applications. Such a representation cannot be the same as a "frame" used in a conventional camera.

[0006] In general, for each pixel of the sensor, some data is stored about one or more events received from the pixel during a time window. Such data can include histogram data, i.e., how many events were received from the pixel during the time window, timestamps of the most recent events received from the pixel, etc. Data can be stored in two channels to distinguish between positive or negative polarity events, i.e., to indicate whether they increase or decrease the light intensity sensed by the pixel. More elaborate data types can be stored, such as those disclosed in "EV-SegNet: Semantic Segmentation for Event-based Cameras," I. Alonso and AC Murillo, arXiv:1811.12039v1, November 29, 2018.

[0007] A data structure so derived from a stream of events can be accessed using pixel addresses, and thus can encode spatial relationships between events. It is typically stored in a RAM (random access memory) associated with the processor that processes the incoming stream of events. The processor receives events chronologically from a spatially uncorrelated array of pixels. This means that an access to the RAM occurs every time an event is received, which consumes a lot of processing resources. Local memory concepts such as caching cannot be used conveniently. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] US Patent Application Publication No. 2008 / 0135731 [Patent Document 2] European Patent Application No. 18305063.2 [Patent Document 3] International Publication No. 2013 / 093378 [Patent Document 4] U.S. Pat. No. 9,628,777 [Patent Document 5] International Publication No. 2016 / 146938 [Patent Document 6] European Patent Application No. 19154401.4 [Non-patent literature]

[0009] [Non-Patent Document 1] Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B., and Delbruck, T. et al. (2014), "Retinomorphic event-based vision sensors: bioinspired cameras with spiking output", Proceedings of the IEEE, 102(10), 1470-1484 [Non-Patent Document 2] "EV-SegNet: Semantic Segmentation for Event-based Cameras", I. Alonso and A.C. Murillo, arXiv:1811.12039v1, November 29, 2018. [Non-Patent Document 3] "A 128×128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor", P. Lichtsteiner et al., IEEE Journal of Solid-State Circuits, Vol. 43, No. 2, February 2008, pages 566-576 [Non-Patent Document 4] “A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS,” C. Posch et al., IEEE Journal of Solid-State Circuits, Vol. 46, No. 1, January 2011, pages 259-275. Summary of the Invention [Problem to be solved by the invention]

[0010] For processing event-based data, new approaches are needed to improve efficiency in the use of computing resources, especially with regard to memory access. [Means for solving the problem]

[0011] We propose a method for processing a sequence of events received asynchronously from a pixel array of an event-based light sensor, each event in said sequence comprising the address of a pixel of the array from which the event is received and an attribute value that depends on the incident light sensed by said pixel. The method comprises: storing a data structure including event data for at least a portion of the pixels of the array, the event data for the pixels relating to at least one event most recently received from the pixels; constructing a map for pixels of the array as the sequence of events is received during a time slot, the map including an information element for each pixel of the array, the information element having one of a plurality of values ​​including a zero value indicating an absence of an event during the time slot; and updating the data structure after the time slot using the map.

[0012] Grouping events within a time slot has the fundamental benefit of removing chronological ordering within the group, and therefore allows the use of more convenient ordering for computational and architectural needs.

[0013] Typically, the time slots are relatively short, e.g., less than 2 milliseconds. Once the time slot has expired, the map is used to update the data structure. Because the addressing scheme for both the map and the data structure is based on pixel location (rather than event chronology), the updates can be performed efficiently using an appropriate memory architecture associated with the processor managing performance of the method. The memory area in which the data structure is stored is accessed to be updated in batches. The number of memory accesses is dramatically reduced, even when the time slot is short.

[0014] Typically, the data structures are stored in a first memory and the map is constructed in a second memory that has faster access than the first memory.

[0015] For example, the memory in which the map is built may be a local memory, while the data structures are stored in a global memory. The local memory may also be referred to as a level 1 (L1) memory or an on-chip memory. In one possible embodiment, the global memory is a random access memory (RAM memory) and the local memory is a cache memory.

[0016] Without being limited thereto, using the map to update the data structure after a time slot may include accessing the data structure stored in raster scan order.

[0017] A time slot may have its duration adapted based on the events received from the pixel array.

[0018] In one embodiment, using the map to update the data structure after a time slot includes modifying the event data for each pixel for which the map has an information element with a value other than a zero value.

[0019] For example, modifying the event data for a pixel may include setting the event data for the pixel by referencing a timestamp defined for a timeslot. In this case, some time accuracy is lost because event timing changes within a timeslot are ignored. However, it is not detrimental to the application if the timeslot duration is appropriately selected. Referencing a timestamp defined for a timeslot has the beneficial effect of reducing noise on the event's timestamp.

[0020] In one embodiment, constructing the map during the time slots comprises: initializing the map with zero values ​​of information elements for all pixels of the array; when receiving an event during a timeslot from a pixel for which an information element included in the map has a zero value, setting the value of the information element included in the map for said pixel based on the attribute value of the received event; and outputting the map when the time slot ends.

[0021] Constructing the map during a timeslot may further include, upon receiving an event during the timeslot from a pixel for which an information element included in the map has a value other than a zero value, setting a value of an information element included in the map for said pixel based on an attribute value of the received event.

[0022] If the attribute value of an event received from a pixel of the array includes a polarity of the event having a first value, such as +1, when the light sensed by the pixel increases and a second value, such as -1, when the light sensed by the pixel decreases, then the information element included in the map for the pixel from which the event was received during the time slot may include the polarity of the event.

[0023] If the attribute value of an event received from a pixel of the array includes a measure of the light intensity sensed by the pixel, the information elements included in the map for a pixel from which an event is received during a timeslot may include elements that depend on the measure of the light intensity of that event.

[0024] Another aspect of the invention relates to a processor for coupling to a pixel array of an event-based light sensor, the processor being configured to implement the method defined above for processing a sequence of events received asynchronously from the pixel array.

[0025] Another aspect of the invention relates to a method for 3D reconstruction of a scene, the method comprising: positioning at least first and second event-based light sensors in front of the scene at spatially separated locations, the first event-based light sensor having a pixel array that provides a first set of events and the second event-based light sensor having a pixel array that provides a second set of events; processing each of the first and second series of events in accordance with the method defined above to obtain a first and second data structure, respectively; and stereo matching a pair of events received from the first and second event-based light sensors, respectively, using the first and second data structures.

[0026] Other features and advantages of the present invention will become apparent during the course of the following description which refers to the accompanying drawings. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram of a device adapted for implementation of the present invention. [Diagram 2] 4 is a flow chart of an exemplary method according to the present invention. [Diagram 3] 3 is a flow chart of a stereo processing method according to the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The apparatus shown in Figure 1 includes an event-based asynchronous vision sensor 10 positioned facing a scene and receiving light from the scene through collecting optics 15, which include one or more lenses. The sensor 10 is positioned in the image plane of the collecting optics 15. It includes an array of sensing elements, such as photosensitive elements, arranged into a matrix of pixels. Each sensing element, which corresponds to a pixel, creates a succession of events in response to variations in the light in the scene.

[0029] The processor 12 processes the information coming from the sensor 10, i.e. for example the sequence of events received asynchronously from the various pixels. It can be implemented by programming using a suitable programming language. A hardware implementation of the processor 12 using dedicated logic circuits (ASIC, FPGA, ...) is also possible.

[0030] The device further includes at least a first and a second memory 16, 18, which in the illustrative example shown in FIG. 1 are respectively a RAM 16 coupled to the processor chip via a bus and an on-chip memory area 18 managed by the processor 12 according to a cache scheme. The two memories 16, 18 may be accessed at different speeds by the central processing unit (CPU) of the processor 12, i.e., access to the second memory 18 is faster than access to the first memory 16. It will be appreciated that other memory architectures and technologies are possible within the context of the present invention. For example, the two memories 16, 18 may be external to the processor chip. Alternatively, they may both be part of the same processor chip.

[0031] A first memory 16, hereafter referred to indefinitely as RAM, is used to store a relatively large amount of data that is written and / or read by the processor 12. A second memory 18, hereafter referred to indefinitely as cache memory, temporarily stores a more limited amount of data that needs faster access by the processor 12. It generally has a smaller storage size than the RAM 16 due to its performance and cost.

[0032] The processor 12 and memory shown in Figure 1 are integrated away from the event-based sensor 10. Alternative architectures are possible. In particular, some or all of the processing shown in Figure 2 may be performed in the same component as the event-based sensor 10, in an on-chip architecture. It is also possible to divide tasks between two or more processors operating sequentially or in parallel.

[0033] For each sensing element, or pixel, the sensor 10 generates a sequence of event-based signals in response to variations in light received by the pixel from a scene appearing within the sensor's field of view.

[0034] The asynchronous sensor 10 calculates for each pixel a series of instants t at which an activation threshold Q is reached. kAcquisition is performed to output a signal that can be of the form (k=0, 1, 2, ...). At time t k Starting from the luminance at k+1 is identified, and the spike occurs at this moment t k+1 In contrast, the intensity observed by a pixel changes over time t k Starting from the luminance at k+1 is identified, and the spike occurs at this moment t k+1 The signal sequence for a pixel is determined according to the light profile for the pixel at instant t k The signal sequence generally includes a series of spikes spaced over time at . Without being limited thereto, the output of the sensor 10 is then in the form of an Address Event Representation (AER). In addition, the signal sequence generally includes a brightness attribute that corresponds to the variations in the incident light.

[0035] The activation threshold Q may be fixed or may be adapted as a function of luminance, for example, the threshold may be compared to the variation in the logarithm of luminance to generate an event when exceeded.

[0036] By way of example, the sensor 10 may be a dynamic vision sensor (DVS) of the type described in "A 128x128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor", P. Lichtsteiner et al., IEEE Journal of Solid-State Circuits, Vol. 43, No. 2, February 2008, pp. 566-576, or in patent application US 2008 / 0135731 A1. The dynamics of the human retina (the minimum duration between action potentials) can be addressed with this type of DVS. Its dynamic behavior exceeds that of conventional video cameras with limited sampling frequencies. When the DVS is used as an event-based sensor 10, data related to an event arising from a pixel includes the address of the pixel, the time of occurrence of the event, and the polarity of the event, e.g., a brightness attribute corresponding to +1 for an increase in brightness and -1 for a decrease in brightness.

[0037] Another example of an asynchronous sensor 10 that may be advantageously used in the context of the present invention is the Asynchronous Time-Based Image Sensor (ATIS), described in "A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS," C. Posch et al., IEEE Journal of Solid-State Circuits, Vol. 46, No. 1, January 2011, pp. 259-275. When the ATIS is used as an event-based sensor 10, data related to an event originating from a pixel includes the address of the pixel, the time of occurrence of the event, and a brightness attribute corresponding to an estimate of the absolute brightness. The polarity of each event may also be output by the ATIS.

[0038] The processing methods disclosed herein utilize a data structure stored in a first memory or RAM 16 that contains event data for some or all of the pixels in the array of sensor 10 .

[0039] For a given pixel, the event data relate to the most recent events. They also relate to the number of events most recently received from that pixel. For example, the event data may relate to the N most recent events from the pixel, where N is a fixed positive integer. The event data may relate to any events received from the pixel within a window of fixed duration T ending at the current time. The parameter T may be fixed (e.g., T=10 ms as an example) or may be locally or globally adaptable as a function of the events generated by the array. The parameter T may be different for different regions of the pixel array, such that it is variable as a function of the incoming light sensed by the pixels across the array.

[0040] Thus, at any point in time, the data structure keeps track of recent events, which may be useful for signal processing applications taking advantage of event-based sensors.

[0041] As an example, a data structure might consist of an address represented as an index i, and a coordinate in an array (x i , y i ) may include one or more of the following items forming the event data for a pixel having: - For each most recent event ev(i,t i ) at timestamp t i . - the luminance value L associated with the pixel at address i i , For ATIS type sensors, the brightness value L i is the most recent event ev(i,t i ) is available. - For each recent event ev(i,t i ) polarity p i , polar p i is generally the event ev(i,t i ) is available. - The most recently received event ev(i,t i ) for a pixel at address i, such frequency information may be estimated using, for example, the method disclosed in European Patent Application No. 18305063.2. - Recently received events ev(i,t i ) to find the position (x i , y i ), such visual flow data may be estimated using, for example, the method disclosed in WO 2013 / 093378 A1. - For each recent event ev(i,t i ), the depth value, which may also be expressed as a stereo disparity value, may be obtained from signal flows from two or more event-based sensors viewing the scene from different angles. - The most recent event ev(i,t i ) is considered to belong to the cluster of events. - the above parameter T representing the duration of said time window, if parameter T can vary from one pixel of the array to another.

[0042] An event ev(i,t) received from address i in the array i ) at timestamp t i Instead of directly storing i , t cur ), etc. i The following functions can be included in the event data: where f is a function and t cur is the current time. The function f can be an exponential decay with parameter τ, so that the above formula becomes σ(i)=exp[-(t cur -t i ) / τ] become.

[0043] Alternatively, the function f can be σ(i)=t i if t cur ≦t i + τ, and σ(i)=(t cur -τ) if t cur >t i + τ (meaning there are no recent events from pixel i) It could be.

[0044] Some items of event data are time stamped t i , polar p i , luminance value L i Some of the information that may be updated may be directly linked to one event, such as depth value (if available for the event), cluster ID, etc. Other optional items may be calculated based on recent events to be associated with pixel location i, such as flickering frequency, visual flow data, etc. In general, the calculation of these other items may not be as fast as the time dynamics enabled by the event-based sensor 10. Their update frequency is determined by the application.

[0045] It will be appreciated that other types of information may be included in the data structure depending on the function provided by the event-based sensor itself and / or the processing performed on the event-based signal flow. The contents of the data structure may also be adapted depending on the application using the method.

[0046] Moreover, the data structure is organized into two channels, one for timestamp t iis for the positive polarity that is stored for the most recent event of positive polarity, and one is for the timestamp t i is for negative polarity stored for the most recent event of negative polarity.

[0047] In one example, FIG. 2 illustrates a flowchart of a method that may be performed by processor 12 to process a sequence of events received asynchronously from a pixel array of event-based light sensor 10.

[0048] In this method, we refer to consecutive time slots with fixed duration Δt. The time slots are denoted by an integer index k (k=1, 2, 3, ...). At each time slot, a map is constructed taking into account updating the event data stored in the data structure.

[0049] In the example of FIG. 2, the map consists of two channels and is generated as two respective bitmaps, one {M+(i)} for positive polarity events and one {M-(i)} for negative polarity events. At the end of a time slot, M+(i)=0 if no positive polarity events were received from a pixel at address i during the time slot, otherwise M+(i)=1, and M-(i)=0 if no negative polarity events were received from a pixel at address i during the time slot, otherwise M-(i)=1. As a result, the map {(M+(i), M-(i))} can be used to update event data after a time slot. The information elements (M+(i), M-(i)) of the map at pixel locations i that do not have an event during the time slot have zero values ​​(M+(i), M-(i))=(0, 0).

[0050] The duration Δt of a timeslot can be seen, without limitation, as the maximum time interval such that two events occurring at the same pixel within the interval are indistinguishable from a computational point of view. As a result, the duration Δt depends on the characteristics of the sensor (jitter, noise, etc.), but also on the characteristics of the signal processing algorithm for the subsequent application that utilizes the data structure. An increase in Δt results in some loss of temporal accuracy, but uses less computational resources, and vice versa. A typical range for Δt is between 250 μs and 2 ms.

[0051] An interesting option for the value of Δt is to make it adaptive, i.e., to change it during processing by some function on the previously received data. This can be described using a Boolean function f(E)=true / false, where E is the set of events received so far in the current timeslot, and true / false indicates whether the current timeslot should end and a new timeslot should start. For a fixed timeslot of duration Δt, f(E) can be defined as f(E)=last_event_time-prev_timeslot_last_time>Δt. In an example of the adaptive case, the function f(E) defines a timeslot in terms of a fixed number of events N, i.e., f(E)=number_of_events_received_since_start_timeslot≧N. In other embodiments, the function f(E) can count events with some weights that give more importance to some regions (e.g., the center) of the pixel array. It can also be learned by using machine learning algorithms.

[0052] The map {(M+(i), M-(i))} is of modest size, so it can be generated by the processor 12 without accessing RAM 16, but by utilizing cache memory 18 (or some other more local and / or faster type of memory).

[0053] In FIG. 2, step 20 corresponds to the start of time slot k, where the map {(M+(i),M-(i))} is initialized as (M+(i),M-(i))=(0,0) for all pixels i. In one embodiment, the start of time slot k is at time k×Δt, i.e., the time slots follow each other uninterrupted. Alternatively, the start of time slot k may be triggered by receiving the first event from the pixel array after time slot k-1 ends.

[0054] In an ongoing timeslot k (it is checked in test 21 in the example of FIG. 2 whether Δt has passed since step 20), each event received from pixel position i in step 22 causes the value 1 to be set to M−(i) (negative polarity, step 24) or to M+(i) (positive polarity, step 25), depending on the polarity of the event, which is checked in test 23. When the end of the timeslot is detected in step 21, the map {(M+(i),M−(i))} for timeslot k is output, i.e. it is made available in the cache memory 18 for the next step 26.

[0055] In steps 24 or 25, it is possible to perform a write operation M±(i)=1 only if the value of M±(i) was previously 0. This does not change the resulting map, since the method makes no difference between two events of the same polarity received in a timeslot.

[0056] After the time slot has ended, processor 12 proceeds to update the event data of the data structure at pixel location i if the information element of the map does not have a zero value, i.e., (M+(i),M-(i))≠(0,0). If M+(i)=1 or M-(i)=1, then processor 12 updates the new timestamp t associated with each pixel location i. i The update is performed in step 26 by using the time slot index k as .

[0057] Using the time slot index k as a time stamp reduces the time resolution in receiving the events. However, there is no disadvantage to the reduced time resolution, provided that the time slot duration Δt is selected small enough having regard to the application utilizing the data structures stored in RAM 16. Inherent variations in the operation of the pixel array, which result in noise and jitter, can be compensated for through the use of the discrete time slot index k as a time stamp.

[0058] The update 26 consists of successively accessing spatial regions of the data structure from the RAM 16, reading the event data for each region into the cache memory 18, and updating the data structure by time stamp t i =k to the negative channel if M-(i)=1 and to the positive channel if M+(i)=1, and writing the updated event data for the field back to RAM 16. This is much more efficient than accessing RAM 16 every time an event is received.

[0059] In the simple case where only the timestamp of the most recent event of each polarity is stored as the event data for pixel i, the update 26 simply stores t i If more than one event is stored in the data structure for pixel i and a given polarity, a new timestamp t i =k is written and any previous timestamps for the same pixel are shifted out in a FIFO manner.

[0060] In the above embodiment, the event data for pixel i consists of two bits M+(i), M-(i). If more than two bits per pixel are available in the cached map, it is possible to encode more information at the time of receiving the event.

[0061] For example, with each event, a luminance value Li For ATIS light sensors that provide luminance, the luminance value or a sub-sampled version of it may be stored in the map in steps 24 or 25. This allows the event data to hold the current value of luminance for each pixel.

[0062] For some applications, it may be sufficient to have only one bit in the map information element for a given pixel that indicates whether or not an event was received from the given pixel during the time slot, regardless of the polarity of the event.

[0063] There are many applications of the above method of maintaining event data in RAM 16 using a map stored in cache memory 18 .

[0064] 3 shows an example of an application for 3D stereo processing based on two event-based sensors. The application utilizes at least two event-based light sensors 10 placed in front of a scene at spatially separated locations, where one of the event-based light sensors 10 is called the left camera and the other is called the right camera.

[0065] The left camera produces a first stream of events which are processed according to a method such as that described with reference to FIG. 2. The same method is applied to a second stream of events produced by the right camera. The time axis is divided into time slots k and for each time slot a respective map is generated from each of the first and second streams of events in stages 31 and 32 which may correspond to steps 20-25 shown in FIG. 2. The two maps are then used in steps 33 and 34 (corresponding to step 26 in FIG. 2) to obtain a first data structure for the left camera and a second data structure for the right camera.

[0066] Then, in step 35, processor 12 performs stereo matching of the pair of events from the two cameras using the first and second data structures updated in steps 33, 34.

[0067] The stereo matching methods disclosed in US Pat. No. 9,628,777 (B2) and WO 2016 / 146938 A1 may be used in step 35. In these methods, events from one of the two cameras are considered, and for each of these events from a given pixel, candidate events are checked along the epipolar line that corresponds to the pixel in the array of the other camera. For each candidate event, a metric is calculated, and if a criterion is satisfied (e.g., metric within a threshold), a match is confirmed with the best candidate according to the metric.

[0068] The use of maps calculated for consecutive time slots also provides advantages in the implementation of the stereo matching step 35. After each time slot, only pixels whose information elements do not have zero values ​​need to be considered for both cameras, since matched events are expected to be present at the same time. Taking into account factors such as synchronization noise, jitter, and scene dynamics, the notion of "simultaneous" events from the two sensors is covered by events received in the same time slot or in a limited number of consecutive time slots, which limits the number of memory accesses required. If the metric used for stereo matching requires event data to be restored from a data structure, the event data can be retrieved from the RAM 16 by groups of pixels arranged around the epipolar line, and matching can be performed in batches, limiting RAM memory accesses.

[0069] Once stereo matching is performed, a depth value may be assigned to the pixel where the event was received in the last time slot, and such depth value may be stored in a data structure in RAM 16 to provide 3D positioning of the event.

[0070] Another possible example of an application is to label connected components using event-based sensors. The data structure obtained by the present invention can be used to cluster events into connected components in order to extract and segment objects of interest in an image or scene for object detection and tracking. An illustrative example for such an application is described in European patent application EP 19154401.4.

[0071] The above method may be implemented using program instructions recorded in a non-transitory computer readable medium to implement various operations that may be executed by a computer. The medium may also include, alone or in combination with program instructions, data files, data structures, and the like. The program instructions recorded on the medium may be those specially designed and constructed for the purposes of the exemplary embodiments, or they may be of a type well known in the art of computer software and available to those skilled in the art. Examples of non-transitory computer readable media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD ROM disks and DVDs, magneto-optical recording media such as optical disks, and hardware devices specially configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of program instructions include both machine code, such as code produced by a compiler, and files containing higher level code that may be executed by a computer using an interpreter. The hardware devices described may be configured to act as one.

[0072] Implementations of the present invention may be used in many applications, including computer-human interaction (e.g., recognition of gestures, voice, posture, face, and / or other applications), controlling a process (e.g., industrial robots, autonomous vehicles and other vehicles), tracking the movement of a set of points or objects of interest (e.g., vehicles or humans) relative to an image plane within a visual scene, augmented reality applications, virtual reality applications, access control (e.g., opening doors based on gestures, opening access roads based on detection of an authorized person), detecting events (e.g., for visual surveillance or humans or animals), counting, tracking, etc. There are numerous other applications that will be recognized by those of skill in the art given this disclosure.

[0073] The above-described embodiments are examples of the present invention. Various modifications can be made thereto without departing from the scope of the invention which arises from the appended claims. [Explanation of symbols]

[0074] 10 Event-based asynchronous vision sensor 12 processors 15 Optical instruments for collection 16 First memory, Random Access Memory (RAM) 18 Secondary memory, cache memory 21 Exams 23 Exams 26 Update

Claims

1. A method for processing a sequence of events received asynchronously from a pixel array of an event-based light sensor (10), comprising the steps of: each event in the series includes an address of a pixel in the array from which the event is received and a polarity dependent on incident light sensed by the pixel; The method further comprising: storing in a first memory (16) a data structure including event data for at least a portion of the pixels of the array, the event data for a pixel including a timestamp and the polarity of at least one event most recently received from the pixel; building, when the series of events is received during a timeslot, a map for the pixels of the array using a second memory (18) having faster access than the first memory (16), the map including an information element for each pixel of the array, the information element including the polarity of a corresponding event received during the timeslot, or a zero value in the absence of a corresponding event received during the timeslot; and updating the data structure after the time slot using the non-zero information elements of the map.

2. 2. The method of claim 1, wherein the data structures are stored in a global memory (16) and the map is constructed using a local memory (18).

3. 3. The method of claim 2, wherein said global memory (16) is a random access memory (RAM) memory, while said local memory (18) is a cache memory.

4. 2. The method of claim 1, wherein using the map to update the data structure after the time slot comprises accessing the stored data structure in a raster scan order.

5. 2. The method of claim 1, wherein said time slots are at most 2 milliseconds long.

6. The method of claim 1 , wherein the time slots have a duration that is adapted based on events received from a pixel array.

7. 2. The method of claim 1, wherein updating the data structure after the time slot using the map comprises modifying the event data for each pixel for which the map has the information element not equal to zero.

8. 8. The method of claim 7, wherein modifying the event data for a pixel comprises setting the event data for the pixel by referencing a timestamp defined for the time slot.

9. constructing the map during a time slot, initializing said map with said information elements having a zero value for all said pixels of said array; upon receiving an event during the time slot from a pixel for which the information element in the map has the zero value, setting the information element for the pixel to the polarity of the received event; and outputting said map when said time slot ends.

10. 10. The method of claim 9, wherein constructing the map during a time slot further comprises, upon receiving an event during the time slot from a pixel corresponding to a non-zero information element in the map, setting the information element to the polarity of the received event.

11. The method of claim 1 , wherein the polarity of an event has a first value when light sensed by the corresponding pixel increases and has a second value when light sensed by the corresponding pixel decreases.

12. the event data for an event received from a pixel of the array includes a measure of light intensity sensed by the pixel; the information element included in the map for a pixel from which an event was received during the time slot comprises a value dependent on the measure of light intensity of the event. The method of claim 1.

13. 13. A processor for coupling to a pixel array of an event-based light sensor, the processor (12) configured to implement the method of any one of claims 1 to 12 for processing a sequence of events asynchronously received from the pixel array.

14. 1. A method for 3D reconstruction of a scene, comprising: placing at least first and second event-based light sensors (10) in front of the scene at spatially separated locations, the first event-based light sensor having a pixel array that provides a first set of events and the second event-based light sensor having a pixel array that provides a second set of events; - processing each of said first and second series of events by the method according to any one of claims 1 to 12 to obtain a first and second data structure respectively; and using the first and second data structures to stereo match pairs of events received from the first and second event-based light sensors, respectively.

Citation Information

Patent Citations

  • Method and apparatus of processing a signal from an event-based sensor

    EP3518529A1

  • Method of processing information from an event-based sensor

    EP3690736A1

  • Event signal processing method and apparatus, and pixel augmentation method and apparatus based on event-based vision sensor

    JP2017130185A

  • Interfacing event-based systems with frame-based processing systems

    JP2017530467A

  • Feature calculation in sensor element array

    JP2018501675A