Event sensing device and method

Abstract

The invention relates to an event camera sensor. To this end, the invention proposes an event sensing device comprising a plurality of pixel sensors, wherein one or more thresholds are associated with each of the plurality of pixel sensors. Each pixel sensor is configured to detect a time-dependent change in intensity of incident light at the pixel sensor, generate an event if the time-dependent change in intensity exceeds any of the one or more thresholds associated with the pixel sensor, wherein each event is associated with a time stamp and a threshold, and generate one or more event streams, each event stream comprising a plurality of the events associated with the same threshold. The event sensing device is further configured to reconstruct a plurality of independent intensity signals from the plurality of event streams generated by the plurality of pixel sensors based on the plurality of thresholds.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to data processing, and more specifically to image processing for event-based cameras. The invention proposes a device and method for sensing events, wherein the device and method can be used for image processing in event-based cameras. BACKGROUND

[0002] Event-based cameras use various types of sensors that respond to changes in incident light intensity. In standard cameras, each pixel captures the amount of incident light at a fixed rate, in contrast, event-based cameras are asynchronous, and only activate when a change in incident light intensity is detected. The output data rate of such cameras is therefore variable. When there is no change in incident light intensity, there are no activated pixels, and therefore no data generated by the camera. For example, in the case of a moving object, the pixels that capture the object intensity will generate events (triggered by a change in incident light sensed by the pixel). The pixels of event-based camera sensors typically capture the log of incident light intensity, which is then further processed in additional sensor circuitry.

[0003] Typically, events are generated in the pixels based on a sensitivity threshold. For noiseless cases (i.e. clean log intensity signals), a small sensitivity threshold is the best choice. However, for real-world noisy cases, it is not easy to choose an appropriate sensitivity threshold. SUMMARY

[0004] In view of the above limitations, embodiments of the invention aim to introduce a method to obtain high quality (i.e. very high signal-to-noise ratio) reconstructed log intensity signals from events. In particular, the goal is to use a more accurate sensitivity threshold. One aim is to enable a larger number of time samples of the reconstructed intensity signal, while maintaining a low noise level.

[0005] The aims are achieved by the embodiments provided in the independent claims appended hereto. Advantageous implementations of these embodiments are further defined in the dependent claims.

[0006] The first aspect of the invention provides an event sensing device, wherein the event sensing device comprises one or more pixel sensors, wherein one or more threshold values are associated with each of the one or more pixel sensors, and each pixel sensor is configured to: detect a time-dependent change in intensity of incident light at the pixel sensor; generate an event if the time-dependent change in intensity exceeds any of the one or more threshold values associated with the pixel sensor, wherein each event is associated with a time stamp and a threshold value; generate one or more event streams, each event stream comprising a plurality of the events associated with the same threshold value; wherein the event sensing device is further configured to: reconstruct a plurality of independent intensity signals from the plurality of event streams generated by the one or more pixel sensors based on a plurality of threshold values.

[0007] The device of the first aspect is able to reconstruct high-quality intensity signals from event streams generated based on a plurality of different sensitivity threshold values. Thus, the sensitivity threshold values can be more accurate. Compared to a conventional approach where each pixel sensor uses a single sensitivity threshold value, the device of the first aspect is able to provide a larger number of time samples for reconstructing an intensity frame while maintaining a low noise level.

[0008] In an implementation form of the first aspect, the time-dependent change in intensity comprises a change in the intensity from one time point to a consecutive time point.

[0009] In an implementation form of the first aspect, each pixel sensor comprises one or more pairs of differentiator and comparator, each differentiator configured to calculate the time-dependent change in intensity of the incident light, and each comparator configured to generate an event if the time-dependent change in intensity exceeds a predetermined threshold value, and configured to generate an event stream.

[0010] Conventionally, a pixel sensor comprises one differentiator and one comparator. The differentiator calculates a difference between a current value and a previous value of the logarithmic intensity. The comparator generates a “+1” event when the difference is larger than a positive threshold value Th (also referred to as sensitivity), and generates a “-1” event when the difference is smaller than a negative threshold value -Th. In this implementation form of the device of the first aspect, the pixel sensor can comprise more than one pair of differentiator and comparator. This means that, when the pixel sensor comprises multiple pairs of differentiator and comparator, the pixel sensor will generate multiple event streams. It is noted that each event stream corresponds to a predetermined threshold value.

[0011] In an implementation form of the first aspect, each pixel sensor comprises at least two pairs of differentiator and comparator, each comparator being provided with a respective predetermined threshold value and configured to generate an event stream based on the respective predetermined threshold value, wherein the predetermined threshold values are different from each other.

[0012] It is to be noted that different threshold values can be set or predetermined for different comparators of the same pixel sensor. If the same threshold value is set for two comparators, two similar event streams can be generated.

[0013] In an implementation form of the first aspect, the event sensing device is further configured to reconstruct an individual intensity signal for each comparator from each event stream generated by the comparator, and to generate a final intensity signal for each pixel sensor by fusing the reconstructed individual intensity signals of at least two comparators of the pixel sensor.

[0014] From each event stream, an individual intensity signal can be calculated. Here, a conventional algorithm can be used to calculate the individual intensity signal. When a pixel sensor comprises multiple comparators, each comparator can generate an event stream, and thus the pixel sensor can reconstruct multiple individual intensity signals. The fusion of the multiple individual intensity signals can obtain a final intensity signal of the pixel sensor. Since different threshold values can be considered when each pixel sensor generates an event stream, more temporal samples of the reconstructed intensity signal can be ensured while a low noise level can also be maintained.

[0015] In an implementation form of the first aspect, the event sensing device is further configured to reconstruct the intensity image from the multiple final intensity signals of the one or more pixel sensors.

[0016] As described above, a respective final intensity signal can be generated for each pixel sensor in the event sensing device. Thus, an intensity image can be reconstructed from multiple final intensity signals, which are generated for multiple pixel sensors comprised in the event sensing device.

[0017] In an implementation form of the first aspect, the event sensing device further comprises a plurality of pixel sensors, wherein the plurality of pixel sensors are grouped into a plurality of super-pixels, wherein each super-pixel comprises at least two pixel sensors, and each pixel sensor of the at least two pixel sensors is provided with a respective predetermined threshold value, wherein the predetermined threshold values are different from each other.

[0018] According to the present embodiment, a super-pixel comprising multiple pixel sensors can be designed. Specifically, the pixel sensors in the same super-pixel can be provided with different threshold values. It is to be noted that the present embodiment can be implemented using existing event sensor implementations (without changing the hardware). It can be regarded as a software implementation for generating an event stream with multiple threshold values for one “pixel” (in particular, a super-pixel).

[0019] In an implementation form of the first aspect, the event sensing device is further configured to reconstruct an individual intensity signal for each pixel sensor from the event stream generated by the pixel sensor, and to generate a final intensity signal for each super-pixel by fusing the reconstructed individual intensity signals of the at least two pixel sensors in the super-pixel.

[0020] Similar to the conventional approach, an individual intensity signal can be reconstructed for each pixel sensor (based on the event stream generated with a single threshold). It is noted that for a super-pixel comprising multiple pixel sensors, multiple reconstructed individual intensity signals can be used to generate a final intensity signal. As mentioned above, the pixel sensors in the same super-pixel can be provided with different thresholds. Thus, in the present embodiment, the different thresholds can also be considered when the event sensing device generates the final intensity signal. In this way, more temporal samples of the reconstructed intensity signal can be obtained while also maintaining a low noise level.

[0021] In an implementation form of the first aspect, the event sensing device is further configured to reconstruct the intensity image from the multiple final intensity signals of the multiple super-pixels.

[0022] Thus, the intensity image can be reconstructed from multiple final intensity signals, which are generated for multiple super-pixels comprised in the event sensing device.

[0023] In an implementation form of the first aspect, the at least two pixel sensors are adjacent.

[0024] It can be found that the pixel sensors belonging to the same super-pixel do not actually collect light from exactly the same point of the scene. Therefore, it is preferred to have super-pixels comprising pixel sensors that are close to each other.

[0025] In an implementation form of the first aspect, each super-pixel comprises four pixel sensors.

[0026] In a specific implementation form, each super-pixel can comprise a pixel array of four pixel sensors (2x2 array).

[0027] In an implementation form of the first aspect, each pixel sensor is configured to detect a temporal dependency of a logarithmic intensity of incident light at the pixel sensor.

[0028] It is noted that the pixel sensor can actually capture the logarithmic intensity of the incident light.

[0029] The second aspect of the present application provides a method for sensing events using an event sensing device, wherein the event sensing device comprises one or more pixel sensors, wherein one or more thresholds are associated with each of the one or more pixel sensors, the method comprising: detecting a time-dependent change in intensity of incident light at the pixel sensors; generating an event if the time-dependent change in intensity exceeds any of the one or more thresholds associated with the pixel sensors, wherein each event is associated with a time stamp and a threshold; generating one or more event streams, each event stream comprising a plurality of the events associated with the same threshold; wherein the method further comprises: reconstructing a plurality of independent intensity signals from the plurality of event streams generated by the one or more pixel sensors based on a plurality of thresholds.

[0030] Implementations of the method of the second aspect can correspond to the implementations of the event sensing device of the first aspect described above. The method of the second aspect and its implementations achieve the same advantages and effects as described above for the event sensing device of the first aspect and its respective implementations.

[0031] The third aspect of the present application provides a computer program product comprising program code for performing the method according to the second aspect and any implementation of the second aspect when implemented in a processor.

[0032] It has to be noted that all devices, elements, units and means described in the present application can be implemented in software or hardware elements or any combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities, are intended to mean respective entities are adapted to or configured to perform the respective steps and functionalities. Even if a specific functionality or step to be performed by an external entity is not reflected in the description of a specific detailed element of that entity performing that specific step or functionality, it shall be clear to the skilled in the art that these methods and functionalities can be implemented in respective software or hardware elements, or in any kind of combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0033] The above described aspects and implementations thereof according to the present application will be explained in the following description of specific embodiments in view of the enclosed drawings, which show, by way of example:

[0034] Figure 1 A block diagram of a pixel sensor provided by embodiments of the present application is shown;

[0035] Figure 2 A noiseless logarithmic intensity signal and two event streams generated using different thresholds are shown;

[0036] Figure 3The noise-free logarithmic intensity signal and two reconstructed logarithmic intensity signals (from Figure 2 The two event flows shown);

[0037] Figure 4 Shown are a noisy log-intensity signal, two event streams generated using two different sensitivity thresholds, and the corresponding reconstructed log-intensity signals from the events;

[0038] Figure 5 An event sensing device provided by an embodiment of the present invention is shown;

[0039] Figure 6 A block diagram of an event sensing device including a pixel sensor provided by an embodiment of the present invention is shown;

[0040] Figure 7 The diagram shows a plurality of pixel sensors (and their sensitivity thresholds) of an event sensing device provided by an embodiment of the present invention.

[0041] Figure 8 A block diagram of an event sensing device including super pixels provided by an embodiment of the present invention is shown;

[0042] Figure 9 shows the input noise logarithmic intensity and the reconstructed logarithmic intensity signal provided by an embodiment of the present invention;

[0043] Figure 10 The method provided by the embodiment of the present invention is shown. DETAILED DESCRIPTION

[0044] Exemplary embodiments of methods, devices, and program products for sensing events are described in conjunction with the accompanying drawings. Although the description provides detailed examples of possible implementations, it should be noted that these details are intended to be exemplary and do not limit the scope of the present application.

[0045] In addition, embodiments / examples may refer to other embodiments / examples. For example, any description including but not limited to the terms, elements, processes, explanations and / or technical advantages mentioned in one embodiment / example is applicable to other embodiments / examples.

[0046] In order to introduce the present invention, the working principle of the pixel sensor is first described here. Figure 1 FIG. 1 shows a block diagram of a pixel sensor 100 provided by an embodiment of the present invention. Figure 1 As shown in the block diagram of FIG. 1 , the photoreceptor 101 can capture the incident light and provide the logarithm of the incident light to the next circuit (i.e. Figure 1The difference between the current value of the logarithmic intensity and the previous value 1021 can be computed by the differentiator 102, as illustrated. When the difference is greater than a positive threshold Th 1031 (also called sensitivity), the comparator 103 can generate a "+1" event, as Figure 1 Similarly, if the difference is less than a negative threshold -Th, a "-1" event can be generated.

[0047] The stream of events can comprise +1 or -1, indicating respectively a case of increase or decrease of the incident light intensity, the pixel coordinates and a timestamp of each event. Some event sensors can also capture intensity values, which can also be available at the output of the camera module.

[0048] Figure 2 An example of simulated events is illustrated, which are generated from a noiseless logarithmic intensity signal by the comparator 103 of the pixel sensor 100, for example. Figure 1 Specifically, one event stream is generated for a Th value of 1.75 and another event stream is generated for a Th value of 3.5. It can be observed that the more events are generated when Th 1031 is small, while the number of generated events decreases for larger Th 1031.

[0049] The events generated by each pixel 100 can be considered as the sign of the time derivative of the incident logarithmic intensity. The threshold Th 1031 can be considered as a quantization step of an analog-to-digital converter, although the output of the pixel 100 is not a digital value of the intensity.

[0050] It is emphasized here that an event sensor can not output the light intensity (except for the particular case of the DAVIS sensor), but only some estimate of the derivative (of the event).

[0051] In some applications, it can be of interest to also have an estimate of the intensity component (for example for high-dynamic-range, HDR). Therefore, in known solutions, methods have been proposed to obtain the scene intensity from pure events and / or from combined events and intensity values.

[0052] A classical algorithm is proposed to compute the logarithmic intensity of each pixel using only events. It can be described as follows:

[0053] At each pixel (for example, at the pixel sensor 100) illustrated: Figure 1

[0054] • Initialization: L0 = 0 (i.e. estimate of the logarithmic intensity), t0 = 0 (i.e. current timestamp), a0 = 0 (i.e. algorithm parameter)

[0055] • For each new event, perform:

[0056] ​o At = t - t0, where t is the timestamp of the new event

[0057] o L0= exp(-a0- At) - L0+ s - Th, where Th 1031 is the sensitivity threshold, s is the event (i.e. -1 or 1)

[0058] o t0= t

[0059] It is noted that this algorithm is just an example that can be used to compute the log intensity. The present application is not limited to any particular algorithm.

[0060] Figure 3 Examples of reconstructed log intensity signals are shown in Fig. 5. In particular, Figure 3 The two log intensity signals shown are reconstructed from Figure 2 events generated using a smaller sensitivity threshold. It is noted that using a smaller sensitivity threshold not only generates more events, but also allows a better reconstruction of the log intensity. This is actually to be expected, since in the case of digital-to-analog conversion, a finer quantization step also provides a smoother output signal (which is somewhat similar to the task at hand). However, this observation only holds for clean log intensity signals.

[0061] To illustrate the problem of existing approaches, Gaussian noise is added to the input log intensity. Events are now generated from the noisy signal. Figure 4 This case of a noisy log intensity signal is illustrated, showing the generated events and the reconstructed log intensity for two values of the sensitivity threshold.

[0062] It is noted that while for the noiseless case (as Figure 3 illustrated) a small sensitivity threshold can be the best choice; for the noisy case, the choice of the sensitivity threshold is not so easy.

[0063] For example, using a larger threshold (e.g. Th = 3.5, as Figure 4 illustrated) seems to provide a smoother reconstructed signal, while a smaller threshold (e.g. Th = 1.75) is more affected by the noise. It is noted here that the noise can have several different sources, such as additive noise due to additional electronics (e.g. the differentiator 102, the comparator 103, etc. as Figure 1 illustrated) and photon shot noise.

[0064] Using a single sensitivity threshold trades off between the number of time samples (time instances for which intensity frames are available) and the noise level. Specifically, by reducing the sensitivity threshold, the number of time samples (events) can be increased, but this also increases the noise (variance of the reconstructed log intensity); on the other hand, by increasing the sensitivity threshold, the impact of noise is reduced, but fewer samples are available for intensity reconstruction.

[0065] As shown in Figure 4 Experiments show that perhaps an adaptive threshold should be used to generate events in an event-based sensor. In this case, the threshold would have a variable value that would adapt to the noise characteristics.

[0066] However, updating the threshold takes time, and since events are generated at a relatively fast rate, events generated during the threshold update time interval can not provide improved reconstruction results. Thus, the reconstructed log intensity can have time periods of low quality (e.g., due to noise and / or lack of samples) in the quality of the reconstruction.

[0067] Embodiments of the present invention propose a method to obtain high quality (high signal-to-noise ratio) reconstructed log intensity components from events only. Thus, the present invention is able to reconstruct high quality intensity frames from an event sensor using multiple different sensitivity thresholds. Here, high quality means low noise and more time samples.

[0068] Figure 5An embodiment of the application provides an event sensing device 10 is shown. The event sensing device 10 can comprise a processing circuitry (not shown) for performing, implementing or initiating the various operations of the event sensing device 10 described herein. The processing circuitry can comprise hardware and software. The hardware can comprise analog circuitry or digital circuitry, or both analog circuitry and digital circuitry. The digital circuitry can comprise components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors, etc. The event sensing device 10 can further comprise a storage circuitry storing one or more instructions that can be executed by the processor or the processing circuitry, in particular under control of software. For example, the storage circuitry can comprise a non-transitory storage medium storing executable software code that, when executed by the processor or the processing circuitry, causes the various operations of the event sensing device 10 to be performed. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory can carry executable program code that, when executed by the one or more processors, causes the event sensing device 10 to perform, implement or initiate the operations or methods described herein.

[0069] In particular, the event sensing device 10 comprises one or more pixel sensors 100, 100'. In particular embodiments, the event sensing device 10 can comprise a plurality of pixel sensors 100, 100'. It is possible that the event sensing device 10 can comprise a pixel array and the plurality of pixel sensors 100, 100' can be pixels of the pixel array. According to embodiments of the application, one or more threshold values are associated with each of the one or more pixel sensors 100, 100'. That is, an event is to be generated based on any one of these one or more threshold values.

[0070] Each pixel sensor 100, 100' is configured to detect a time-dependent change in intensity of incident light at the pixel sensor 100, 100'. Optionally, each pixel sensor 100 can comprise a respective detection unit 102, 102' for detecting the time-dependent change. The detection unit 102, 102' can each be implemented as a Figure 1The time-dependent change in intensity can comprise a change in intensity from one point in time to a consecutive point in time. Each pixel sensor 100, 100' is further configured to generate an event if the time-dependent change in intensity exceeds any of one or more threshold values associated with the pixel sensor 100, 100'. In particular, each event can be associated with a time stamp and a threshold value. Optionally, each pixel sensor 100, 100' can further comprise a respective event generation unit 103, 103' configured to generate an event based on a particular threshold value at a particular time when the intensity of the incident light changes. The event generation unit 103, 103' can be a comparator 103 as shown in Figure 1

[0071] In addition, each pixel sensor 100, 100' is configured to generate one or more event streams, each event stream comprising a plurality of events associated with the same threshold value. For example, an event event_Thl is generated each time a change in intensity exceeds the threshold value Thl. The event stream generated with Thl comprises all events event_Thl generated within a certain time period.

[0072] According to an embodiment of the present application, the event sensing device 10 is further configured to reconstruct a plurality of independent intensity signals 11, 11' from the plurality of event streams generated by the plurality of pixel sensors 100, 100' based on the plurality of threshold values.

[0073] Optionally, each pixel sensor 100, 100' comprises one or more pairs of a respective differentiator 102, 102' configured to calculate a time-dependent change in intensity of the incident light and a respective comparator 103, 103' configured to generate an event if the time-dependent change in intensity exceeds a predetermined threshold value and configured to generate an event stream.

[0074] It is noted that when each of two different pixel sensors 100 and 100' comprises only one pair of differentiator 102, 102' and comparator 103, 103', different threshold values 1031, 1031' can be set for the two pixel sensors 100 and 100'.

[0075] One idea of the present application is to generate several event streams for each pixel, each event stream generated with a different (sensitivity) threshold value. For each event stream, an independent logarithmic intensity signal can be reconstructed, which can then be fused together in order to generate a final logarithmic intensity signal for that particular pixel.

[0076] It is noted that the use of several threshold values can of course be combined with an adaptive threshold method, such that the value of one or more threshold values can be adapted.​

[0077] In particular, two embodiments are proposed to implement the multi-sensitivity of the present application, as described below.

[0078] Optionally, the first embodiment proposes an event sensing device 10 comprising one or more pixel sensors 100, 100', wherein each pixel sensor 100, 100' comprises a plurality of differentiators 102, 102' and comparators 103, 103'. In a particular example of the first embodiment, as shown in Figure 6 there are three pairs of differentiators (102, 104, 106) and comparators (103, 105, 107) in each pixel sensor 100.

[0079] As shown in Figure 6 for each pixel, each differentiator-comparator pair will generate its own event stream, which is independent of the other event streams. Thus, each pixel can generate several independent event streams. In the present example, comparator 103 compares the result from differentiator 102 with threshold Thi 1031 and generates an output event stream 1 accordingly. Similarly, output event stream 2 and output event stream 3 are generated by comparator 105 and comparator 107, respectively. It is noted that each comparator (103, 105, 107) can be provided with a respective predetermined threshold and used to generate an event stream based on the respective predetermined threshold. As shown in the example Figure 6 thresholds Thi 1031, Th2 1051 and Th3 1071 can be provided for comparator 103, comparator 105 and comparator 107, respectively. It is also noted that the predetermined thresholds (e.g. thresholds Thi 1031, Th2 1051 and Th3 1071) can be different from each other. That is, thresholds Thi 1031, Th2 1051 and Th3 1071 can have different values.

[0080] It is noted that it can not be possible to use one common differentiator and several comparators. This can be because each differentiator (102, 104, 106) calculates the change in logarithmic intensity since the time of the last event. Since each sensitivity threshold generates an independent event stream, it is preferred to have as many differentiators (102, 104, 106) as comparators (103, 105, 107).

[0081] When reconstructing the logarithmic intensity, one version of the logarithmic intensity can be calculated from each event stream (for each pixel). Optionally, the event sensing device 10 can also be used to reconstruct an independent intensity signal (108, 109, 1010) for each comparator (103, 105, 107) from each event stream generated by the comparators (103, 105, 107). In Figure 6In the example shown, an independent intensity signal (108, 109, 1010) is reconstructed from each of the output event stream 1, the output event stream 2 and the output event stream 3.

[0082] Then, a final log intensity signal can be obtained by fusing the signals reconstructed from different event streams of the same pixel. Moreover, the event sensing device 10 can be used to generate a final intensity signal 11 for each pixel sensor 100 by fusing the reconstructed independent intensity signals (108, 109, 1010) of at least two comparators (103, 105, 107) of the pixel sensor 100. In Figure 6 In the example shown, the final log intensity signal 11 can be computed by fusing the three independent intensity signals (108, 109, 1010) reconstructed from the output event stream 1, the output event stream 2 and the output event stream 3.

[0083] The method that can be used to reconstruct the estimated log intensity component from independent event streams is not limited in the present application. Moreover, the present application does not limit the fusion method either. Any suitable conventional method can be used.

[0084] Moreover, the event sensing device 10 can be used to reconstruct an intensity image from the plurality of final intensity signals 11, 11' of one or more pixel sensors 100, 100'.

[0085] The second embodiment of the present application enables the implementation of the solution in existing event sensors. Figure 7 An implementation of the second embodiment is shown. It is noted that while the previous implementation of the first embodiment proposes a new hardware structure (i.e. each pixel sensor 100, 100' comprises a plurality of differentiators and comparators), the second embodiment can be implemented with existing event sensors.

[0086] According to this second embodiment, the plurality of pixel sensors 100, 100' can be grouped into a plurality of super-pixels 110. Each super-pixel 110 can comprise at least two pixel sensors 100, 100'. In particular, each of the at least two pixel sensors 100, 100' can be provided with a respective predetermined threshold, wherein the predetermined thresholds can be different from each other.

[0087] As Figure 7 In the example shown, the pixel array is grouped into a plurality of super-pixels (or can be named multi-pixels) 110. Each super-pixel 110 can comprise four adjacent pixel sensors 100, 100'. Moreover, each of these adjacent pixel sensors 100, 100' can have a different sensitivity threshold, wherein the thresholds are denoted as Thl, Th2, Th3 and Th4 in Figure 7 In the example shown, the pixel array is grouped into a plurality of super-pixels (or can be named multi-pixels) 110. Each super-pixel 110 can comprise four adjacent pixel sensors 100, 100'. Moreover, each of these adjacent pixel sensors 100, 100' can have a different sensitivity threshold, wherein the thresholds are denoted as Thl, Th2, Th3 and Th4 in

[0088] Figure 8 A super-pixel 110 is shown Figure 7 The super-pixel 110 according to the shown embodiment provides a block diagram of the super-pixel 110. Specifically, the super-pixel 110 comprises four neighboring pixel sensors 100-1, 100-2, 100-3 and 100-4.

[0089] As Figure 8 shown, each pixel sensor 100-1, 100-2, 100-3 and 100-4 can comprise only one differentiator-comparator pair. As Figure 8 shown in the shown example, threshold values Th1 1031-1, Th2 1031-2, Th3 1031-3 and Th4 1031-4 can be set for the comparators 103-1, 103-2, 103-3 and 103-4, respectively. It is further noted that the predetermined threshold values can differ from each other. That is, the threshold values Th1 1031-1, Th2 1031-2, Th3 1031-3 and Th4 1031-4 can have different values.

[0090] It is noted that based on the respective predetermined threshold value (e.g. Figure 8 Th1 1031-1, Th2 1031-2, Th3 1031-3 or Th4 1031-4) shown, each of the pixel sensors 100-1, 100-2, 100-3 and 100-4 can generate one event stream.

[0091] Optionally, according to one embodiment of the present application, the event sensing device 10 can further be configured to reconstruct the individual intensity signals 11-1, 11-2, 11-3, 11-4 of each of the pixel sensors 100-1, 100-2, 100-3 and 100-4 from the event streams generated by the pixel sensors 100-1, 100-2, 100-3 and 100-4.

[0092] The event sensing device 10 can further be configured to generate a final intensity signal 12 for each super-pixel 110 by fusing the reconstructed individual intensity signals 11-1, 11-2, 11-3, 11-4 of at least two of the pixel sensors 100-1, 100-2, 100-3 and 100-4 in the super-pixel 110. In Figure 7 or Figure 8 the shown implementation, the four estimates corresponding to the four event streams of the super-pixel 110 are then combined to obtain a single log-intensity estimate assigned to the super-pixel 110.

[0093] It is noted that the number of pixel sensors 100, 100' in a super-pixel 110 does not necessarily have to be four. In principle, any number greater than one can be applied to the second embodiment of the present application.

[0094] It can also be seen that in this implementation, the four event streams belonging to the same super-pixel 110 can not actually collect light from exactly the same point of the scene. Therefore, combining the event streams from these four sensors 100-1, 100-2, 100-3 and 100-4 to reconstruct one log-intensity component can introduce some artifacts. However, if any, these artifacts are expected to be only visible for very small objects.

[0095] Furthermore, it is therefore preferred to have super-pixels 110 comprising pixel sensors 100, 100' next to each other. As Figure 7 or Figure 8 As an example, shown in Fig. 2, each super-pixel 110 is preferably designed to comprise a 2x2 array of pixel sensors 100-1, 100-2, 100-3 and 100-4.

[0096] Similar to the previous embodiment, the event sensing device 10 can also be used to reconstruct an intensity image from the multiple final intensity signals 12 of the multiple super-pixels 110.

[0097] To implement the second embodiment, an existing event sensor can be used, as no hardware (of the pixel sensors) needs to be changed. It can be considered as a software implementation of generating an event stream with multiple thresholds for one "pixel" (i.e. super-pixel).

[0098] The embodiments of the present invention, in particular the first and second embodiments as described before, enable to obtain more temporal samples of the reconstructed intensity frame while at the same time a low noise level can be maintained. The increased number of temporal samples can be important in the synchronization with other cameras and also to be able to perform de-noising (in the time domain) of the reconstructed log-intensity. The low noise in the frame is also important in certain applications such as HDR.

[0099] To illustrate how the approach presented in the present invention works, Figure 9 Simulation results of the reconstructed log-intensity signals are described in the Figure 9 Input noise log-intensity, four reconstructed log-intensity signals from the four event streams and the fusion result of the four reconstructed log-intensity signals are shown.

[0100] It is noted that for each pixel sensor 100, 100' four sensitivity thresholds (e.g. 2.75, 3, 3.25 and 3.5 as shown) are set. It is to be understood that each pixel sensor 100, 100' can comprise four pairs of differentiator and comparator, each comparator being set with one of the thresholds (i.e. 2.75, 3, 3.25 or 3.5) and will generate an event stream based on this threshold. Figure 9

[0101] ​In this simulation, four reconstructed log intensity signals (e.g., reconstructed th=2.75, reconstructed th=3, reconstructed th=3.25, and reconstructed th=3.5, as shown in Figure 8 It is noted that the present application is not limited to this algorithm; any other algorithm or method can be used as well.

[0102] For the fusion result of the four reconstructed log intensity signals, if the four reconstructed log intensity signals are collected at exactly the same time stamp, a simple method of calculating the sample average of the four reconstructed log intensity signals can be used. Otherwise, a low-pass filtered version can be used. It is worth mentioning that any other fusion method can be used as well. In addition, a denoising method can be used to smooth the fused log intensity as well.

[0103] As can be seen from Figure 9 For smaller sensitivity thresholds (e.g., 2.75 or 3), the reconstructed signals are noisier and have more samples compared to the signals obtained from the event stream with larger thresholds. It is noted that the result (fusion result) of the scheme proposed in this embodiment has more samples and is also smoother compared to the other four reconstructed independent signals.

[0104] It is noted that Figure 9 The simulation results shown are based on the first embodiment of the present application described previously, i.e., each pixel sensor 100, 100’ of the event sensing device 10 comprises four pairs of differentiators and comparators. It is worth mentioning that similar simulation results can be obtained for the second embodiment, i.e., different threshold values are set for the super-pixel 110. However, the second embodiment can result in a loss of resolution in the reconstructed intensity image.

[0105] Figure 10 A method 1000 for sensing events provided by an embodiment of the present application is shown. In a particular embodiment of the present application, the method 1000 is performed by an event sensing device 10 as shown in Figure 5 , Figure 6 or Figure 8The illustrated event sensing device 10 performs. In particular, the event sensing device 10 comprises one or more pixel sensors 100, 100’ with one or more thresholds associated with each of the plurality of pixel sensors 100, 100’. The method 1000 comprises a step 1001 of detecting a time-dependent change in intensity of incident light at the pixel sensor, a step 1002 of generating an event if the time-dependent change in intensity exceeds any of the one or more thresholds associated with the pixel sensor 100, 100’, wherein each event is associated with a time stamp and a threshold, and a step 1003 of generating one or more event streams, each event stream comprising a plurality of events associated with the same threshold. The method further comprises a step 1004 of reconstructing a plurality of independent intensity signals 11, 11’ from the plurality of event streams generated by the one or more pixel sensors 100, 100’ based on the plurality of thresholds. It is possible that each of the one or more pixel sensors 100, 100’ in the event sensing device 10 is a pixel sensor as illustrated, Figure 6 a pixel sensor as illustrated, or Figure 8 a pixel sensor as illustrated.

[0106] The application has been described in connection with various embodiments and implementations as examples. However, those skilled in the art will understand and appreciate that other variations are possible in light of the teachings and examples given herein and that the scope of the claimed application is not limited to what has been particularly shown and described herein above. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit can fulfill the functions of several entities or items recited in the claims. Reference to an item or a component can refer to one or more items or components. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

[0107] Furthermore, any of the methods according to the embodiments of the application can be implemented in a computer program for a processing means, which when carried out by the processing means causes the processing means to execute the method steps. The computer program is included in a computer program product comprising a computer-readable medium, which can be essentially any memory, for example a ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash memory, an electrically erasable PROM (EEPROM), or a hard disk. The computer program product can also comprise a computer-readable medium, which can be essentially any memory, for example a ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash memory, an electrically erasable PROM (EEPROM), or a hard disk.

[0108] Furthermore, it is recognized that embodiments of the event sensing device 10 include the necessary communication capabilities in the form of, for example, functions, modules, units, elements, etc., for performing the schemes. Examples of other such modules, units, elements and functions are: processors, memories, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, de-interleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiving units, transmitting units, DSPs, trellis-coded modulation (TCM) encoders, TCM decoders, power supply units, power feeders, communication interfaces, communication protocols, etc., which are suitably arranged together to perform the technical schemes.

[0109] In particular, the one or more processors of the event sensing device 10 can comprise one or more instances of, e.g., a central processing unit (CPU), a processing unit, a processing circuit, a processor, an application-specific integrated circuit (ASIC), a microprocessor, or other processing logic that can interpret and execute instructions. The expression “processor” can thus represent a processing circuit comprising a plurality of processing circuits, such as, e.g., any, some or all of the ones enumerated above. The processing circuitry can further perform data processing functions for inputting, outputting, and processing of data comprising data buffering and device control functions, such as call processing control, user interface control, or the like.

Claims

1. An event sensing device (10), characterized by The event sensing device (10) comprises a plurality of pixel sensors (100, 100'), wherein a plurality of thresholds is associated with each of the plurality of pixel sensors (100, 100'), and each pixel sensor (100, 100') is configured to: detect a time-dependent change of an intensity of an incident light at the pixel sensor (100, 100'); generate an event if the time-dependent change of the intensity exceeds any of the plurality of thresholds associated with the pixel sensor (100, 100'), wherein each event is associated with a time stamp and a threshold value; generate a plurality of event streams, each event stream comprising a plurality of the events associated with the same threshold value; wherein the event sensing device (10) is further configured to: reconstruct a plurality of independent intensity signals (11, 11') from the plurality of event streams generated by the plurality of pixel sensors (100, 100') based on the plurality of thresholds; wherein each pixel sensor (100) comprises at least two pairs of a differentiator (102, 104, 106) and a comparator (103, 105, 107), wherein one pair of a differentiator and a comparator comprises one differentiator and one comparator, each comparator (103, 105, 107) is provided with a respective predetermined threshold value (1031, 1051, 1071) and is configured to generate an event stream based on the respective predetermined threshold value (1031, 1051, 1071), wherein the predetermined threshold values (1031, 1051, 1071) are different from each other, wherein each of the predetermined threshold values (1031, 1051, 1071) is one of the plurality of thresholds; wherein each event stream generated from the comparators (103, 105, 107) reconstructs an independent intensity signal (108, 109, 1010) of each comparator (103, 105, 107); by fusing the reconstructed independent intensity signals (108, 109, 1010) of at least two comparators (103, 105, 107) of the pixel sensor (100), a final intensity signal (11) is generated for each pixel sensor (100).

2. The event sensing device (10) according to claim 1, wherein the time-dependent change of the intensity comprises a change of the intensity from one point in time to a consecutive point in time.

3. The event sensing device (10) according to claim 1 or 2, wherein each differentiator (102, 102') is configured to calculate the time-dependent change of the intensity of the incident light, and each comparator (103, 103') is configured to generate an event if the time-dependent change of the intensity exceeds a predetermined threshold value (1031, 1031'), and to generate an event stream.

4. The event sensing device (10) according to claim 3, characterized in that for: reconstructing an intensity image from the plurality of final intensity signals (11, 11') of the plurality of pixel sensors (100, 100').

5. The event sensing device (10) according to claim 3, wherein The plurality of pixel sensors (100, 100') are grouped into a plurality of super-pixels (110), wherein each super-pixel (110) comprises at least two pixel sensors (100, 100') and each pixel sensor of the at least two pixel sensors (100, 100') of the same super-pixel is provided with a respective predetermined threshold value, wherein the predetermined threshold values differ from each other.

6. The event sensing device (10) according to claim 5, characterized in that for: reconstructing an individual intensity signal (11-1, 11-2, 11-3, 11-4) of each pixel sensor (100-1, 100-2, 100-3, 100-4) from the event stream generated by the pixel sensor (100, 100-2, 100-3, 100-4); generating a final intensity signal (12) for each super-pixel (110) by fusing the reconstructed individual intensity signals (11-1, 11-2, 11-3, 11-4) of the at least two pixel sensors (100, 100-2, 100-3, 100-4) in the super-pixel (110).

7. The event sensing device (10) according to claim 6, characterized in that for: reconstructing an intensity image from the plurality of final intensity signals (12) of the plurality of super-pixels (110).

8. The event sensing device (10) according to any one of claims 5 to 7, characterized in that The at least two pixel sensors (100, 100') are adjacent.

9. The event sensing device (10) according to any one of claims 5 to 7, characterized in that Each super-pixel (110) comprises four pixel sensors (100-1, 100-2, 100-3, 100-4).

10. The event sensing device (10) according to claim 1 or 2, characterized in that The time-dependent change in intensity is a time-dependent change in the logarithm of intensity.

11. A method (1000) for sensing an event using an event sensing device, characterized by, The event sensing device comprises a plurality of pixel sensors (100, 100'), wherein a plurality of threshold values is associated with each pixel sensor of the plurality of pixel sensors (100, 100'), the method comprising: detecting (1001) a time-dependent change in intensity of incident light at the pixel sensor (100, 100'); generating (1002) an event if the time-dependent change in intensity exceeds any one of the plurality of threshold values associated with the pixel sensor (100, 100'), wherein each event is associated with a time stamp and a threshold value; generating (1003) a plurality of event streams, each event stream comprising a plurality of the events associated with the same threshold value; wherein the method (1000) further comprises: reconstructing (1004) a plurality of individual intensity signals (11, 11') from the plurality of event streams generated by the plurality of pixel sensors (100, 100') based on a plurality of threshold values; wherein each pixel sensor (100) comprises at least two pairs of differentiator (102, 104, 106) and comparator (103, 105, 107), wherein a pair of differentiator and comparator comprises one differentiator and one comparator, each comparator (103, 105, 107) is provided with a respective predetermined threshold value (1031, 1051, 1071), the method (1000) further comprises generating an event stream based on the respective predetermined threshold value (1031, 1051, 1071), wherein the predetermined threshold values (1031, 1051, 1071) are different from each other, wherein each of the predetermined threshold values (1031, 1051, 1071) is one of the plurality of threshold values; wherein the method (1000) further comprises reconstructing an individual intensity signal (108, 109, 1010) for each comparator (103, 105, 107) from each event stream generated by the comparator (103, 105, 107); generating a final intensity signal (11) for each pixel sensor (100) by fusing the reconstructed individual intensity signals (108, 109, 1010) of at least two comparators (103, 105, 107) of the pixel sensor (100).

12. A computer program product, characterised in that, The computer program product comprises program code for performing the method according to claim 11 when implemented on a processor. The computer program product comprises program code for performing the method according to claim 11 when implemented on a processor.

Images