Sensor device and method for operating a sensor device

Abstract

Sensor for event detection comprising, a control unit (110) that determines an event generation rate that indicates how many events have occurred during a predetermined time interval, and a timestamp unit (120) that receives from an event detection unit (100) requests for permission for outputting event data and adds to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed. The control unit instructs the event detection unit to asynchronously request permission for outputting event data from the timestamp unit, if the event generation rate is below or equal to a readout threshold, and the control unit instructs the event detection unit to output event data synchronously at predetermined time points and to add a timestamp that indicates the respective time point to the event data, if the event generation rate is above the readout threshold.

Description

[0001] SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE

[0002] FIELD OF THE INVENTION

[0003] The present technology relates to a sensor device, and a method for operating a sensor device, in particular, to a sensor device and a method for operating a sensor device that allows an improved generation of event data.

[0004] BACKGROUND

[0005] Presently, sensor data obtained in dynamic / event vision sensors, DVS / EVS, are used to obtain temporally highly resolved image streams that might be updated asynchronously. This works well as long as the number of events remains in a regime that allows processing of the event data without a delay that would jeopardize the advantages of the event-based vision sensors. Therefore, improved sensor devices and methods for operating these sensor devices are desirable that mitigate this problem.

[0006] SUMMARY OF INVENTION

[0007] To this end, a sensor device is provided that comprises a pixel array having a plurality of pixels each being configured to receive light and to perform photoelectric conversion to generate an electrical signal indicating the intensity of the received light, an event detection unit that is configured to receive the electrical signals from each of the plurality of pixels and to generate event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel above an event threshold, a control unit that is configured to determine an event generation rate that indicates how many events have occurred during a predetermined time interval, and a timestamp unit that is configured to receive from the event detection unit requests for permission for outputting of event data and to add to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed. Here, the control unit is configured to instruct the event detection unit to asynchronously request permission for outputting of event data from the timestamp unit, if the determined event generation rate is below or equal to a first readout threshold, and the control unit is configured to instruct the event detection unit to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time, if the determined event generation rate is above the first readout threshold..

[0008] Further, a method for operating such a sensor device is provided, the method comprising: at each of the plurality of pixels, receiving light to perform photoelectric conversion and generating an electrical signal indicating the intensity of the received light; at the event detection unit, receiving the electrical signals from each of the plurality of pixels and generating event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel above an event threshold; at the control unit, determining an event generation rate that indicates how many events have occurred during a predetermined time interval; at the timestamp unit, receiving from the event detection unit requests for permission for outputting of event data and adding to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed; by the control unit, instructing the event detection unit to asynchronously request permission for outputting of event data from the timestamp unit, if the determined event generation rate is below or equal to a first readout threshold; and by the control unit, instructing the event detection unit to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time, if the determined event generation rate is above the first readout threshold

[0009] In the above, the usual concept of an event-based vision sensor to process occurring events asynchronously at the time of occurrence is further refined by switching to a synchronous processing, once the event generation rate crosses a certain threshold. In asynchronous processing, readout permission has to be requested for each event. If the event generation rate is low, readout permission is granted basically instantaneously (in comparison to the time resolution of the event timestamp). However, if the event generation rate becomes too high, readout requests pile-up before they can be processed. Thus, a time delay is generated between the actual occurrence of the event and the time at which a timestamp is allocated to the event. This time delay depends on the number of pending readout requests and is therefore variable. Accordingly, for high event generation rates jitter for the time information provided by the event timestamps increases, which means that this time information becomes less reliable. This jitter is avoided by switching from asynchronous, requested readout to a synchronous readout that assigns the same timestamp to all events that occurred since the last readout. Although also in this manner information on the temporal order of event occurrences is lost, the loss is known and can be dealt with when processing the event data. In this manner processing of the event data becomes more reliable, since the information on the time of event occurrence becomes more reliable.

[0010] BRIEF DESCRIPTION OF DRAWINGS

[0011] Fig. 1 is a schematic diagram of a sensor device.

[0012] Fig. 2 is a schematic block diagram of a sensor section.

[0013] Fig. 3 is a schematic block diagram of a pixel array section.

[0014] Fig. 4 is a schematic circuit diagram of a pixel block.

[0015] Fig. 5 is a schematic block diagram illustrating of an event detecting section.

[0016] Fig. 6 is a schematic circuit diagram of a current-voltage converting section.

[0017] Fig. 7 is a schematic circuit diagram of a subtraction section and a quantization section.

[0018] Fig. 8 is a schematic diagram of a frame data generation method based on event data. Fig. 9 is a schematic block diagram of another quantization section.

[0019] Fig. 10 is a schematic diagram of another event detecting section.

[0020] Fig. 11 is a schematic block diagram of another pixel array section.

[0021] Fig. 12 is a schematic circuit diagram of another pixel block.

[0022] Fig. 13 is a schematic block diagram of a scan-type sensor device.

[0023] Fig. 14 is a schematic block diagram of a sensor device.

[0024] Fig. 15 is a schematic illustration of asynchronous allocation of time information.

[0025] Figs. 16A to 16D are schematic illustrations of processing of asynchronous readout requests.

[0026] Fig. 17 is a schematic illustration of different timestamp processing regimes.

[0027] Figs. 18A to 18C are further schematic illustrations of different timestamp processing regimes.

[0028] Figs. 19A to 19D are schematic illustration of processing of readout requests.

[0029] Fig. 20 is a schematic illustration of timestamp allocation processing.

[0030] Fig. 21 is a schematic illustration of a calibration process.

[0031] Fig. 22 is a schematic process flow of a method for operation a sensor device.

[0032] Fig. 23 is a schematic block diagram of a vehicle control system.

[0033] Fig. 24 is a diagram of assistance in explaining an example of installation positions of an outside-vehicle information detecting section and an imaging section.

[0034] Fig. 25A and 25B are schematic illustrations of a mobile device and a head mounted display comprising a sensor device.

[0035] DETAILED DESCRIPTION

[0036] The present disclosure is directed to mitigating problems related to the generation and processing of the data of event based / dynamic vision sensors, EVS / DVS. The present disclosure is based on the operation of conventional EVS / DVS. Thus, at first a possible implementation of an EVS / DVS will be described. This is of course purely exemplary. It is to be understood that EVSs / DVSs could also be implemented differently.

[0037] Fig. 1 is a diagram illustrating a configuration example of a sensor device 10, which is in the example of Fig. 1 constituted by a sensor chip.

[0038] The sensor device 10 is a single-chip semiconductor chip and includes a sensor die (substrate) 11, which serves as a plurality of dies (substrates), and a logic die 12 that are stacked. Note that, the sensor device 10 can also include only a single die or three or more stacked dies.

[0039] In the sensor device 10 of Fig. 1, the sensor die 11 includes (a circuit serving as) a sensor section 21, and the logic die 12 includes a logic section 22. Note that, the sensor section 21 can be partly formed on the logic die 12. Further, the logic section 22 can be partly formed on the sensor die 11.

[0040] The sensor section 21 includes pixels configured to perform photoelectric conversion on incident light to generate electrical signals, and generates event data indicating the occurrence of events that are changes in the electrical signal of the pixels. The sensor section 21 supplies the event data to the logic section 22. That is, the sensor section 21 performs imaging of performing, in the pixels, photoelectric conversion on incident light to generate electrical signals, similarly to a synchronous image sensor, for example. The sensor section 21, however, generates event data indicating the occurrence of events that are changes in the electrical signal of the pixels instead of generating image data in a frame format (frame data). The sensor section 21 outputs, to the logic section 22, the event data obtained by the imaging.

[0041] Here, the synchronous image sensor is an image sensor configured to perform imaging in synchronization with a vertical synchronization signal and output frame data that is image data in a frame format. The sensor section 21 can be regarded as asynchronous (an asynchronous image sensor) in contrast to the synchronous image sensor, since the sensor section 21 does not operate in synchronization with a vertical synchronization signal when outputting event data. In particular, the sensor section 21 can output event data with a temporal precision of 10'6s.

[0042] Note that, the sensor section 21 may generate and output, other than event data, frame data, similarly to the synchronous image sensor. In addition, the sensor section 21 can output, together with event data, electrical signals of pixels in which events have occurred, as pixel signals that are pixel values of the pixels in frame data.

[0043] The logic section 22 controls the sensor section 21 as needed. Further, the logic section 22 performs various types of data processing, such as data processing of generating frame data on the basis of event data from the sensor section 21 and image processing on frame data from the sensor section 21 or frame data generated on the basis of the event data from the sensor section 21, and outputs data processing results obtained by performing the various types of data processing on the event data and the frame data. The logic section 22 may implement the functions of a control unit as described below.

[0044] Fig. 2 is a block diagram illustrating a configuration example of the sensor section 21 of Fig. 1. The sensor section 21 includes a pixel array section 31, a driving section 32, an arbiter 33, an AD (Analog to Digital) conversion section 34, and an output section 35.

[0045] The pixel array section 31 includes a plurality of pixels 51 (Fig. 3) arrayed in a two-dimensional lattice pattern. The pixel array section 31 detects, in a case where a change larger than a predetermined threshold (including a change equal to or larger than the threshold as needed) has occurred in (a voltage corresponding to) a photocurrent that is an electrical signal generated by photoelectric conversion in the pixel 51, the change in the photocurrent as an event. In a case of detecting an event, the pixel array section 31 outputs, to the arbiter 33, a request for requesting the output of event data indicating the occurrence of the event. Then, in a case of receiving a response indicating event data output permission from the arbiter 33, the pixel array section 31 outputs the event data to the driving section 32 and the output section 35. In addition, the pixel array section 31 may output an electrical signal of the pixel 51 in which the event has been detected to the AD conversion section 34.

[0046] The driving section 32 supplies control signals to the pixel array section 31 to drive the pixel array section 31. For example, the driving section 32 drives the pixel 51 regarding which the pixel array section 31 has output event data, so that the pixel 51 in question supplies (outputs) a pixel signal to the AD conversion section 34.

[0047] The arbiter 33 arbitrates the requests for requesting the output of event data from the pixel array section 31, and returns responses indicating event data output permission or prohibition to the pixel array section 31.

[0048] The AD conversion section 34 includes, for example, a single-slope ADC (AD converter) (not illustrated) in each column of pixel blocks 41 (Fig. 3) described later, for example. The AD conversion section 34 performs, with the ADC in each column, AD conversion on pixel signals of the pixels 51 of the pixel blocks 41 in the column, and supplies the resultant to the output section 35. Note that, the AD conversion section 34 can perform CDS (Correlated Double Sampling) together with pixel signal AD conversion.

[0049] The output section 35 performs necessary processing on the pixel signals from the AD conversion section 34 and the event data from the pixel array section 31 and supplies the resultant to the logic section 22 (Fig. 1).

[0050] Here, a change in the photocurrent generated in the pixel 51 can be recognized as a change in the amount of light entering the pixel 51, so that it can also be said that an event is a change in light amount (a change in light amount larger than the threshold) in the pixel 51.

[0051] Event data indicating the occurrence of an event at least includes location information (coordinates or the like) indicating the location of a pixel block in which a change in light amount, which is the event, has occurred. Besides, the event data can also include the polarity (positive or negative) of the change in light amount. With regard to the series of event data that is output from the pixel array section 31 at timings at which events have occurred, it can be said that, as long as the event data interval is the same as the event occurrence interval, the event data implicitly includes time point information indicating (relative) time points at which the events have occurred. However, for example, when the event data is stored in a memory and the event data interval is no longer the same as the event occurrence interval, the time point information implicitly included in the event data is lost. Thus, the output section 35 includes, in event data, time point information indicating (relative) time points at which events have occurred, such as timestamps, before the event data interval is changed from the event occurrence interval. The processing of including time point information in event data can be performed in any block other than the output section 35 as long as the processing is performed before time point information implicitly included in event data is lost.

[0052] Fig. 3 is a block diagram illustrating a configuration example of the pixel array section 31 of Fig. 2.

[0053] The pixel array section 31 includes the plurality of pixel blocks 41. The pixel block 41 includes the IxJ pixels 51 that are one or more pixels arrayed in I rows and J columns (I and J are integers), an event detecting section 52, and a pixel signal generating section 53. The one or more pixels 51 in the pixel block 41 share the event detecting section 52 and the pixel signal generating section 53. Further, in each column of the pixel blocks 41, a VSL (Vertical Signal Line) for connecting the pixel blocks 41 to the ADC of the AD conversion section 34 is wired.

[0054] The pixel 51 receives light incident from an object and performs photoelectric conversion to generate a photocurrent serving as an electrical signal. The pixel 51 supplies the photocurrent to the event detecting section 52 under the control of the driving section 32.

[0055] The event detecting section 52 detects, as an event, a change larger than the predetermined threshold in photocurrent from each of the pixels 51, under the control of the driving section 32. In a case of detecting an event, the event detecting section 52 supplies, to the arbiter 33 (Fig. 2), a request for requesting the output of event data indicating the occurrence of the event. Then, when receiving a response indicating event data output permission to the request from the arbiter 33, the event detecting section 52 outputs the event data to the driving section 32 and the output section 35.

[0056] The pixel signal generating section 53 generates, in the case where the event detecting section 52 has detected an event, a voltage corresponding to a photocurrent from the pixel 51 as a pixel signal, and supplies the voltage to the AD conversion section 34 through the VSL, under the control of the driving section 32.

[0057] Here, detecting a change larger than the predetermined threshold in photocurrent as an event can also be recognized as detecting, as an event, absence of change larger than the predetermined threshold in photocurrent. The pixel signal generating section 53 can generate a pixel signal in the case where absence of change larger than the predetermined threshold in photocurrent has been detected as an event as well as in the case where a change larger than the predetermined threshold in photocurrent has been detected as an event.

[0058] Fig. 4 is a circuit diagram illustrating a configuration example of the pixel block 41.

[0059] The pixel block 41 includes, as described with reference to Fig. 3, the pixels 51, the event detecting section 52, and the pixel signal generating section 53.

[0060] The pixel 51 includes a photoelectric conversion element 61 and transfer transistors 62 and 63.

[0061] The photoelectric conversion element 61 includes, for example, a PD (Photodiode). The photoelectric conversion element 61 receives incident light and performs photoelectric conversion to generate charges.

[0062] The transfer transistor 62 includes, for example, an N (Negative)-type MOS (Metal-Oxide- Semiconductor) FET (Field Effect Transistor). The transfer transistor 62 of the n-th pixel 51 of the IxJ pixels 51 in the pixel block 41 is turned on or off in response to a control signal OFGn supplied from the driving section 32 (Fig. 2). When the transfer transistor 62 is turned on, charges generated in the photoelectric conversion element 61 are transferred (supplied) to the event detecting section 52, as a photocurrent.

[0063] The transfer transistor 63 includes, for example, an N-type MOSFET. The transfer transistor 63 of the n- th pixel 51 of the IxJ pixels 51 in the pixel block 41 is turned on or off in response to a control signal TRGn supplied from the driving section 32. When the transfer transistor 63 is turned on, charges generated in the photoelectric conversion element 61 are transferred to an FD 74 of the pixel signal generating section 53.

[0064] The IxJ pixels 51 in the pixel block 41 are connected to the event detecting section 52 of the pixel block 41 through nodes 60. Thus, photocurrents generated in (the photoelectric conversion elements 61 of) the pixels 51 are supplied to the event detecting section 52 through the nodes 60. As a result, the event detecting section 52 receives the sum of photocurrents from all the pixels 51 in the pixel block 41. Thus, the event detecting section 52 detects, as an event, a change in sum of photocurrents supplied from the IxJ pixels 51 in the pixel block 41.

[0065] The pixel signal generating section 53 includes a reset transistor 71, an amplification transistor 72, a selection transistor 73, and the FD (Floating Diffusion) 74.

[0066] The reset transistor 71, the amplification transistor 72, and the selection transistor 73 include, for example, N-type MOSFETs.

[0067] The reset transistor 71 is turned on or off in response to a control signal RST supplied from the driving section 32 (Fig. 2). When the reset transistor 71 is turned on, the FD 74 is connected to a power supply VDD, and charges accumulated in the FD 74 are thus discharged to the power supply VDD. With this, the FD 74 is reset.

[0068] The amplification transistor 72 has a gate connected to the FD 74, a drain connected to the power supply VDD, and a source connected to the VSL through the selection transistor 73. The amplification transistor 72 is a source follower and outputs a voltage (electrical signal) corresponding to the voltage of the FD 74 supplied to the gate to the VSL through the selection transistor 73.

[0069] The selection transistor 73 is turned on or off in response to a control signal SEL supplied from the driving section 32. When the selection transistor 73 is turned on, a voltage corresponding to the voltage of the FD 74 from the amplification transistor 72 is output to the VSL.

[0070] The FD 74 accumulates charges transferred from the photoelectric conversion elements 61 of the pixels 51 through the transfer transistors 63, and converts the charges to voltages.

[0071] With regard to the pixels 51 and the pixel signal generating section 53, which are configured as described above, the driving section 32 turns on the transfer transistors 62 with control signals OFGn, so that the transfer transistors 62 supply, to the event detecting section 52, photocurrents based on charges generated in the photoelectric conversion elements 61 of the pixels 51. With this, the event detecting section 52 receives a current that is the sum of the photocurrents from all the pixels 51 in the pixel block 41, which might also be only a single pixel.

[0072] When the event detecting section 52 detects, as an event, a change in photocurrent (sum of photocurrents) in the pixel block 41, the driving section 32 turns off the transfer transistors 62 of all the pixels 51 in the pixel block 41, to thereby stop the supply of the photocurrents to the event detecting section 52. Then, the driving section 32 sequentially turns on, with the control signals TRGn, the transfer transistors 63 of the pixels 51 in the pixel block 41 in which the event has been detected, so that the transfer transistors 63 transfers charges generated in the photoelectric conversion elements 61 to the FD 74. The FD 74 accumulates the charges transferred from (the photoelectric conversion elements 61 of) the pixels 51. Voltages corresponding to the charges accumulated in the FD 74 are output to the VSL, as pixel signals of the pixels 51, through the amplification transistor 72 and the selection transistor 73.

[0073] As described above, in the sensor section 21 (Fig. 2), only pixel signals of the pixels 51 in the pixel block 41 in which an event has been detected are sequentially output to the VSL. The pixel signals output to the VSL are supplied to the AD conversion section 34 to be subjected to AD conversion.

[0074] Here, in the pixels 51 in the pixel block 41, the transfer transistors 63 can be turned on not sequentially but simultaneously. In this case, the sum of pixel signals of all the pixels 51 in the pixel block 41 can be output.

[0075] In the pixel array section 31 of Fig. 3, the pixel block 41 includes one or more pixels 51, and the one or more pixels 51 share the event detecting section 52 and the pixel signal generating section 53. Thus, in the case where the pixel block 41 includes a plurality of pixels 51, the numbers of the event detecting sections 52 and the pixel signal generating sections 53 can be reduced as compared to a case where the event detecting section 52 and the pixel signal generating section 53 are provided for each of the pixels 51, with the result that the scale of the pixel array section 31 can be reduced.

[0076] Note that, in the case where the pixel block 41 includes a plurality of pixels 51, the event detecting section 52 can be provided for each of the pixels 51. In the case where the plurality of pixels 51 in the pixel block 41 share the event detecting section 52, events are detected in units of the pixel blocks 41. In the case where the event detecting section 52 is provided for each of the pixels 51, however, events can be detected in units of the pixels 51.

[0077] Yet, even in the case where the plurality of pixels 51 in the pixel block 41 share the single event detecting section 52, events can be detected in units of the pixels 51 when the transfer transistors 62 of the plurality of pixels 51 are temporarily turned on in a time-division manner.

[0078] Further, in a case where there is no need to output pixel signals, the pixel block 41 can be formed without the pixel signal generating section 53. In the case where the pixel block 41 is formed without the pixel signal generating section 53, the sensor section 21 can be formed without the AD conversion section 34 and the transfer transistors 63. In this case, the scale of the sensor section 21 can be reduced. The sensor will then output the address of the pixel (block) in which the event occurred, if necessary with a time stamp.

[0079] Fig. 5 is a block diagram illustrating a configuration example of the event detecting section 52 of Fig. 3.

[0080] The event detecting section 52 includes a current-voltage converting section 81, a buffer 82, a subtraction section 83, a quantization section 84, and a transfer section 85.

[0081] The current-voltage converting section 81 converts (a sum of) photocurrents from the pixels 51 to voltages corresponding to the logarithms of the photocurrents (hereinafter also referred to as a "photovoltage") and supplies the voltages to the buffer 82.

[0082] The buffer 82 buffers photovoltages from the current-voltage converting section 81 and supplies the resultant to the subtraction section 83.

[0083] The subtraction section 83 calculates, at a timing instructed by a row driving signal that is a control signal from the driving section 32, a difference between the current photovoltage and a photovoltage at a timing slightly shifted from the current time, and supplies a difference signal corresponding to the difference to the quantization section 84.

[0084] The quantization section 84 quantizes difference signals from the subtraction section 83 to digital signals and supplies the quantized values of the difference signals to the transfer section 85 as event data.

[0085] The transfer section 85 transfers (outputs), on the basis of event data from the quantization section 84, the event data to the output section 35. That is, the transfer section 85 supplies a request for requesting the output of the event data to the arbiter 33. Then, when receiving a response indicating event data output permission to the request from the arbiter 33, the transfer section 85 outputs the event data to the output section 35.

[0086] Fig. 6 is a circuit diagram illustrating a configuration example of the current-voltage converting section 81 of Fig. 5.

[0087] The current-voltage converting section 81 includes transistors 91 to 93. As the transistors 91 and 93, for example, N-type MOSFETs can be employed. As the transistor 92, for example, a P-type MOSFET can be employed.

[0088] The transistor 91 has a source connected to the gate of the transistor 93, and a photocurrent is supplied from the pixel 51 to the connecting point between the source of the transistor 91 and the gate of the transistor 93. The transistor 91 has a drain connected to the power supply VDD and a gate connected to the drain of the transistor 93.

[0089] The transistor 92 has a source connected to the power supply VDD and a drain connected to the connecting point between the gate of the transistor 91 and the drain of the transistor 93. A predetermined bias voltage Vbias is applied to the gate of the transistor 92. With the bias voltage Vbias, the transistor 92 is turned on or off, and the operation of the current-voltage converting section 81 is turned on or off depending on whether the transistor 92 is turned on or off.

[0090] The source of the transistor 93 is grounded.

[0091] In the current-voltage converting section 81, the transistor 91 has the drain connected on the power supply VDD side. The source of the transistor 91 is connected to the pixels 51 (Fig. 4), so that photocurrents based on charges generated in the photoelectric conversion elements 61 of the pixels 51 flow through the transistor 91 (from the drain to the source). The transistor 91 operates in a subthreshold region, and at the gate of the transistor 91, photovoltages corresponding to the logarithms of the photocurrents flowing through the transistor 91 are generated. As described above, in the current-voltage converting section 81, the transistor 91 converts photocurrents from the pixels 51 to photovoltages corresponding to the logarithms of the photocurrents.

[0092] In the current-voltage converting section 81, the transistor 91 has the gate connected to the connecting point between the drain of the transistor 92 and the drain of the transistor 93, and the photovoltages are output from the connecting point in question.

[0093] Fig. 7 is a circuit diagram illustrating configuration examples of the subtraction section 83 and the quantization section 84 of Fig. 5.

[0094] The subtraction section 83 includes a capacitor 101, an operational amplifier 102, a capacitor 103, and a switch 104. The quantization section 84 includes a comparator 111.

[0095] The capacitor 101 has one end connected to the output terminal of the buffer 82 (Fig. 5) and the other end connected to the input terminal (inverting input terminal) of the operational amplifier 102. Thus, photovoltages are input to the input terminal of the operational amplifier 102 through the capacitor 101.

[0096] The operational amplifier 102 has an output terminal connected to the non-inverting input terminal (+) of the comparator 111.

[0097] The capacitor 103 has one end connected to the input terminal of the operational amplifier 102 and the other end connected to the output terminal of the operational amplifier 102.

[0098] The switch 104 is connected to the capacitor 103 to switch the connections between the ends of the capacitor 103. The switch 104 is turned on or off in response to a row driving signal that is a control signal from the driving section 32, to thereby switch the connections between the ends of the capacitor 103.

[0099] A photovoltage on the buffer 82 (Fig. 5) side of the capacitor 101 when the switch 104 is on is denoted by Vinit, and the capacitance (electrostatic capacitance) of the capacitor 101 is denoted by Cl. The input terminal of the operational amplifier 102 serves as a virtual ground terminal, and a charge Qinit that is accumulated in the capacitor 101 in the case where the switch 104 is on is expressed by Expression (1).

[0100] Qinit = Cl x Vinit (1)

[0101] Further, in the case where the switch 104 is on, the connection between the ends of the capacitor 103 is cut (short-circuited), so that no charge is accumulated in the capacitor 103.

[0102] When a photovoltage on the buffer 82 (Fig. 5) side of the capacitor 101 in the case where the switch 104 has thereafter been turned off is denoted by Vafter, a charge Qafter that is accumulated in the capacitor 101 in the case where the switch 104 is off is expressed by Expression (2).

[0103] Qafter = Cl x Vafter (2)

[0104] When the capacitance of the capacitor 103 is denoted by C2 and the output voltage of the operational amplifier 102 is denoted by Vout, a charge Q2 that is accumulated in the capacitor 103 is expressed by Expression (3).

[0105] Q2 = -C2 x Vout (3)

[0106] Since the total amount of charges in the capacitors 101 and 103 does not change before and after the switch 104 is turned off, Expression (4) is established. Qinit = Qafter + Q2 (4)

[0107] When Expression (1) to Expression (3) are substituted for Expression (4), Expression (5) is obtained.

[0108] Vout = -(C1 / C2) x (Vafter - Vinit) (5)

[0109] With Expression (5), the subtraction section 83 subtracts the photovoltage Vinit from the photovoltage Vafter, that is, calculates the difference signal (Vout) corresponding to a difference Vafter - Vinit between the photovoltages Vafter and Vinit. With Expression (5), the subtraction gain of the subtraction section 83 is C1 / C2. Since the maximum gain is normally desired, Cl is preferably set to a large value and C2 is preferably set to a small value. Meanwhile, when C2 is too small, kTC noise increases, resulting in a risk of deteriorated noise characteristics. Thus, the capacitance C2 can only be reduced in a range that achieves acceptable noise. Further, since the pixel blocks 41 each have installed therein the event detecting section 52 including the subtraction section 83, the capacitances Cl and C2 have space constraints. In consideration of these matters, the values of the capacitances Cl and C2 are determined.

[0110] The comparator 111 compares a difference signal from the subtraction section 83 with a predetermined threshold (voltage) Vth (>0) applied to the inverting input terminal (-), thereby quantizing the difference signal. The comparator 111 outputs the quantized value obtained by the quantization to the transfer section 85 as event data.

[0111] For example, in a case where a difference signal is larger than the threshold Vth, the comparator 111 outputs an H (High) level indicating 1, as event data indicating the occurrence of an event. In a case where a difference signal is not larger than the threshold Vth, the comparator 111 outputs an L (Low) level indicating 0, as event data indicating that no event has occurred.

[0112] The transfer section 85 supplies a request to the arbiter 33 in a case where it is confirmed on the basis of event data from the quantization section 84 that a change in light amount that is an event has occurred, that is, in the case where the difference signal (Vout) is larger than the threshold Vth. When receiving a response indicating event data output permission, the transfer section 85 outputs the event data indicating the occurrence of the event (for example, H level) to the output section 35.

[0113] The output section 35 includes, in event data from the transfer section 85, location / address information regarding (the pixel block 41 including) the pixel 51 in which an event indicated by the event data has occurred and time point information indicating a time point at which the event has occurred, and further, as needed, the polarity of a change in light amount that is the event, i.e. whether the intensity did increase or decrease. The output section 35 outputs the event data.

[0114] As the data format of event data including location information regarding the pixel 51 in which an event has occurred, time point information indicating a time point at which the event has occurred, and the polarity of a change in light amount that is the event, for example, the data format called "AER (Address Event Representation)" can be employed. Note that, a gain A of the entire event detecting section 52 is expressed by the following expression where the gain of the current-voltage converting section 81 is denoted by CGiogand the gain of the buffer 82 is 1.

[0115] A = CGiogC 1 / C2 (ZiPhoto_n) (6)

[0116] Here, iPhoto_n denotes a photocurrent of the n-th pixel 51 of the IxJ pixels 51 in the pixel block 41. In Expression (6), E denotes the summation of n that takes integers ranging from 1 to IxJ.

[0117] Note that, the pixel 51 can receive any light as incident light with an optical fdter through which predetermined light passes, such as a color fdter. For example, in a case where the pixel 51 receives visible light as incident light, event data indicates the occurrence of changes in pixel value in images including visible objects. Further, for example, in a case where the pixel 51 receives, as incident light, infrared light, millimeter waves, or the like for ranging, event data indicates the occurrence of changes in distances to objects. In addition, for example, in a case where the pixel 51 receives infrared light for temperature measurement, as incident light, event data indicates the occurrence of changes in temperature of objects. In the present embodiment, the pixel 51 is assumed to receive visible light as incident light.

[0118] Fig. 8 is a diagram illustrating an example of a frame data generation method based on event data.

[0119] The logic section 22 sets a frame interval and a frame width on the basis of an externally input command, for example. Here, the frame interval represents the interval of frames of frame data that is generated on the basis of event data. The frame width represents the time width of event data that is used for generating frame data on a single frame. A frame interval and a frame width that are set by the logic section 22 are also referred to as a "set frame interval" and a "set frame width," respectively.

[0120] The logic section 22 generates, on the basis of the set frame interval, the set frame width, and event data from the sensor section 21, frame data that is image data in a frame format, to thereby convert the event data to the frame data.

[0121] That is, the logic section 22 generates, in each set frame interval, frame data on the basis of event data in the set frame width from the beginning of the set frame interval.

[0122] Here, it is assumed that event data includes time point information ti indicating a time point at which an event has occurred (hereinafter also referred to as an "event time point") and coordinates (x, y) serving as location information regarding (the pixel block 41 including) the pixel 51 in which the event has occurred (hereinafter also referred to as an "event location").

[0123] In Fig. 8, in a three-dimensional space (time and space) with the x axis, the y axis, and the time axis t, points representing event data are plotted on the basis of the event time point t and the event location (coordinates) (x, y) included in the event data. That is, when a location (x, y, t) on the three-dimensional space indicated by the event time point t and the event location (x, y) included in event data is regarded as the space-time location of an event, in Fig. 8, the points representing the event data are plotted on the space-time locations (x, y, t) of the events.

[0124] The logic section 22 starts to generate frame data on the basis of event data by using, as a generation start time point at which frame data generation starts, a predetermined time point, for example, a time point at which frame data generation is externally instructed or a time point at which the sensor device 10 is powered on.

[0125] Here, cuboids each having the set frame width in the direction of the time axis t in the set frame intervals, which appear from the generation start time point, are referred to as a "frame volume." The size of the frame volume in the x-axis direction or the y-axis direction is equal to the number of the pixel blocks 41 or the pixels 51 in the x-axis direction or the y-axis direction, for example.

[0126] The logic section 22 generates, in each set frame interval, frame data on a single frame on the basis of event data in the frame volume having the set frame width from the beginning of the set frame interval.

[0127] Frame data can be generated by, for example, setting white to a pixel (pixel value) in a frame at the event location (x, y) included in event data and setting a predetermined color such as gray to pixels at other locations in the frame.

[0128] Besides, in a case where event data includes the polarity of a change in light amount that is an event, frame data can be generated in consideration of the polarity included in the event data. For example, white can be set to pixels in the case a positive polarity, while black can be set to pixels in the case of a negative polarity.

[0129] In addition, in the case where pixel signals of the pixels 51 are also output when event data is output as described with reference to Fig. 3 and Fig. 4, frame data can be generated on the basis of the event data by using the pixel signals of the pixels 51. That is, frame data can be generated by setting, in a frame, a pixel at the event location (x, y) (in a block corresponding to the pixel block 41) included in event data to a pixel signal of the pixel 51 at the location (x, y) and setting a predetermined color such as gray to pixels at other locations.

[0130] Note that, in the frame volume, there are a plurality of pieces of event data that are different in the event time point t but the same in the event location (x, y) in some cases. In this case, for example, event data at the latest or oldest event time point t can be prioritized. Further, in the case where event data includes polarities, the polarities of a plurality of pieces of event data that are different in the event time point t but the same in the event location (x, y) can be added together, and a pixel value based on the added value obtained by the addition can be set to a pixel at the event location (x, y).

[0131] Here, in a case where the frame width and the frame interval are the same, the frame volumes are adjacent to each other without any gap. Further, in a case where the frame interval is larger than the frame width, the frame volumes are arranged with gaps. In a case where the frame width is larger than the frame interval, the frame volumes are arranged to be partly overlapped with each other.

[0132] Fig. 9 is a block diagram illustrating another configuration example of the quantization section 84 of Fig. 5.

[0133] Note that, in Fig. 9, parts corresponding to those in the case of Fig. 7 are denoted by the same reference signs, and the description thereof is omitted as appropriate below.

[0134] In Fig. 9, the quantization section 84 includes comparators 111 and 112 and an output section 113.

[0135] Thus, the quantization section 84 of Fig. 9 is similar to the case of Fig. 7 in including the comparator 111. However, the quantization section 84 of Fig. 9 is different from the case of Fig. 7 in newly including the comparator 112 and the output section 113.

[0136] The event detecting section 52 (Fig. 5) including the quantization section 84 of Fig. 9 detects, in addition to events, the polarities of changes in light amount that are events.

[0137] In the quantization section 84 of Fig. 9, the comparator 111 outputs, in the case where a difference signal is larger than the threshold Vth, the H level indicating 1, as event data indicating the occurrence of an event having the positive polarity. The comparator 111 outputs, in the case where a difference signal is not larger than the threshold Vth, the L level indicating 0, as event data indicating that no event having the positive polarity has occurred.

[0138] Further, in the quantization section 84 of Fig. 9, a threshold Vth' (<Vth) is supplied to the non-inverting input terminal (+) of the comparator 112, and difference signals are supplied to the inverting input terminal (-) of the comparator 112 from the subtraction section 83. Here, for the sake of simple description, it is assumed that the threshold Vth' is equal to -Vth, for example, which needs however not to be the case.

[0139] The comparator 112 compares a difference signal from the subtraction section 83 with the threshold Vth' applied to the inverting input terminal (-), thereby quantizing the difference signal. The comparator 112 outputs, as event data, the quantized value obtained by the quantization.

[0140] For example, in a case where a difference signal is smaller than the threshold Vth' (the absolute value of the difference signal having a negative value is larger than the threshold Vth), the comparator 112 outputs the H level indicating 1, as event data indicating the occurrence of an event having the negative polarity. Further, in a case where a difference signal is not smaller than the threshold Vth' (the absolute value of the difference signal having a negative value is not larger than the threshold Vth), the comparator 112 outputs the L level indicating 0, as event data indicating that no event having the negative polarity has occurred. The output section 113 outputs, on the basis of event data output from the comparators 111 and 112, event data indicating the occurrence of an event having the positive polarity, event data indicating the occurrence of an event having the negative polarity, or event data indicating that no event has occurred to the transfer section 85.

[0141] For example, the output section 113 outputs, in a case where event data from the comparator 111 is the H level indicating 1, +V volts indicating +1, as event data indicating the occurrence of an event having the positive polarity, to the transfer section 85. Further, the output section 113 outputs, in a case where event data from the comparator 112 is the H level indicating 1, -V volts indicating -1, as event data indicating the occurrence of an event having the negative polarity, to the transfer section 85. In addition, the output section 113 outputs, in a case where each event data from the comparators 111 and 112 is the L level indicating 0, 0 volts (GND level) indicating 0, as event data indicating that no event has occurred, to the transfer section 85.

[0142] The transfer section 85 supplies a request to the arbiter 33 in the case where it is confirmed on the basis of event data from the output section 113 of the quantization section 84 that a change in light amount that is an event having the positive polarity or the negative polarity has occurred. After receiving a response indicating event data output permission, the transfer section 85 outputs event data indicating the occurrence of the event having the positive polarity or the negative polarity (+V volts indicating 1 or -V volts indicating -1) to the output section 35.

[0143] Preferably, the quantization section 84 has a configuration as illustrated in Fig. 9.

[0144] Fig. 10 is a diagram illustrating another configuration example of the event detecting section 52.

[0145] In Fig. 10, the event detecting section 52 includes a subtractor 430, a quantizer 440, a memory 451, and a controller 452. The subtractor 430 and the quantizer 440 correspond to the subtraction section 83 and the quantization section 84, respectively.

[0146] Note that, in Fig. 10, the event detecting section 52 further includes blocks corresponding to the currentvoltage converting section 81 and the buffer 82, but the illustrations of the blocks are omitted in Fig. 10.

[0147] The subtractor 430 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434. The capacitor 431, the operational amplifier 432, the capacitor 433, and the switch 434 correspond to the capacitor 101, the operational amplifier 102, the capacitor 103, and the switch 104, respectively.

[0148] The quantizer 440 includes a comparator 441. The comparator 441 corresponds to the comparator 111.

[0149] The comparator 441 compares a voltage signal (difference signal) from the subtractor 430 with the predetermined threshold voltage Vth applied to the inverting input terminal (-). The comparator 441 outputs a signal indicating the comparison result, as a detection signal (quantized value). The voltage signal from the subtractor 430 may be input to the input terminal (-) of the comparator 441, and the predetermined threshold voltage Vth may be input to the input terminal (+) of the comparator 441.

[0150] The controller 452 supplies the predetermined threshold voltage Vth applied to the inverting input terminal (-) of the comparator 441. The threshold voltage Vth which is supplied may be changed in a time-division manner. For example, the controller 452 supplies a threshold voltage Vthl corresponding to ON events (for example, positive changes in photocurrent) and a threshold voltage Vth2 corresponding to OFF events (for example, negative changes in photocurrent) at different timings to allow the single comparator to detect a plurality of types of address events (events).

[0151] The memory 451 accumulates output from the comparator 441 on the basis of Sample signals supplied from the controller 452. The memory 451 may be a sampling circuit, such as a switch, plastic, or capacitor, or a digital memory circuit, such as a latch or flip-flop. For example, the memory 451 may hold, in a period in which the threshold voltage Vth2 corresponding to OFF events is supplied to the inverting input terminal (-) of the comparator 441, the result of comparison by the comparator 441 using the threshold voltage Vthl corresponding to ON events. Note that, the memory 451 may be omitted, may be provided inside the pixel (pixel block 41), or may be provided outside the pixel.

[0152] Fig. 11 is a block diagram illustrating another configuration example of the pixel array section 31 of Fig. 2.

[0153] Note that, in Fig. 11, parts corresponding to those in the case of Fig. 3 are denoted by the same reference signs, and the description thereof is omitted as appropriate below.

[0154] In Fig. 11, the pixel array section 31 includes the plurality of pixel blocks 41. The pixel block 41 includes the lx J pixels 51 that are one or more pixels and the event detecting section 52.

[0155] Thus, the pixel array section 31 of Fig. 11 is similar to the case of Fig. 3 in that the pixel array section 31 includes the plurality of pixel blocks 41 and that the pixel block 41 includes one or more pixels 51 and the event detecting section 52. However, the pixel array section 31 of Fig. 11 is different from the case of Fig. 3 in that the pixel block 41 does not include the pixel signal generating section 53.

[0156] As described above, in the pixel array section 31 of Fig. 11, the pixel block 41 does not include the pixel signal generating section 53, so that the sensor section 21 (Fig. 2) can be formed without the AD conversion section 34.

[0157] Fig. 12 is a circuit diagram illustrating a configuration example of the pixel block 41 of Fig. 11.

[0158] As described with reference to Fig. 11, the pixel block 41 includes the pixels 51 and the event detecting section 52, but does not include the pixel signal generating section 53.

[0159] In this case, the pixel 51 can only include the photoelectric conversion element 61 without the transfer transistors 62 and 63.

[0160] Note that, in the case where the pixel 51 has the configuration illustrated in Fig. 12, the event detecting section 52 can output a voltage corresponding to a photocurrent from the pixel 51, as a pixel signal.

[0161] Fig. 13 is a block diagram illustrating a configuration example of a scan type imaging device which may be used as an EVS.

[0162] As illustrated in Fig. 13, an imaging device 510 includes a pixel array section 521, a driving section 522, a signal processing section 525, a read-out region selecting section 527, and an optional signal generating section 528.

[0163] The pixel array section 521 includes a plurality of pixels 530. The plurality of pixels 530 each output an output signal in response to a selection signal from the read-out region selecting section 527. The plurality of pixels 530 can each include an in-pixel quantizer as illustrated in Fig. 10, for example. The plurality of pixels 530 outputs output signals corresponding to the amounts of change in light intensity. The plurality of pixels 530 may be two-dimensionally disposed in a matrix as illustrated in Fig. 13.

[0164] The driving section 522 drives the plurality of pixels 530, so that the pixels 530 output pixel signals generated in the pixels 530 to the signal processing section 525 through an output line 514. Note that, the driving section 522 and the signal processing section 525 are circuit sections for acquiring grayscale information.

[0165] The read-out region selecting section 527 selects some of the plurality of pixels 530 included in the pixel array section 521. For example, the read-out region selecting section 527 selects one or a plurality of rows included in the two-dimensional matrix structure corresponding to the pixel array section 521. The readout region selecting section 527 sequentially selects one or a plurality of rows on the basis of a cycle set in advance, e.g. based on a rolling shutter. Further, the read-out region selecting section 527 may determine a selection region on the basis of requests from the pixels 530 in the pixel array section 521.

[0166] The optional signal generating section 528 may generate, on the basis of output signals of the pixels 530 selected by the read-out region selecting section 527, event signals corresponding to active pixels in which events have been detected of the selected pixels 530. The events mean an event that the intensity of light changes. The active pixels mean the pixel 530 in which the amount of change in light intensity corresponding to an output signal exceeds or falls below a threshold set in advance. For example, the signal generating section 528 compares output signals from the pixels 530 with a reference signal, and detects, as an active pixel, a pixel that outputs an output signal larger or smaller than the reference signal. The signal generating section 528 generates an event signal (event data) corresponding to the active pixel.

[0167] The signal generating section 528 can include, for example, a column selecting circuit configured to arbitrate signals input to the signal generating section 528. Further, the signal generating section 528 can output not only information regarding active pixels in which events have been detected, but also information regarding non-active pixels in which no event has been detected.

[0168] The signal generating section 528 outputs, through an output line 515, address information and timestamp information (for example, (X, Y, T)) regarding the active pixels in which the events have been detected. However, the data that is output from the signal generating section 528 may not only be the address information and the timestamp information, but also information in a frame format (for example, (0, 0, 1, o, -)).

[0169] In the following description reference will mainly be made to sensor devices of the EVS type as described above in order to ease the description and to cover an important application example. However, the principles explained below apply just as well to any event-based vision sensor that is capable to generate events based on the occurrence of temporal intensity changes.

[0170] In the following description the above sensor is modified such as to allow switching between asynchronous readout and synchronous readout depending on the occurrence rate of events. This allows satisfactorily treating both, regimes of low event generation rate and regimes of high event generation rate. In particular, in contrast to concepts of switching between different synchronous readout modes, as disclosed e.g. in WO 2021 / 096825 Al, the present disclosure ensures that at least for low event generation rates each event is provided separately with time information due to asynchronous readout. This makes the temporal information for low event generation rates more reliable.

[0171] To this end, Fig. 14 illustrates in a schematic and simplified manner a sensor device 10. The sensor device 10 comprises a pixel array 1011 as e.g. described above that has a plurality of pixels 51 that are each configured to receive light and to perform photoelectric conversion to generate an electrical signal indicating the intensity of the received light.

[0172] The sensor device 10 comprises further an event detection unit 100 that is configured to receive the electrical signals from each of the plurality of pixels 51 and to generate event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel 51 above an event threshold. The event detection unit 100 may here be constituted by the abovedescribed plurality of event detecting sections 52 within the pixels 51 that form together the event detection unit 100. However, the event detection unit 100 may also be an external entity that is separated from the pixel array 1011. In fact, the pixel array 1011 and the event detection unit 100 may even be arranged at different chips / sensor dies. The pixel array 1011 may be located on the sensor die 11 and the event detection unit 100 may be located on the logic die 12. The event detection unit 100 may even be part of a different processor or a different computing device. However, it is preferred that the pixel array 1011 and the event detection unit 100 are located on the same chip and it is most preferred that the event detection unit 100 is constituted by a plurality of event detecting sections 52 as described above.

[0173] The sensor device 10 further comprises a control unit 110 that is configured to determine an event generation rate that indicates how many events have occurred during a predetermined time interval and that is configured to control the operation of the sensor device 10 or at least of the parts of the sensor device 10 that are described herein in detail. The control unit 110 may be constituted by any processing device, such as a processor, a microcomputer, a CPU, a GPU, a FPGA, an ASIC or the like. The control unit 110 may be implemented in hardware, in software or as a mixture of hardware and software. The control unit 110 may be located on the same chip as the pixel array 1011. However, it may also be located on a different chip. Preferably, the control unit 110 is part of the logic die 12.

[0174] In order to determine the event generation rate, the control unit 110 may comprise a counter that is increased by one for each event that is detected by the event detection unit 100 and that is read out and set to zero every second, millisecond, or microsecond. The readout value of the counter indicates than the occurred events per second / millisecond / mirosecond. Of course, this manner of determining the event generation rate is purely exemplary. Any other manner can be chosen as long as a reliable result for the event generation rate can be obtained.

[0175] The sensor device comprises further a timestamp unit 120 that is configured to receive from the event detection unit 100 requests for permission for outputting of event data and to add to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed. The time stamp unit 120 may for example be realized by (parts of the functions of) the arbiter 33 and (parts of the functions of) the transfer section 85 described above.

[0176] In asynchronous operation the timestamp unit 120 receives for each event that was detected a request for output permission of the corresponding event data, i.e. of the coordinates of the pixel 51 at which the event occurred (and possibly the polarity of the event). The timestamp unit 120 processes these requests in a predetermined order and adds a timestamp to the event data which indicates the time at which the timestamp unit 120 has finished processing the request, i.e. the time of allowing the request. The event data may then be output e.g. in the Address Event Representation, AER, format.

[0177] The time the timestamp unit 120 needs to process the requests for output permission is several magnitudes smaller than the typical time resolution of event data generation. In particular, one request can be processed in a thousandth of the time resolution of event data generation. For example, one request may be processed in 1 ns, while the time resolution of event data generation is in the ps range.

[0178] Thus, as long as the number of events is smaller than the order of magnitude of the ratio of time resolution of event data generation and request processing time, allocation of timestamps can be considered immediate. For example, if 100 events per second are detected, the time needed to process them all is in the range of 100 ns, i.e. it is a tenth of a temporal resolution of 1 ps of event data generation. Thus, at the obtainable precision, there is no delay between event detection / request for output permission and the allocation of the respective timestamp. Thus, for sufficiently small event generation rates, the timestamp can be considered to indicate the time of event occurrence precisely.

[0179] However, if the event generation rate becomes too large, there will be a considerable delay between event detection / request for output permission and actual allocation of a timestamp. For example, if 1,000 events occur per ms, it will according to a rough estimate take 1 ms to process all the corresponding requests. It can then no longer be guaranteed that the timestamps added to the event data represent the time of occurrence of the event. Moreover, the delay between request and permission will fluctuate with the actual number of occurring events, which makes it at least difficult to compensate this delay. As a consequence, event data carrying a timestamp are generated, which is only seemingly precise. This can lead to problems in the processing of event data.

[0180] In order to mitigate this problem, the control unit 110 is configured to instruct the event detection unit 100 to asynchronously request permission for outputting of event data from the timestamp unit 120, if the determined event generation rate is below or equal to a first readout threshold. Otherwise, i.e. if the determined event generation rate is above the first readout threshold, the control unit 110 is configured to instruct the event detection unit 100 to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time.

[0181] Thus, based on a readout threshold for the event generation rate it is decided whether to allow asynchronous readout as described above or whether to readout all events together at predetermined times, e.g. at a certain frame rate such as to generate event frames as e.g. described above with respect to Fig. 8. All the events in such an event frame will then share the same timestamp which indicates the time of the synchronous readout. In this manner, unreliable timestamps are avoided for high event generation rates. Instead, the temporal resolution of the detected events is lowered to the readout cycle of the synchronous readout. This is preferable, since for further processing of the event data it is less detrimental to have less precise, but reliable time information than to be provided with seemingly precise, but unreliable time information.

[0182] Here, the timestamp unit 120 may be configured to process the requests for permission for outputting of event data consecutively according to the time at which the requests have been received at the timestamp unit 120. The event detection unit 100 detects events with different coordinates (xi, yi) on the pixel array 1011 at different time ti and requests output permission at the timestamp unit 120. The timestamp unit 120 receives the requests in temporal order and adds timestamps to the respective event data in this temporal order.

[0183] This is exemplary illustrated in Fig. 15. Here, three events are exemplary shown that occurred at times tl, t2, t3, where tl < t2 < t3. As illustrated in Fig. 15 the time intervals between the single events may differ. In asynchronous operation, output permission is requested for each of the events immediately after event detection, i.e. at times tl, t2, and t3. The timestamp unit 120 receives the requests and allows permission according to the temporal order of reception. Thus, first the event occurring at tl is allowed, then the event at t2, and then the event at t3. Here, allowance of the requests depends on the processing rate of the timestamp unit 120. Thus, as illustrated in Fig. 15, the times of allowance may be separated by equal time intervals, if the event generation rate is higher than the processing rate of the timestamp unit 120. In this case, the timestamps tl’, t2’, t3’ allocated by the timestamp unit 120 may have a delay with respect to the request times tl, t2, t3. Whether or not this delay affects the precision of the timestamp depends on the number of events that is detected. As shown in Fig. 15, the event detection unit 100 may provide the timestamp unit 120 with event data that includes as least the position of the event on the pixel array 1011. The timestamp unit 120 can then add the time information directly to the event data. However, the timestamp unit 120 may also merely receive requests for output permission without event data. Then, the timestamp unit 120 may respond to the requests with sending out the timestamp. In this case, the event data are supplemented with the time information outside the timestamp unit 120. Also, as shown in Fig. 15 the timestamp unit 120 may send event data with time information back to the event detection unit 100 for output. But the timestamp unit 120 may directly output the event data or provide the event data to another component of the sensor device 10 for output, e.g. to the control unit 110.

[0184] It should be noted that the timestamp unit 120 may allocate timestamps also in a different, predetermined order that differs from the above described first-in-first-out processing. The order of allocating timestamps is in principle arbitrary, as long as it is ensured that for low event generation rates the timestamps indicate the time of event occurrence within the precision of event data generation. Preferable is, however, the above-described consecutive timestamp allocation, since it is easy to implement.

[0185] For example, timestamp allocation / processing of output requests may be based on arbitration trees, constituted e.g. from C-elements (see e.g. “Event-Based Neuromorphic Systems” by Liu et al., Wiley & Sons Ltd, 2015, which is incorporated for reference in this regard). The principle functioning of such an arbitration tree is exemplified in Figs. 16A to 16D. How to implement such functions is known to a skilled person and need not be described here.

[0186] As shown in Fig. 16A, four requests have been received at the timestamp unit 120 at times tl to t4 and have not been processed yet. The temporal order is tl < t2 < t3 < t4. The arbitration tree compares request times, forwarding earlier times to a next layer. Thus, in Fig. 16A, tl and t3 are forwarded to the second layer, and tl is forwarded to the third layer. Then, the request received at tl is permitted and is dropped, while positions of non-forwarded events within the arbitration tree are memorized.

[0187] After permitting the tl -request, the arbitration tree is evaluated again, taking also account of newly received requests. This is shown in Fig. 16B, where now the t2 -event is forwarded and permitted and a new request that has been received at time t5 (t4 < t5) has been entered at the lowest level. Figs. 16C and 16D show the next two processing steps.

[0188] In arbitration trees as shown in Fig. 16A to 16B, the entire tree can be processed in (N-l) steps for N requests (for N = 2nand n a natural number). However, since the results of comparisons are memorized in unchanged prongs of the tree, it does not take the full (N-l) processing steps to select a request for permission. For example, in Figs. 16B and 16C only two new processing steps are necessary to this end. On the other hand, the arbitration tree has to be evaluated N times until the last of N pending requests is forwarded. This is also illustrated in Figs. 16A to 16D, where the request received at t4 is only forwarded after N=4 evaluations of the arbitration tree.

[0189] It can therefore be estimated that for N pending requests, it takes roughly between N / 2 and N2processing steps until the last received request is processed. For a processing period of 1 ns for one processing step, one can therefore assume that the timestamp unit 120 is configured to process a request for permission for outputting of event data within a time period between N / 2 ns to N2ns, where N is the number of requests that have been received by the timestamp unit 120 and not processed by the timestamp unit 120 at the time said request for permission for outputting of event data is received by the timestamp unit 120, i.e. for N pending requests.

[0190] It has to be understood that the above example of Figs. 16A to 16D is purely exemplary and that the ordering and decision logic of an arbitration tree used by the timestamp unit 120 may also be different, as long as it can provide output permission after a reasonable time. Moreover, although in the above example requests from all occurred events are fed into a single arbitration tree, it might be advantageously to provide separate arbitration trees for different pixel groups. Then, there is first an arbitration between events occurring in different groups, followed by an arbitration of events occurring (during the first arbitration) in the group selected by the first arbitration.

[0191] For example, one arbitration tree may be provided to select rows of the pixel array 1011 for output and one arbitration tree may be used to select a column position for output within the selected row. The scheme of Figs. 16A to 16D may therefore apply to requests referring to events from different rows (first arbitration tree) as well as to requests referring to events occurring in a single row during the time until the row is selected for readout (second arbitration tree). Here, is should be noted that both arbitration trees may be evaluated anew once a request is granted or that once a row is selected, all requests of events occurring in that row may be processed before a new row is selected. It may also be possible to switch dynamically between these readout schemes, e.g. depending on the event generation rate, the observed scene or the like.

[0192] In the above example, asynchronous readout is fully switched to synchronous readout of event data, once the event generation rate crosses the first readout threshold. Although this leads to reliable time information, it will be advantageous, if the loss of time information that comes with synchronous readout can be reduced as much as possible, while still keeping the time information reliable. To this end, an intermediate readout regime can be introduced, in which the readout is asynchronous for events coming from different pixel groups, but synchronous for events occurring within a single group of pixels 51 . This means that while requests referring to events from different groups of pixels 51 are processed as in the purely asynchronous case, requests referring to events from a single group will all be permitted together, once a request referring to one of those events is permitted.

[0193] This allows reducing the number of requests to be processed compared to the purely asynchronous readout regime due to synchronous readout within the groups of pixels 51, while at the same time the temporal precision for events from different groups is kept at the level of purely asynchronous readout.

[0194] In particular, if the event generation rate is below or at a second readout threshold that is smaller than the first readout threshold, the control unit 110 may be configured to instruct the timestamp unit 120 to process all requests for permission for outputting event data consecutively according to the time at which the requests have been received at the timestamp unit, i.e. to apply purely asynchronous readout.

[0195] However, if the event generation rate is above the second readout threshold and below or at the first readout threshold, the control unit 110 may be configured to instruct the timestamp unit 120 to give permission for outputting event data of all events occurring in one of a plurality of predetermined groups of pixels 51 between a point in time at which permission for outputting event data is requested at the timestamp unit 120 for an event occurring in a pixel 51 of said one group of pixels and a point in time at which the timestamp unit 120 allows said request and to process requests for permission for outputting event data from pixels 51 of different groups of pixels 51 consecutively according to the time at which the requests have been received at the timestamp unit 120.

[0196] One can assume that a request for output permission is received that refers to an event from a group of pixels 51 for which no further requests are pending at the timestamp unit 120. Then, it will take the timestamp unit 120 a certain time until said request can be processed and allowed. During this time further events can be detected in said group, which lead to according, further requests. Once said initial request is allowed, all the further requests referring to events occurring in the same group of pixels 51 will be allowed, too, and all the respective event data will be supplemented with the same timestamp.

[0197] This will lead to a first reduction of readout requests that allows processing of the remaining requests without introducing a recognizable delay. If the event generation rate crosses also the first readout threshold, delays will become recognizable also in this readout regime. Thus, for an event generation rate that is larger than the first readout threshold, synchronized readout becomes effective.

[0198] This is schematically illustrated in Fig. 17 which shows an example of a development of the event generation rate with time together with the first readout threshold T1 and the second readout threshold T2. Here, depending on the used sensor device 10, the first readout threshold may he between 1,000,000 events per second and 2,000,000 events per second, and the second readout threshold may lie between 1,000 events per second and 10,000 events per second, preferably between 3,000 events per second and 7,000 events per second. It should be noted that these figures are purely exemplary, the first and second readout thresholds could also be larger or smaller. In fact, the values of the first and second readout thresholds depend on the temporal precision of event generation and the time that is necessary for allowing an output request.

[0199] As shown in Fig. 17, as long as the event generation rate is below the second readout threshold T2 (regime A), purely asynchronous readout is applied. Between the second readout threshold T2 and the first readout threshold T1 (regime B) groupwise or semi-asynchronous readout is applied. Above the first readout threshold T2, event frames are readout synchronously.

[0200] The different readout regimes are also illustrated in Figs. 18A to 18C. Fig. 18A relates to regime A and shows a low number of events with coordinates (xl, y2, tl) to (xn, yn, tn). These can be transferred into according event data without a temporal delay. Fig. 18B relates to regime B and shows an exemplary division of the pixel array 1011 into four groups of pixels 51. Fig. 18B shows an increased number of events compared to the situation of Fig. 18A. In the first group of pixels 51 events with coordinates (xl, y 1, tl) to (xk, yk, tk) occurred, where tl is the earliest time. In the second group of pixels 51 events with coordinates (xn, yn, t3) to (xo, yo, to) occurred, where t3 is the earliest time. In the third group of pixels 51 events with coordinates (xp, yp, t4) to (xq, yq, tq) occurred, where t4 is the earliest time. In the fourth group of pixels 51 events with coordinates (xl, yl, t2) to (xm, ym, tm) occurred, where t2 is the earliest time. Moreover, tl < t2 < t3 < t4 holds.

[0201] The readout requests from the different groups are processed consecutively, such that the requests received at tl, t2, t3, and t4 are allowed in this order. Together with these requests also all other requests of the respective groups are allowed. Thus, for the first groups event data (xl, y 1, tl), . . . , (xk, yk, tl) are generated and output that have all a tl -timestamp. Next, the event data of the fourth group (xl, yl, t2), . . . , (xn, ym, t2) are generated and output, followed by the event data of the second group (xn, yn, t3), . . . , (xo, yo, t3) and the third group (xp, yp, t4), . . . , (xq, yq, t4).

[0202] Here, it should be noted that the times tl, t2, t3, and t4 indicated in the event data correspond (up to delays beyond the temporal precision of the event data) to the times of the respective event occurrences. However, these times are allocated to all events that have occurred between requesting output of the events occurring at times tl, t2, t3, and t4, respectively, and allowing these requests, i.e. all events that occurred during the arbitration between the different groups. Accordingly, the time precision for each group is higher than a purely synchronous readout, while also the delay caused by processing the output requests remains sufficiently small.

[0203] Fig. 18C relates to regime C. Here, a large number of events has occurred during a predetermined time interval. At the end of this time interval, all these events are read out and provided with the readout time tc. The start of said time interval being either the transition into readout regime C or the last read out of all events. Readout may be periodic, thus defining a frame rate. Readout intervals may also depend on external conditions like e.g. the observed scene, the amount of motion in the scene or the like. As explained above purely synchronous readout ensures reliable time information while sacrificing the in principle available temporal precision.

[0204] A possible, exemplary principle of processing in readout regime B is illustrated in Figs. 19A to 19D. This principle of processing is based on the arbitration processing explained above with respect to Figs. 16A to 16D.

[0205] As shown in Fig. 19A output permission is requested for several events stemming from different groups. In the example of Fig. 19A output of event data of an event occurring at time tl in group gl, of an event occurring at time t2 in group gl, of an event occurring at time t3 at group g3, and of an event occurring at time t4 in group g4 is requested, with tl < t2 < t3 < t4. As in the example of Fig. 16A the tl-request and the t3-request are forwarded to the second layer of the arbitration tree and the tl-request is selected. But then, not only the tl-request is permitted, but also all other requests originating from group gl, i.e. in the example of Fig. 19A also the t2 -event. The permitted requests are then deleted in the arbitration tree. The permited and subsequently deleted requests are indicated in Figs. 19A to 19D by broken lines.

[0206] As illustrated in Fig. 19B, before the next processing of the arbitration tree, three new requests have been added: at t5 one from group gl, at t6 one from group g2, and at t7 one from group g3 (t5 < t6 < t7). In this processing stage the t3-request from group g3 is selected. Accordingly, also the t7-request from group g3 is permited and dropped from the arbitration tree.

[0207] Next, as illustrated in Fig. 19C, readout requests at times t8 to tl 2 are received, coming from groups g3, g4, g2, g2, and gl, respectively. Since the t4-request from g4 is selected, also the t9-request from g4 is permited and subsequently deleted. According to the same principle in the processing stage shown in Fig. 19D all requests from gl are permited.

[0208] Although only an example of an operation principle, it is apparent from the description of Fig. 19A to 19D that by permiting read out of all requests originating from the same group of pixels 51 the processing can be speeded up such that although the number of events that occur during arbitration increases, the processing delay can be kept neglectable.

[0209] In the above description, general groups of pixels 51 were considered that might or might not overlap, and which might or might not be continuous. However, preferably, the pixel array 1011 is a two-dimensional array of pixels 51 and the groups of pixels 51 are constituted either by the rows of pixels or columns of pixels of the pixel array 1011. This means that while in readout regime B columns of pixels are selected for readout in an asynchronous manner, e.g. via an arbitration tree, once a column is selected, all row positions of that column in which an event did occur are readout together and the same timestamp is assigned to all the events in the selected column.

[0210] This is schematically illustrated in Fig. 20. Fig. 20 shows the distribution of events (here, of positive polarity P and of negative polarity N) on the pixel array 1011. The timestamp unit 1011 comprises an arbitration tree 120a that might operate according to the above-described principle. But the arbitration tree 120a may also operate only on the pixel columns, i.e. columns are selected fully asynchronously according to the principle illustrated in Fig. 16A to 16D. Then, the common readout of all events in the same column may be considered as a switching off of a row arbitration that operates fully asynchronously on the rows of a selected column. In any case the exact nature of the arbitration processing is arbitrary as long as an order of readout of columns can be established and as long as all events in one column are read out together that occur during the time it takes to decide which column to read out.

[0211] As shown in Fig. 20, a logic unit 120b gathers the event addresses from all the pixels 51 to be read out and an addressing unit 120c completes the event data by adding the time of the column readout. The event data are then output by the timestamp unit 120.

[0212] Since the above separation between different readout regimes depends on the detected event generation rate, the reliability of the time information can be further improved by ensuring that only true events are counted, while events generated e.g. by defective pixels 51 are not considered. To this end, the control unit 110 may be configured to determine the event generation rate without considering events that occur in pixels 51 when no light is received at the respective pixels 51.

[0213] In particular, such “hot” or “oscillating” pixels 51 that indicate changes in the received illumination although such changes are not present in the received illumination can be determined during calibration of the sensor device 10. The respective calibration steps are schematically illustrated in Fig. 21. First, the sensor device 10 is operated without illumination. Then, the event generation rate is determined. If the event generation rate is above a certain noise threshold that might be in the range of the second readout threshold, it is determined that further calibration is necessary.

[0214] To this end, the pixels 51 are determined at which the events where generated. This is illustrated at the upper left of Fig. 21. From this information a (virtual) mask 1012 is determined in which pixels 51 to be masked or disabled are identified (upper right of Fig. 21). Combining the obtained mask 1012 with the actually detected events allows disregarding events caused by defective pixels 51. In this case, use of the mask 1012 leads effectively (i.e. up to white noise) to the detection of no events during an unilluminated operation of the sensor device 10 (bottom of Fig. 21).

[0215] Here, the mask 1012 might be used to forbid readout of all event data that are generated for pixels 51 indicated in the mask 1012. The mask 1012 may also be used to not allow requests for readout permission from such pixels 51. However, most preferably the mask 1012 is used to avoid detection of events at the pixels 51 indicated in the mask 1012 at all.

[0216] To this end, said pixels 51, i.e. pixels in which events are occurring without the reception of light, are disabled. For example, disabling transistors may be provided in all pixels 51 that disable the pixels 51, if they are switched on. As such transistors any transistor may be used that can put a sensitive pixel node to ground or VDD, if switched on. Disabling a pixel 51 can then be easily implemented by setting switch on bits for each disabling transistor according to the mask 1012. This ensures that no false events are generated by the respective defective pixels 51, which false events may influence the event generation rate and may unnecessarily delay the request processing by requesting readout permission forthem.

[0217] By applying an according calibration, the reliability of the decision increases, which readout regime is to be used best. This increases in turn the reliability of the time information allocated to the event data.

[0218] Fig. 22 illustrates again the method used for operating the sensor device 10 described above. At S 101 , at each of the plurality of pixels 51, light is received to perform photoelectric conversion and an electrical signal is generated that indicates the intensity of the received light.

[0219] At SI 02 the event detection unit 100 is receiving the electrical signals from each of the plurality of pixels 51 and is generating event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel 51 above an event threshold.

[0220] At SI 03 the control unit 110 determines an event generation rate that indicates how many events have occurred during a predetermined time interval.

[0221] At S104 the timestamp unit 120 receives from the event detection unit 100 requests for permission for outputting of event data and adds to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed.

[0222] At SI 05 the control unit 110 instructs the event detection unit 100 to asynchronously request permission for outputting of event data from the timestamp unit 120, if the determined event generation rate is below or equal to a first readout threshold.

[0223] And at SI 06 the control unit 110 instructs the event detection unit 100 to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time, if the determined event generation rate is above the first readout threshold.

[0224] In this manner it is possible to generate event data that include reliable time information irrespective of whether the event generation rate is low or high.

[0225] The technology according to the above (i.e. the present technology) is applicable to various products. For example, the technology according to the present disclosure may be realized as a device that is installed on any kind of moving bodies, for example, vehicles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobilities, airplanes, drones, ships, and robots.

[0226] Fig. 23 is a block diagram depicting an example of schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied.

[0227] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In the example depicted in Fig. 23, the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside-vehicle information detecting unit 12030, an in-vehicle information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are illustrated as a functional configuration of the integrated control unit 12050.

[0228] The driving system control unit 12010 controls the operation of devices related to the driving system of the vehicle in accordance with various kinds of programs. For example, the driving system control unit 12010 functions as a control device for a driving force generating device for generating the driving force of the vehicle, such as an internal combustion engine, a driving motor, or the like, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, and the like. The body system control unit 12020 controls the operation of various kinds of devices provided to a vehicle body in accordance with various kinds of programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various kinds of lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals of various kinds of switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, the power window device, the lamps, or the like of the vehicle.

[0229] The outside-vehicle information detecting unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the outside-vehicle information detecting unit 12030 is connected with an imaging section 12031. The outside-vehicle information detecting unit 12030 makes the imaging section 12031 image an image of the outside of the vehicle, and receives the imaged image. On the basis of the received image, the outside-vehicle information detecting unit 12030 may perform processing of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or processing of detecting a distance thereto.

[0230] The imaging section 12031 is an optical sensor that receives light, and which outputs an electric signal corresponding to a received light amount of the light. The imaging section 12031 can output the electric signal as an image, or can output the electric signal as information about a measured distance. In addition, the light received by the imaging section 12031 may be visible light, or may be invisible light such as infrared rays or the like.

[0231] The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle. The in-vehicle information detecting unit 12040 is, for example, connected with a driver state detecting section 12041 that detects the state of a driver. The driver state detecting section 12041, for example, includes a camera that images the driver. On the basis of detection information input from the driver state detecting section 12041, the in-vehicle information detecting unit 12040 may calculate a degree of fatigue of the driver or a degree of concentration of the driver, or may determine whether the driver is dozing.

[0232] The microcomputer 12051 can calculate a control target value for the driving force generating device, the steering mechanism, or the braking device on the basis of the information about the inside or outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control intended to implement functions of an advanced driver assistance system (ADAS) which functions include collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, vehicle speed maintaining driving, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, or the like.

[0233] In addition, the microcomputer 12051 can perform cooperative control intended for automatic driving, which makes the vehicle to travel autonomously without depending on the operation of the driver, or the like, by controlling the driving force generating device, the steering mechanism, the braking device, or the like on the basis of the information about the outside or inside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040.

[0234] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 on the basis of the information about the outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030. For example, the microcomputer 12051 can perform cooperative control intended to prevent a glare by controlling the headlamp so as to change from a high beam to a low beam, for example, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detecting unit 12030.

[0235] The sound / image output section 12052 transmits an output signal of at least one of a sound and an image to an output device capable of visually or auditorily notifying information to an occupant of the vehicle or the outside of the vehicle. In the example of Fig. 23, an audio speaker 12061, a display section 12062, and an instrument panel 12063 are illustrated as the output device. The display section 12062 may, for example, include at least one of an on-board display and a head-up display.

[0236] Fig. 24 is a diagram depicting an example of the installation position of the imaging section 12031.

[0237] In Fig. 24, the imaging section 12031 includes imaging sections 12101, 12102, 12103, 12104, and 12105.

[0238] The imaging sections 12101, 12102, 12103, 12104, and 12105 are, for example, disposed at positions on a front nose, sideview mirrors, a rear bumper, and a back door of the vehicle 12100 as well as a position on an upper portion of a windshield within the interior of the vehicle. The imaging section 12101 provided to the front nose and the imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle obtain mainly an image of the front of the vehicle 12100. The imaging sections 12102 and 12103 provided to the sideview mirrors obtain mainly an image of the sides of the vehicle 12100. The imaging section 12104 provided to the rear bumper or the back door obtains mainly an image of the rear of the vehicle 12100. The imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle is used mainly to detect a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, or the like.

[0239] Incidentally, Fig. 24 depicts an example of photographing ranges of the imaging sections 12101 to 12104. An imaging range 12111 represents the imaging range of the imaging section 12101 provided to the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging sections 12102 and 12103 provided to the sideview mirrors. An imaging range 12114 represents the imaging range of the imaging section 12104 provided to the rear bumper or the back door. A bird’s-eye image of the vehicle 12100 as viewed from above is obtained by superimposing image data imaged by the imaging sections 12101 to 12104, for example.

[0240] At least one of the imaging sections 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging sections 12101 to 12104 may be a stereo camera constituted of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0241] For example, the microcomputer 12051 can determine a distance to each three-dimensional object within the imaging ranges 12111 to 12114 and a temporal change in the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, and thereby extract, as a preceding vehicle, a nearest three-dimensional object in particular that is present on a traveling path of the vehicle 12100 and which travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or more than 0 km / hour). Further, the microcomputer 12051 can set a following distance to be maintained in front of a preceding vehicle in advance, and perform automatic brake control (including following stop control), automatic acceleration control (including following start control), or the like. It is thus possible to perform cooperative control intended for automatic driving that makes the vehicle travel autonomously without depending on the operation of the driver or the like.

[0242] For example, the microcomputer 12051 can classify three-dimensional object data on three-dimensional objects into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a largesized vehicle, a pedestrian, a utility pole, and other three-dimensional objects on the basis of the distance information obtained from the imaging sections 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of an obstacle. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can recognize visually and obstacles that are difficult for the driver of the vehicle 12100 to recognize visually. Then, the microcomputer 12051 determines a collision risk indicating a risk of collision with each obstacle. In a situation in which the collision risk is equal to or higher than a set value and there is thus a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display section 12062, and performs forced deceleration or avoidance steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist in driving to avoid collision.

[0243] At least one of the imaging sections 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 can, for example, recognize a pedestrian by determining whether or not there is a pedestrian in imaged images of the imaging sections 12101 to 12104. Such recognition of a pedestrian is, for example, performed by a procedure of extracting characteristic points in the imaged images of the imaging sections 12101 to 12104 as infrared cameras and a procedure of determining whether or not it is the pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the imaged images of the imaging sections 12101 to 12104, and thus recognizes the pedestrian, the sound / image output section 12052 controls the display section 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the recognized pedestrian. The sound / image output section 12052 may also control the display section 12062 so that an icon or the like representing the pedestrian is displayed at a desired position. An example of the vehicle control system to which the technology according to the present disclosure is applicable has been described above. The technology according to the present disclosure is applicable to the imaging section 12031 among the above-mentioned configurations. Specifically, the sensor device 10 is applicable to the imaging section 12031. The imaging section 12031 to which the technology according to the present disclosure has been applied flexibly acquires event data and performs data processing on the event data, thereby being capable of providing appropriate driving assistance.

[0244] Further possible implementations of the sensor device 10 are mobile devices 3000 such as cell phones, tablets, smart watches and the like as shown in Fig. 25A or head-mounted displays 4000 as shown in Fig. 25B. Further, the sensor device 10 is useable in augmented and / or virtual reality applications / cameras or in surveillance systems like 360° cameras.

[0245] Note that, the embodiments of the present technology are not limited to the above-mentioned embodiment, and various modifications can be made without departing from the gist of the present technology.

[0246] Further, the effects described herein are only exemplary and not limited, and other effects may be provided.

[0247] Note that, the present technology can also take the following configurations.

[0248] [1] A sensor device (10) comprising: a pixel array (1011) having a plurality of pixels (51) each being configured to receive light and to perform photoelectric conversion to generate an electrical signal indicating the intensity of the received light; an event detection unit (100) that is configured to receive the electrical signals from each of the plurality of pixels (51) and to generate event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel (51) above an event threshold; a control unit (110) that is configured to determine an event generation rate that indicates how many events have occurred during a predetermined time interval; and a timestamp unit (120) that is configured to receive from the event detection unit (100) requests for permission for outputting of event data and to add to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed; wherein the control unit (110) is configured to instruct the event detection unit (100) to asynchronously request permission for outputting of event data from the timestamp unit (120), if the determined event generation rate is below or equal to a first readout threshold; and the control unit (110) is configured to instruct the event detection unit (100) to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time, if the determined event generation rate is above the first readout threshold.

[0249] [2] The sensor device (10) according to [1], wherein the timestamp unit (120) is configured to process the requests for permission for outputting of event data consecutively according to the time at which the requests have been received at the timestamp unit.

[0250] [3] The sensor device (10) according to [1] or [2], wherein the timestamp unit (120) is configured to process a request for permission for outputting of event data within a time period between N / 2 ns to N2ns, where N is the number of requests that have been received by the timestamp unit (120) and not processed by the timestamp unit (120) at the time said request for permission for outputting of event data is received by the timestamp unit (120).

[0251] [4] The sensor device (10) according to any one of [1] to [3], wherein if the event generation rate is below or at a second readout threshold that is smaller than the first readout threshold, the control unit (110) is configured to instruct the timestamp unit (120) to process all requests for permission for outputting event data consecutively according to the time at which the requests have been received at the timestamp unit; and if the event generation rate is above the second readout threshold and below or at the first readout threshold, the control unit (110) is configured to instruct the timestamp unit (120) to give permission for outputting event data of all events occurring in one of a plurality of predetermined groups of pixels (51) between a point in time at which permission for outputting event data is requested at the timestamp unit (120) for an event occurring in a pixel (51) of said one group of pixels and a point in time at which the timestamp unit (120) allows said request and to process requests for permission for outputting event data from pixels (51) of different groups of pixels (51) consecutively according to the time at which the requests have been received at the timestamp unit (120).

[0252] [5] The sensor device (10) according to [4], wherein the pixel array is (1011) a two-dimensional array of pixels (51); and the predetermined groups of pixels (51) are rows of pixels or columns of pixels.

[0253] [6] The sensor device (10) according to any one of [1] to [5], wherein the first readout threshold is between 1,000,000 events per second and 2,000,000 events per second; and the second readout threshold is between 1,000 events per second and 10,000 events per second, preferably between 3,000 events per second and 7,000 events per second.

[0254] [7] The sensor device (10) according to any one of [1] to [6], wherein the control unit (110) is configured to determine the event generation rate without considering events occurring in pixels (51) without the reception of light.

[0255] [8] The sensor device (10) according to [7], wherein pixels (51) in which events are occurring without the reception of light are disabled by switching on a disabling transistors in said pixels (51). [9] A method for operating the sensor device (10) according to any one of [1] to [8], the method comprising: at each of the plurality of pixels (51), receiving light to perform photoelectric conversion and generating an electrical signal indicating the intensity of the received light; at the event detection unit (100), receiving the electrical signals from each of the plurality of pixels (51) and generating event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel (51) above an event threshold; at the control unit (110), determining an event generation rate that indicates how many events have occurred during a predetermined time interval; at the timestamp unit (120), receiving from the event detection unit (100) requests for permission for outputting of event data and adding to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed; by the control unit (110), instructing the event detection unit (100) to asynchronously request permission for outputting of event data from the timestamp unit (120), if the determined event generation rate is below or equal to a first readout threshold; and by the control unit (110), instructing the event detection unit (100) to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time, if the determined event generation rate is above the first readout threshold.

Claims

CLAIMS1. A sensor device comprising: a pixel array having a plurality of pixels each being configured to receive light and to perform photoelectric conversion to generate an electrical signal indicating the intensity of the received light; an event detection unit that is configured to receive the electrical signals from each of the plurality of pixels and to generate event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel above an event threshold; a control unit that is configured to determine an event generation rate that indicates how many events have occurred during a predetermined time interval; and a timestamp unit that is configured to receive from the event detection unit requests for permission for outputting of event data and to add to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed; wherein the control unit is configured to instruct the event detection unit to asynchronously request permission for outputting of event data from the timestamp unit, if the determined event generation rate is below or equal to a first readout threshold; and the control unit is configured to instruct the event detection unit to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time, if the determined event generation rate is above the first readout threshold.

2. The sensor device according to claim 1, wherein the timestamp unit is configured to process the requests for permission for outputting of event data consecutively according to the time at which the requests have been received at the timestamp unit.

3. The sensor device according to claim 1, wherein the timestamp unit is configured to process a request for permission for outputting of event data within a time period between N / 2 ns to N2ns, where N is the number of requests that have been received by the timestamp unit and not processed by the timestamp unit at the time said request for permission for outputting of event data is received by the timestamp unit.

4. The sensor device according to claim 1, wherein if the event generation rate is below or at a second readout threshold that is smaller than the first readout threshold, the control unit is configured to instruct the timestamp unit to process all requests for permission for outputting event data consecutively according to the time at which the requests have been received at the timestamp unit; and if the event generation rate is above the second readout threshold and below or at the first readout threshold, the control unit is configured to instruct the timestamp unit to give permission for outputting event data of all events occurring in one of a plurality of predetermined groups of pixels between a point in time at which permission for outputting event data is requested at the timestamp unitfor an event occurring in a pixel of said one group of pixels and a point in time at which the timestamp unit allows said request and to process requests for permission for outputting event data from pixels of different groups of pixels consecutively according to the time at which the requests have been received at the timestamp unit.

5. The sensor device according to claim 4, wherein the pixel array is a two-dimensional array of pixels; and the predetermined groups of pixels are rows of pixels or columns of pixels.

6. The sensor device according to claim 4, wherein the first readout threshold is between 1,000,000 events per second and 2,000,000 events per second; and the second readout threshold is between 1,000 events per second and 10,000 events per second, preferably between 3,000 events per second and 7,000 events per second.

7. The sensor device according to claim 1, wherein the control unit is configured to determine the event generation rate without considering events occurring in pixels without the reception of light.

8. The sensor device according to claim 7, wherein pixels in which events are occurring without the reception of light are disabled by switching on a disabling transistors in said pixels.

9. A method for operating the sensor device of claim 1, the method comprising: at each of the plurality of pixels, receiving light to perform photoelectric conversion and generating an electrical signal indicating the intensity of the received light; at the event detection unit, receiving the electrical signals from each of the plurality of pixels and generating event data that indicate as an event the occurrence of a change of the difference of the intensities of light received at the respective pixel above an event threshold; at the control unit, determining an event generation rate that indicates how many events have occurred during a predetermined time interval; at the timestamp unit, receiving from the event detection unit requests for permission for outputting of event data and adding to the event data a timestamp that indicates the time at which the request for permission for outputting of said event data had been allowed; by the control unit, instructing the event detection unit to asynchronously request permission for outputting of event data from the timestamp unit, if the determined event generation rate is below or equal to a first readout threshold; and by the control unit, instructing the event detection unit to output event data synchronously at predetermined points in time and to add a timestamp to the event data that indicates the respective predetermined point in time, if the determined event generation rate is above the first readout threshold.

Images