Event sensor and event detecting device

Abstract

One or more event detection pixels 11A, which are any of a plurality of pixels arranged in a light receiving portion of an event sensor, each include a photodiode 21, an IV converting unit 22, an amplifier 23, an event detecting unit 24, and an output unit 25. Each event detection pixel 11A: uses one or more nearby pixels as comparison pixels, and detects a directional event related to a direction of movement in an image on a sensor light receiving surface or an edge event related to an edge in the image, on the basis of a comparison between light intensity information of the event detection pixel at a first time point and a second time point later than the first time point, and a comparison between the incident light intensity information of the event detection pixel at the first time point and the incident light intensity information of the comparison pixels at the second time point; and outputs an event signal indicating the detection of the directional event or the edge event. As a result, an event sensor is realized that is capable of detecting a direction event or an edge event in real time.

Description

Event sensor and event detection device

[0001] The present disclosure relates to an event sensor and an event detection device.

[0002] In an event sensor, which is an event-driven imaging element, when each of the pixels arranged on the light receiving surface detects a temporal change in the intensity of incident light as an event, it outputs an event signal indicating that the event has been detected.If each of the pixels does not detect an event, it does not output an event signal.

[0003] In this event sensor, only pixels that detect an event among the multiple pixels arranged on the light-receiving surface asynchronously output a position signal indicating the pixel's position and an event signal to a downstream processing unit.The processing unit then receives the pixel's position signal and event signal and can determine changes in the image of light incident on the light-receiving surface.This event sensor can reduce power consumption depending on the object being imaged.

[0004] Conventional event sensors can detect events that indicate the occurrence of motion in an image on a light-receiving surface (hereinafter referred to as "motion events"). If event sensors could detect other types of events in addition to motion events, the application fields of event sensors would be expected to expand. Examples of other types of events include directional events that indicate the direction of motion when there is motion in an image on the light-receiving surface, and edge events that indicate spatial brightness changes (edges) in an image.

[0005] The imaging device disclosed in Patent Document 1 accumulates event signals output when a motion event is detected by an event sensor at regular intervals to generate a frame sequence, and performs template matching or block matching between the frames to obtain information about the direction of motion.

[0006] Japanese Patent Application Laid-Open No. 2022-111437

[0007] The information regarding the direction of movement acquired by the imaging device disclosed in Patent Document 1 is obtained by performing required processing on a frame sequence generated by accumulating event signals at regular intervals, and is not information for each event. In other words, this imaging device does not detect directional events in real time every time an event occurs in each pixel.

[0008] Embodiments aim to provide an event sensor that can detect other types of events (directional events or edge events) in real time.

[0009] An embodiment is an event sensor that includes: (1) a light-receiving unit having a light-receiving surface on which a plurality of pixels are arranged, where an image is formed by incident light on the light-receiving surface, and at least one of the plurality of pixels is an event detection pixel; (2) a readout unit that reads out an event signal output from the light-receiving unit; and (3) a control unit that controls the operation of each of the light-receiving unit and the readout unit; (4) the event detection pixel uses one or more pixels among the plurality of pixels that are near the event detection pixel as a comparison pixel, and detects a directional event related to the direction of movement in the image or an edge event related to an edge in the image based on a comparison of incident light intensity information of the event detection pixel at a first time and a comparison of incident light intensity information of the event detection pixel at the first time with incident light intensity information of the comparison pixel at the second time, and outputs an event signal representing the detection of the directional event or the edge event.

[0010] An embodiment is an event detection device that includes the event sensor having the above configuration and an inference unit that performs inference regarding an image based on an event signal read out by a readout unit of the event sensor.

[0011] According to the event sensor and event detection device of the embodiment, a directional event or an edge event can be detected in real time.

[0012] FIG. 1 is a diagram illustrating a configuration example of an event sensor 1. FIG. 2 is a diagram illustrating a configuration example of an event detection pixel. FIG. 3 is a diagram illustrating an example of the arrangement of the event detection pixel and the comparison pixel. FIG. 4 is a diagram illustrating another example of the arrangement of the event detection pixel and the comparison pixel. FIG. 5 is a diagram illustrating a first configuration example of the event detection unit 24 of the event detection pixel 11A. FIG. 6 is a diagram illustrating a second configuration example of the event detection unit 24 of the event detection pixel 11A. FIG. 7 is a diagram illustrating another configuration example of the event detection pixel. FIG. 8 is a diagram illustrating an example of the configuration of the event detection unit 44 of the event detection pixel 11C. FIG. 9 is a diagram illustrating another configuration example of the event detection unit 24 of the event detection pixel 11A. FIG. 10 is a diagram illustrating another configuration example of the event detection unit 24 of the event detection pixel 11A. FIG. 11 is a diagram illustrating another configuration example of the event detection unit 24 of the event detection pixel 11A. FIG. 12 is a diagram illustrating another configuration example of the event detection unit 24 of the event detection pixel 11A. FIG. 13 is a diagram showing an example configuration of the selection unit 63 of the event detection unit 24G of FIG. 12 . FIG. 14 is a diagram showing another configuration of the event sensor 1. FIG. 15 is a diagram explaining an example of a use of the event sensor having the example configuration of FIG. 14 . FIG. 16 is a diagram explaining another example of a use of the event sensor having the example configuration of FIG. 14 . FIG. 17 is a diagram showing another configuration of the event sensor 1. FIG. 18 is a diagram explaining an example of a use of the event sensor having the example configuration of FIG. 17 . FIG. 19 is a diagram showing the configuration of an event detection device including the event sensor 1. FIG. 20 is a diagram showing another configuration of an event detection device including the event sensor 1.

[0013] Hereinafter, an embodiment of an event sensor and an event detection device will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicated explanations will be omitted. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0014] 1 is a diagram showing the configuration of an event sensor 1. The event sensor 1 includes a light receiving unit 10, a readout unit 12, and a control unit 13. The light receiving unit 10 has a light receiving surface on which a plurality of pixels 11 are arranged two-dimensionally (8 rows and 10 columns in the figure), and an image is formed on this light receiving surface by incident light. At least one of the plurality of pixels 11 is an event detection pixel.

[0015] The event detection pixels detect an event based on incident light intensity information and output an event signal indicating the event detection. All pixels 11 may be event detection pixels. Alternatively, for example, 3×3 pixels may be used as a unit block, and only the pixel at the center of each unit block may be an event detection pixel. The readout unit 12 reads out the event signal output from the light receiving unit 10. The control unit 13 controls the operations of the light receiving unit 10 and the readout unit 12.

[0016] 2 is a diagram showing an example of the configuration of an event detection pixel. In addition to an event detection pixel 11A, this diagram also shows a readout unit 12 and a control unit 13. The event detection pixel 11A shown in this diagram includes a photodiode 21, an IV conversion unit 22, an amplifier 23, an event detection unit 24, and an output unit 25. Note that pixels that do not detect events also include the photodiode 21, the IV conversion unit 22, and the amplifier 23.

[0017] The photodiode 21 generates electric charges in response to incident light and outputs a current signal to the IV conversion unit 22. The magnitude of the current signal output from the photodiode 21 (the amount of electric charges generated per unit time) corresponds to the intensity of light incident on the photodiode 21.

[0018] The IV conversion unit 22 receives the current signal output from the photodiode 21 and outputs a voltage signal having a value corresponding to the input current value to the amplifier 23. The amplifier 23 receives the voltage signal output from the IV conversion unit 22, amplifies the received voltage signal, and outputs the amplified voltage signal to the event detection unit 24. The voltage signal V １ The magnitude of the signal .theta.

[0019] The event detection unit 24 detects the voltage signal V output from the amplifier 23 of the pixel 11A. １ , and a voltage signal V output from the amplifier 23 of a comparison pixel located near the pixel 11A among the plurality of pixels 11. ２ These voltage signals (incident light intensity information) V １ , V ２ An event is detected in the image on the light receiving surface based on the

[0020] The comparison pixel may be a pixel adjacent to the subject pixel 11A in a direction above, below, left, right, or diagonally. One or more pixels may exist between the subject pixel 11A and the comparison pixel. The comparison pixel may be a pixel that detects an event, or a pixel that does not detect an event.

[0021] The output unit 25 is connected to the control unit 13 via a Y request (Y-Req) line and a Y acknowledge (Y-Ack) line. The output unit 25 also outputs an event signal indicating an event detection by the event detection unit 24 to the readout unit 12. The output of the event signal from the output unit 25 to the readout unit 12 is performed, for example, as follows.

[0022] The output unit 25 of the pixel 11A that detected the event sets the Y-Req line connected to that output unit 25 to High, thereby notifying the control unit 13 of an event signal output request. Based on the information on the Y-Req line that has gone High, the control unit 13 determines the Y address of the pixel 11A for which the event signal output request has been made, and sets the Y-Ack line corresponding to that Y address to High. Of the multiple pixels in the row whose Y-Ack line has gone High, the output unit 25 of the pixel 11A that has issued the event signal output request outputs an event signal to the readout unit 12 and sets the Y-Req line to Low.

[0023] In this way, even if event signal output requests are output simultaneously from multiple pixels, arbitration can be performed, making it possible to read out the event signal while avoiding collisions between event signal outputs. In addition to reading out the event signal, the readout unit 12 also acquires position information (X address and Y address) of the pixel 11A that output the event signal.

[0024] The request line and acknowledge line may be provided for each row, but may also be provided for each column or pixel, or may also be provided for each region including a plurality of pixels.

[0025] 3 is a diagram illustrating an example of the arrangement of event detection pixels and comparison pixels. This diagram shows 2×2 unit blocks, each consisting of 2×2 pixels A to D. For each pixel for which an event is to be detected, it may be determined in which direction a pixel in that pixel's direction should be used as a comparison pixel.

[0026] For example, assuming that all of pixels A to D are event detection pixels, in each unit block, the comparison pixel for event detection pixel A can be the pixel C below, the comparison pixel for event detection pixel C can be the pixel D to the right, the comparison pixel for event detection pixel D can be the pixel B above, and the comparison pixel for event detection pixel B can be the pixel A to the left. Also, there may be multiple comparison pixels for one event detection pixel.

[0027] 4 is a diagram illustrating another example of the arrangement of event detection pixels and comparison pixels. This diagram shows one unit block including 3×3 pixels A to I. In this case, for example, pixel E, which is located in the center of the 3×3 pixels, can be the only event detection pixel, and the other eight pixels A to D and F to I can be used as comparison pixels for this event detection pixel. The eight pixels A to D and F to I may be pixels that do not detect an event.

[0028] 5 is a diagram showing a first configuration example of the event detection unit 24 of the event detection pixel 11A. In addition to the event detection unit 24A of the first configuration example, this diagram also shows an output unit 25. The event detection unit 24A includes differentiators 31 and 32, comparators 33 and 34, a voltage holding unit 35, and a threshold value holding unit 36. The event detection unit 24A detects a voltage at each time t ｎ It operates as follows:

[0029] The difference calculator 31 calculates the difference at time t ｎ , the voltage V output from the amplifier 23 of the pixel 11A. １is input, and the voltage V held by the voltage holding unit 35 is input. ０ The difference calculator 31 also inputs the voltage V １ and voltage V ０ The difference ΔV １ (=V １ -V ０ ) and calculate the voltage difference ΔV １ is output to the comparator 33.

[0030] The comparator 33 detects the voltage difference ΔV output from the differentiator 31. １ The comparator 33 also receives the threshold value held by the threshold value holding unit 36. The comparator 33 then receives the voltage difference ΔV １ The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0031] The difference calculator 32 calculates the difference at time t ｎ , the voltage V output from the amplifier 23 of the comparison pixel ２ is input, and the voltage V held by the voltage holding unit 35 is input. ０ The differencer 32 also receives the voltage V ２ and voltage V ０ The difference ΔV ２ (=V ２ -V ０ ) and calculate the voltage difference ΔV ２ is output to the comparator 34.

[0032] The comparator 34 detects the voltage difference ΔV output from the differentiator 32. ２ The comparator 34 receives the voltage difference ΔV ２ The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0033] The voltage holding unit 35 ｎ－１ The voltage V output from the amplifier 23 of the pixel 11A at (first time) １ and input it at a later time t ｎ (second time) until the voltage V ０ and the voltage V ０ are respectively provided to the differentiators 31 and 32. The voltage V held by the voltage holding unit 35 is ０may be updated when the pixel is reset, when an event is detected, when the Y-Ack line goes high, etc.

[0034] The threshold value holding unit 36 ​​holds a threshold value and provides the held threshold value to each of the comparators 33 and 34. The threshold values ​​provided to each of the comparators 33 and 34 may be the same or different from each other. The threshold value may be fixed or may be changed as appropriate in response to an instruction from the control unit 13 or the like.

[0035] The threshold value held by the threshold value holding unit 36 ​​may be both a positive value and a negative value, or may be either one of the positive and negative values. １ A positive event occurs when the voltage difference ΔV exceeds the positive threshold. １ The comparator 34 may detect events by distinguishing between positive and negative events, which occur when the positive voltage Vcc falls below a negative threshold, or may detect events without distinguishing between positive and negative events.

[0036] The output unit 25 outputs the difference between the difference calculator 31 and the comparator 33 (time t ｎ－１ and time t ｎ The incident light intensity information (voltage V ０ , V １ )), a motion event relating to a movement in the image on the light receiving surface of the light receiving unit 10 can be detected.

[0037] The output unit 25 outputs the difference between the difference calculator 32 and the comparator 34 (time t ｎ－１ The incident light intensity information (voltage V ０ ) and time t ｎ Incident light intensity information (voltage V ２ Based on the result of the comparison with the target object, a directional event or an edge event can be detected. Specifically, the event detection mode includes the following modes.

[0038] The first event detection mode is a mode in which a direction event is further detected when a motion event is detected. ｎ－１ and time tｎ The incident light intensity information (voltage V ０ , V １ ) difference ΔV １ is outside the threshold range, a motion event is detected to have occurred.

[0039] The output unit 25 also calculates the time t ｎ－１ The incident light intensity information (voltage V ０ ) and time t ｎ Incident light intensity information (voltage V ２ ) and the difference ΔV ２ is within the threshold range, it is detected that a directional event has occurred in which the pixel has moved in the direction from the pixel 11A to the comparison pixel.

[0040] In the first event detection mode, when the output unit 25 detects a direction event, it outputs an event signal representing the detection of the direction event. The output unit 25 may output an event signal representing the detection of both a motion event and a direction event. Furthermore, when a motion event is detected but no direction event is detected (when the voltage difference ΔV １ is outside the threshold range and the voltage difference ΔV ２ If the detected motion event is outside the threshold range, an event signal may be output indicating only the detection of a motion event.

[0041] The second event detection mode is a mode in which an edge event is detected when no motion event is detected. ｎ－１ and time t ｎ The incident light intensity information (voltage V ０ , V １ ) difference ΔV １ is within the threshold range, it is detected that no motion event has occurred.

[0042] The output unit 25 also calculates the time t ｎ－１ The incident light intensity information (voltage V ０ ) and time t ｎIncident light intensity information (voltage V ２ ) and the difference ΔV ２ is outside the threshold range, it is detected that an edge event has occurred.

[0043] In the second event detection mode, when a motion event is not detected but an edge event is detected, the output unit 25 outputs an event signal indicating the detection of an edge event. Alternatively, when a motion event is detected, the output unit 25 may output an event signal indicating the detection of a motion event.

[0044] The third event detection mode is a mode in which, when a motion event is detected, a direction event is further detected, and when no motion event is detected, an edge event is detected. １ is outside the threshold range, it is detected that a motion event has occurred, and conversely, the voltage difference ΔV １ is within the threshold range, it is detected that no motion event has occurred.

[0045] When a motion event occurs, the output unit 25 outputs a voltage difference ΔV ２ is within the threshold range, the output unit 25 detects that a directional event has occurred, that is, a movement from the subject pixel 11A to the comparative pixel. On the other hand, when a movement event has not occurred, the output unit 25 detects that the voltage difference ΔV ２ is outside the threshold range, the output unit 25 detects that an edge event has occurred. １ is outside the threshold range and the voltage difference ΔV ２ If the detected motion event is outside the threshold range, an event signal may be output indicating only the detection of a motion event.

[0046] In the third event detection mode, when a motion event and a directional event are detected, the output unit 25 outputs an event signal indicating the detection of the directional event. When no motion event is detected but an edge event is detected, the output unit 25 outputs an event signal indicating the detection of the edge event. In the third event detection mode, the threshold for detecting a directional event and the threshold for detecting an edge event may be the same or different from each other.

[0047] In any of the first to third event detection modes, when an event signal representing multiple types of events is output from the output unit 25 to the readout unit 12, a signal line may be provided between the output unit 25 and the readout unit 12 for each type of event, or an encoded event signal may be sent from the output unit 25 to the readout unit 12.

[0048] For example, as described with reference to FIG. 4 , when comparing eight pixels adjacent to an event detection pixel above, below, left, right, and diagonally as comparison pixels, eight sets of difference calculators 32 and comparators 34 are provided corresponding to the eight comparison pixels, and directional events can be detected in each of the eight directions, i.e., above, below, left, right, and diagonally. In this case, eight signal lines may be provided between the output unit 25 and the readout unit 12 corresponding to the directional events in the eight directions, or an encoded 4-bit event signal may be sent over four signal lines, even in cases where there is no directional event.

[0049] Furthermore, if it is not necessary to distinguish between the directions of movement, fewer than eight signal lines may be provided for eight directional events. The same applies to edge events. The same also applies when two or more types of events from the output unit 25 to the readout unit 12 are sent.

[0050] In any of the first to third event detection modes, it is preferable that the output unit 25 holds the event signal after detecting the event until the Y-Ack line goes high. ０is updated when an event is detected, it is assumed that the event detection unit 24A will no longer output an event, and therefore the event signal of the output unit 25 is not updated at that time.

[0051] In any of the first to third event detection modes, the output unit 25 may have a time measurement unit that counts clocks in case a new event is detected after an event is detected and before the Y-Ack line goes high, and may also be able to hold timestamp information for each of multiple events, and output an event signal including the timestamp information.

[0052] In any of the first to third event detection modes, the output unit 25 outputs incident light intensity information (for example, a voltage V １ The event signal may include the following:

[0053] Furthermore, in any of the first to third event detection modes, the output unit 25 may output an event signal only when requested or permitted by the control unit 13. In other words, an event detection request line is connected from the control unit 13 to the output unit 25 of each pixel, and the Y request line is set to High only when the event detection request line is High and an event has occurred for each pixel.

[0054] When detecting an edge event in the second or third event detection mode, it is preferable to do the following: If there is no change in the luminance of the image on the light-receiving surface of the light-receiving unit 10, edge events may occur all the time, which increases the number of readouts and makes the process inefficient.

[0055] Therefore, after an edge event is detected and output, the voltage difference ΔV １ The Y request line may not be set to High by an edge event, and when the edge event is no longer detected, the Y request line may be set to High and a Low signal may be output to the edge event line to notify the reading unit 12 that the edge has disappeared.

[0056] Alternatively, an edge event may be detected at reset, and then only once at a certain time interval. Alternatively, an edge event may be detected only for pixels in which a motion event has been detected, and once an edge event has been detected, no edge event detection may be performed until a motion event is detected again. Alternatively, the operation of the comparator 34 may be enabled to detect an edge event when permission or a request is received from the control unit 13.

[0057] 6 is a diagram showing a second configuration example of the event detection unit 24 of the event detection pixel 11A. In addition to the event detection unit 24B of the second configuration example, the figure also shows an output unit 25. The event detection unit 24B includes comparators 37 and 38 and a threshold setting unit 39. The event detection unit 24B outputs a signal at each time t ｎ It operates as follows:

[0058] The comparator 37 detects the time t ｎ , the voltage V output from the amplifier 23 of the pixel 11A. １ The comparator 37 also inputs the threshold value set and held by the threshold value setting unit 39. １ The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0059] The comparator 38 detects the time t ｎ , the voltage V output from the amplifier 23 of the comparison pixel ２ The comparator 38 also inputs the threshold value set and held by the threshold value setting unit 39. Then, the comparator 38 outputs the voltage V ２ The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0060] The threshold setting unit 39 sets the threshold value at time t ｎ－１ The voltage V output from the amplifier 23 of the pixel 11A at (first time) １ Input the voltage V １ The threshold value set based on the ｎ The threshold is held until (second time), and the held threshold is provided to each of the comparators 37 and 38. The threshold may be updated when the pixel is reset, when an event is detected, when the Y-Ack line goes high, etc.

[0061] The threshold values ​​that the threshold setting unit 39 provides to the comparators 37 and 38 may be the same or different from each other. ｎ The threshold value output from the threshold value setting unit 39 at (second time) is the threshold value at time t ｎ－１ The voltage V output from the amplifier 23 of the pixel 11A at (first time) １ (i.e., at time t ｎ Voltage V held until (second time) ０ ) greater than V Ｈ and a small value V Ｌ The value may be both of these values, or may be either one of these values.

[0062] The comparator 37 detects the voltage V １ is the threshold V Ｈ A positive event occurs when the voltage exceeds V １ is the threshold V Ｌ The comparator 38 may detect events by distinguishing between positive and negative events and events when the output voltage Vcc falls below 10 V, or may detect events without distinguishing between positive and negative events.

[0063] The output unit 25 receives the comparison signal from the comparator 37 (at time t ｎ－１ and time t ｎ The incident light intensity information (voltage V ０ , V １ Based on the result of the comparison by the comparator 38 (time t ｎ－１ The incident light intensity information (voltage V ０ ) and time t ｎ Incident light intensity information (voltage V ２ A directional event or an edge event can be detected based on the result of the comparison with the first to third event detection modes.

[0064] The event detection pixels described above using FIGS. 2 to 6 convert the current signal output from the photodiode 21 into a voltage signal using the IV converter 22 and the amplifier 23, and detect an event by using the magnitude (voltage value) of the voltage signal as incident light intensity information.

[0065] Alternatively, as will be described below with reference to Figures 7 and 8, the event detection pixel may count the pulses output when the photodiode detects a photon, and detect an event by using the time it takes for the pulse count value to reach a predetermined value as incident light intensity information. In this case, the shorter the time it takes for the pulse count value to reach the predetermined value, the greater the incident light intensity.

[0066] 7 is a diagram showing another example of the configuration of an event detection pixel. This diagram shows a readout unit 12 and a control unit 13 in addition to an event detection pixel 11C. The event detection pixel 11C shown in this diagram includes an avalanche photodiode (APD) 41, a quenching element 42, a waveform shaping unit 43, an event detection unit 44, and an output unit 45. It should be noted that pixels that do not detect events also include the APD 41, the quenching element 42, and the waveform shaping unit 43.

[0067] A voltage V is applied to the anode of the APD 41. ＤＤ is applied to the cathode of the APD 41 via the quench element 42, and a voltage V ｂｄ is applied between the anode and cathode. A voltage greater than the breakdown voltage of the APD 41 is applied between the anode and cathode. The APD 41 avalanche-multiplies electrons generated by the incidence of a single photon, allowing a large current (avalanche current) to flow. The quench element 42 converts the avalanche current into a voltage signal, discharges the electrons generated and accumulated by the avalanche multiplication, and returns the voltage to the initial voltage.

[0068] The waveform shaping unit 43 receives the voltage at the connection point between the APD 41 and the quench element 42, shapes the waveform of the pulse generated by the incidence of one photon on the APD 41, and outputs the shaped pulse P １ to the event detection unit 44. The waveform shaping unit 43 may be configured to include, for example, an inverter and a buffer.

[0069] The event detection unit 44 detects the pulse P output from the waveform shaping unit 43 of the pixel 11C. １ and a pulse P output from the waveform shaping unit 43 of a comparison pixel located near the pixel 11C among the plurality of pixels 11. ２ are also input, and these pulses P １ , P ２ An event is detected in the image on the light receiving surface based on each counting result (incident light intensity information). The output unit 45 is the same as the output unit 25 described above.

[0070] 8 is a diagram showing an example of the configuration of the event detection unit 44 of the event detection pixel 11C. This diagram shows an output unit 45 in addition to the event detection unit 44. The event detection unit 44 includes pulse counting units 51 and 52, time measurement units 53 and 54, a determination unit 55, comparators 56 and 57, and a threshold setting unit 58.

[0071] The pulse counter 51 counts the pulse P output from the waveform shaping unit 43 of the pixel 11C. １ The pulse counter 52 receives the pulse P output from the waveform shaping unit 43 of the comparison pixel and counts it to count the photons incident on the APD 41 of the pixel 11C. ２ The time measurement units 53 and 54 count clock pulses of a clock with a constant frequency (for example, a clock provided from outside the sensor) and count the number of photons incident on the APD 41 of the comparison pixel.

[0072] The determination unit 55 instructs the initialization of the count values ​​of the pulse counters 51 and 52 and the time measurement units 53 and 54. After the initialization, the pulse counters 51 and 52 and the time measurement units 53 and 54 start counting operations.

[0073] The determination unit 55 receives the count values ​​of the pulse counting units 51 and 52 and the time measuring units 53 and 54. The determination unit 55 receives the count value C of the time measuring unit 53 when the count value of the pulse counting unit 51 reaches a predetermined value. １ The count value C of the time measurement unit 54 when the count value of the pulse counting unit 52 reaches a predetermined value is acquired. ２After the count values ​​of both pulse counters 51 and 52 reach a predetermined value, determination unit 55 instructs pulse counters 51 and 52 and time measurement units 53 and 54 to initialize their respective count values.

[0074] The comparator 56 receives the count value C １ The comparator 56 also inputs the threshold value set and held by the threshold setting unit 58. Then, the comparator 56 receives the count value C １ The output unit 45 is notified of the comparison result.

[0075] The comparator 57 receives the count value C ２ The comparator 57 also inputs the threshold value set and held by the threshold setting unit 58. Then, the comparator 57 receives the count value C ２ The output unit 45 is notified of the comparison result.

[0076] The threshold setting unit 58 receives the count value C １ Enter the count value C ０ and the count value C ０ The threshold value set based on the count value C is also stored, and the stored threshold value is provided to the comparators 56 and 57. ０ may be updated when the pixel is reset, when an event is detected, when the Y-Ack line goes high, etc.

[0077] The thresholds that the threshold setting unit 58 gives to the comparators 56 and 57 may be the same or different from each other. The thresholds output from the threshold setting unit 58 are calculated based on the count value C ０ Larger value C Ｈ and a small value C Ｌ The value may be both of these values, or may be either one of these values.

[0078] The comparator 56 calculates the count value C １ is the threshold C Ｈ A positive event occurs when the count value C exceeds １ is the threshold C ＬThe comparator 57 may detect events by distinguishing between positive and negative events, or may detect events without distinguishing between positive and negative events.

[0079] The output unit 45 receives the comparison by the comparator 56 (information on the incident light intensity of the pixel 11C (count value C ０ , C １ Based on the result of the comparison by the comparator 57 (the incident light intensity information (count value C ０ ) and the incident light intensity information of the comparison pixel (count value C ２ A directional event or an edge event can be detected based on the result of the comparison with the first to third event detection modes.

[0080] The configuration example of the event detection unit 44 shown in FIG. ０ The threshold and count value C set based on １ , C ２ 5. The event detection unit 44 is configured to compare the count value C ０ and count value C １ , C ２ The difference may be calculated and compared with a threshold value.

[0081] There are various configurations as modified examples. In any of the configurations, the event detection unit may use either a voltage value or a count value as incident light intensity information, and may correspond to either the first or second configuration example. The configuration example described below uses a voltage value as incident light intensity information and corresponds to the first configuration example. Event sensors of any of the configurations can detect not only motion events but also other types of events (directional events or edge events) in real time.

[0082] Fig. 9 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24A shown in Fig. 5, the event detection unit 24D shown in Fig. 9 differs in that it is provided with multiple sets (two sets in this figure) of differentiators 32 and comparators 34 for comparing the incident light intensities between the subject pixel 11A and multiple comparison pixels.

[0083] Differentiator 32 １ is the time t ｎ , the voltage V output from the amplifier 23 of the first comparison pixel ２１ is input, and the voltage V held by the voltage holding unit 35 is input. ０ is also input. １ is the voltage V ２１ and voltage V ０ The difference ΔV ２１ (=V ２１ -V ０ ) and calculate the voltage difference ΔV ２１ is input to comparator 34. １ Output to.

[0084] Comparator 34 １ is the difference calculator 32 １ The voltage difference ΔV output from ２１ The comparator 34 also inputs the threshold value held by the threshold value holding unit 36. １ is the voltage difference ΔV ２１ The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0085] Differentiator 32 ２ is the time t ｎ , the voltage V output from the amplifier 23 of the second comparison pixel ２２ is input, and the voltage V held by the voltage holding unit 35 is input. ０ is also input. ２ is the voltage V ２２ and voltage V ０ The difference ΔV ２２ (=V ２２ -V ０ ) and calculate the voltage difference ΔV ２２ is input to comparator 34. ２ Output to.

[0086] Comparator 34 ２ is the difference calculator 32 ２ The voltage difference ΔV output from ２２ The comparator 34 also inputs the threshold value held by the threshold value holding unit 36. ２ is the voltage difference ΔV ２２ The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0087] The output unit 25 can detect a more detailed directional event or edge event based on the comparison results for each of the plurality of comparison pixels. The output unit 25 may output an event signal for each comparison pixel, or may output an event signal that represents the most likely directional event or edge event.

[0088] Fig. 10 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24A shown in Fig. 5, the event detection unit 24E shown in Fig. 10 differs in that it further includes a selection unit 61.

[0089] The selection unit 61 selects the voltage V output from the amplifier 23 of each of a plurality of comparison pixels (four in this figure). ２１ ~V ２４ Input the voltage V ２１ ~V ２４ The selector 61 sequentially selects one of the voltages V ２１ ~V ２４ and the held voltage V ２１ ~V ２４ The voltage V ２１ ~V ２４ The selection may be made by the selection unit 61 itself or in response to an instruction from the control unit 13.

[0090] The difference calculator 32 calculates the voltage V ２１ ~V ２４ The voltage V held by the voltage holding unit 35 is input. ０The difference calculator 32 then calculates the difference between the two input voltages and calculates the voltage difference ΔV ２ is output to the comparator 34.

[0091] The output unit 25 can detect a more detailed directional event or edge event based on the comparison results for each of the multiple comparison pixels. The output unit 25 may output an event signal for each comparison pixel, or may output an event signal representing the most likely directional event or edge event. The output unit 25 may also output an event signal including position information of the comparison pixel (information indicating the direction of the comparison pixel relative to the target pixel 11A).

[0092] Fig. 11 is a diagram showing another example configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24E shown in Fig. 10, the event detection unit 24F shown in Fig. 11 differs in that it does not include the differentiator 31 and the comparator 33, and that it includes a selection unit 62 instead of the selection unit 61.

[0093] The selection unit 62 selects the voltage V output from the amplifier 23 of the pixel 11A. １ is input, and the voltage V output from the amplifier 23 of each of the plurality of comparison pixels (four in this figure) is input. ２１ ~V ２４ Then, the selection unit 62 inputs the voltage V １ , V ２１ ~V ２４ The selector 62 sequentially selects one of the voltages V １ , V ２１ ~V ２４ and the held voltage V １ , V ２１ ~V ２４ The voltage V １ , V ２１ ~V ２４ The selection may be made by the selection unit 62 itself or in response to an instruction from the control unit 13.

[0094] The difference calculator 32 calculates the voltage V １ , V ２１~V ２４ The voltage V held by the voltage holding unit 35 is input. ０ The differentiator 32 then finds the difference between the two input voltages and outputs this voltage difference ΔV to the comparator 34.

[0095] The output section 25 receives the voltage V output from the amplifier 23 of the pixel 11A. １ is selected by the selection unit 62. The output unit 25 outputs the voltage V ２１ ~V ２４ When either of these is selected by the selection unit 62, a directional event or an edge event can be detected.

[0096] Voltage V １ When a motion event is detected while the voltage V is selected, the voltage holding unit 35 is not updated immediately. ２１ ~V ２４ The output unit 25 may output an event signal including information indicating which of the subject pixel 11A and the plurality of comparison pixels has been selected by the selector 62.

[0097] Fig. 12 is a diagram showing another example configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24A shown in Fig. 5, the event detection unit 24G shown in Fig. 12 differs in that it includes multiple (two in this figure) differentiators 32 and further includes a selection unit 63.

[0098] Differentiator 32 １ is the time t ｎ , the voltage V output from the amplifier 23 of the first comparison pixel ２１ is input, and the voltage V held by the voltage holding unit 35 is input. ０ is also input. １ is the voltage V ２１ and voltage V ０ The difference ΔV ２１ (=V ２１ -V ０) and calculate the voltage difference ΔV ２１ to the selection unit 63.

[0099] Differentiator 32 ２ is the time t ｎ , the voltage V output from the amplifier 23 of the second comparison pixel ２２ is input, and the voltage V held by the voltage holding unit 35 is input. ０ is also input. ２ is the voltage V ２２ and voltage V ０ The difference ΔV ２２ (=V ２２ -V ０ ) and calculate the voltage difference ΔV ２２ to the selection unit 63.

[0100] The selector 63 selects the difference calculator 32 １ The voltage difference ΔV output from ２１ is input, and the difference calculator 32 ２ The voltage difference ΔV output from ２２ Then, the selector 63 inputs the voltage difference ΔV ２１ , ΔV ２２ and sequentially select one of the voltage differences ΔV ２ is output to the comparator 34. The voltage difference ΔV ２１ , ΔV ２２ The selection may be made by the selection unit 63 itself or in response to an instruction from the control unit 13.

[0101] The output unit 25 can detect a more detailed directional event or edge event based on the comparison results for each of the multiple comparison pixels. The output unit 25 may output an event signal for each comparison pixel, or may output an event signal representing the most likely directional event or edge event. The output unit 25 may also output an event signal including position information of the comparison pixel (information indicating the direction of the comparison pixel relative to the target pixel 11A).

[0102] The selector 63 selects the voltage difference ΔV output from the differentiator 31. １ Also input the voltage difference ΔV １ , ΔV ２１ , ΔV ２２may be selected one by one in sequence and the selected voltage difference may be output to the comparator 34. In this case, the comparator 33 is not required. １ When a motion event is detected while the voltage holding unit 35 is selected, the voltage difference ΔV ２１ , ΔV ２２ Directional events can be detected by selecting one of the following.

[0103] The selector 63 also selects the voltage difference ΔV ２１ , ΔV ２２ , as will be explained with reference to FIG. 13, the voltage difference ΔV ２１ , ΔV ２２ The voltage difference with the smallest or largest absolute value may be selected.

[0104] 13 is a diagram showing an example of the configuration of the selection unit 63 of the event detection unit 24G in FIG. 12. In this diagram, the voltages V ２１ ~V ２４ and V ０ Voltage difference ΔV ２１ ~ΔV ２４ is input to the selection unit 63. The four comparison pixels are, for example, four pixels adjacent to the top, bottom, left, and right of the event detection unit 24G. The selection unit 63 shown in this figure includes comparators 71 to 73 and selectors 74 to 76.

[0105] The comparator 71 and the selector 74 each detect a voltage difference ΔV ２１ (=V ２１ -V ０ ) and voltage difference ΔV ２２ (=V ２２ -V ０ The selector 74 inputs the voltage difference ΔV ２１ , ΔV ２２ The voltage difference judged by the comparator 71 to have the smaller absolute value is selected and output.

[0106] The comparator 72 and the selector 75 each detect a voltage difference ΔV ２３ (=V ２３ -V ０ ) and voltage difference ΔV ２４ (=V２４ -V ０ The selector 75 inputs the voltage difference ΔV ２３ , ΔV ２４ The voltage difference judged by the comparator 72 to have the smaller absolute value is selected and output.

[0107] The comparator 73 and the selector 76 each receive the voltage difference output from the selectors 74 and 75. The selector 76 selects the voltage difference determined by the comparator 73 to have the smaller absolute value of the two input voltage differences, and outputs the selected voltage difference to the comparator 73.

[0108] The results of the comparisons made by the comparators 71 to 73 are sent to the output unit 25. The voltage difference ΔV output from the selector 76 to the comparator 34 ２ is the voltage difference ΔV ２１ ~ΔV ２４ This allows the output unit 25 to grasp the direction of the movement in detail based on the comparison result by the comparator 34 when a movement event is detected based on the comparison result by the comparator 33.

[0109] In this figure, four comparison pixels are assumed to be adjacent in the vertical and horizontal directions of the event detection unit 24G, but even if eight comparison pixels are assumed to be adjacent in the vertical and horizontal and diagonal directions of the event detection unit 24G, eight voltage differences ΔV ２１ ~ΔV ２８ The voltage difference with the smallest absolute value is selected and input to the comparator 34.

[0110] Conversely, depending on the selection method of each of the selectors 74 to 76, the voltage difference ΔV input from the selector 76 to the comparator 34 ２ is the voltage difference ΔV ２１ ~ΔV ２４ In this case, when the output unit 25 does not detect a motion event based on the comparison result by the comparator 33, the output unit 25 can grasp the edge direction in detail based on the comparison result by the comparator 34.

[0111] Fig. 14 is a diagram showing another configuration of the event sensor 1. Compared to the configuration shown in Fig. 2, the configuration shown in Fig. 14 differs in that a mask signal is provided from the control unit 13 to the output unit 25 of the event detection pixel 11A.

[0112] The mask signal permits or prohibits the output of an event signal from the event detection pixels 11A. The permitting or prohibiting of the output of an event signal may be performed for each individual event detection pixel 11A, or for each region, row, or column. Furthermore, the permitting or prohibiting of the output of an event signal may be performed for each direction of movement in the case of a directional event, or for each edge direction in the case of an edge event.

[0113] Based on the mask signal given from the control unit 13, the output unit 25 of the event detection pixel 11A outputs an event signal to the readout unit 12 for permitted events, and does not set the Y request line high and does not output an event signal for prohibited events.

[0114] Generally, an event sensor detects a motion event not only when a subject is moving but also when the subject is stationary and the event sensor is moving. In particular, the device described in Patent Document 1 cannot extract a specific event only after performing required processing on the event signal output from the event sensor. Therefore, even if the necessity of extraction for each event is determined in advance, the device outputs an event signal for events that do not require extraction, which is inefficient.

[0115] In contrast to this, in the present configuration, events that are desired to be extracted can be selectively output from the output unit 25, which is efficient.

[0116] In the above description, the mask signal is provided from the control unit 13 to the event detection pixel 11A, but the mask signal may be provided from the control unit 13 to the readout unit 12. In the latter case, the readout unit 12 outputs an event signal to the outside for a permitted event, and does not output an event signal to the outside for a prohibited event.

[0117] FIG. 15 is a diagram illustrating an example of an application of the event sensor of the configuration example of FIG. 14 . This diagram shows a schematic image formed on the light receiving surface of an event sensor mounted on an automobile, of an object located ahead of the automobile in the traveling direction. The image shows the objects as other automobiles parked on the road, roadside trees off the road, and a pedestrian about to cross the road from left to right. When an automobile equipped with an event sensor is traveling, it is highly important to detect other automobiles and pedestrians on the road, but it is less important to detect roadside trees.

[0118] If the event sensor installed in the vehicle is a conventional one, it will detect movement events, regardless of their importance, for other vehicles on the road, roadside trees outside the road, and pedestrians in the image formed on the light receiving surface of the event sensor.

[0119] In contrast, if the event sensor mounted on a vehicle is the type of this embodiment, in the image formed on the light-receiving surface of the event sensor, a downward direction event will be generated for other vehicles on the road and roadside trees off the road, and a rightward or downward-right direction event will be generated for pedestrians. The event sensor of the configuration example in Figure 14 can be set to allow or prohibit the output of an event signal depending on the area of ​​the image (the area corresponding to the road, the area off the road) and the direction of movement.

[0120] Specifically, in the region of the image formed on the light-receiving surface of the event sensor that corresponds to the left side of the road, the output of an event signal for a rightward direction event is permitted and the output of event signals for direction events in other directions is prohibited. In the region of the image formed on the light-receiving surface of the event sensor that corresponds to the right side of the road, the output of an event signal for a leftward direction event is permitted and the output of event signals for direction events in other directions is prohibited. In the region of the image formed on the light-receiving surface of the event sensor that corresponds to the road, the output of an event signal for a downward direction event is permitted and the output of event signals for direction events in other directions is prohibited.

[0121] Information about the movement of the event sensor may be received from an external source (for example, the vehicle or a passenger) or from a sensor that can obtain the direction of movement, such as a gyro sensor. This information may be used to enable or disable the output of an event signal.

[0122] In this way, in the event sensor of the configuration example of Figure 14, by setting the output of an event signal to be allowed or prohibited depending on the level of importance, it becomes possible to detect and recognize important events quickly and efficiently, and subsequent responses can be made quickly.

[0123] Fig. 16 is a diagram illustrating another example of the use of the event sensor of the configuration example of Fig. 14. This diagram shows a schematic image of a subject in front of the robot, formed on the light receiving surface of an event sensor mounted on a robot. The image shows a person's hand waving up and down and a walking person as the subject. The robot equipped with the event sensor recognizes the hand waving up and down as a signal to cancel its standby state and move on to the next action.

[0124] If the event sensor installed on the robot is a conventional one, the up and down movement that signals the end of the standby state cannot be extracted until the required processing is performed on the event signal output from the event sensor.

[0125] In contrast, if the event sensor mounted on the robot is the one of this embodiment, a directional event of up or down will be generated for a person's hand waving up and down in the image formed on the light-receiving surface of the event sensor, and a directional event of right or left will be generated for a person walking. The event sensor of the configuration example in Figure 14 can be set to allow or prohibit the output of an event signal depending on the direction of movement.

[0126] Specifically, in the image formed on the light receiving surface of the event sensor, the output of an event signal for a directional event in the upward or downward direction is permitted, and the output of an event signal for a directional event in any other direction is prohibited. By setting it in this way, it is possible to prevent the robot from misinterpreting a signal due to a person or object moving left or right.

[0127] 15 and 16, the event sensor of the configuration example of Fig. 14 can output event signals less frequently than conventional event sensors, thereby reducing power consumption. Furthermore, by detecting and using edge events in addition to directional events, the accuracy of event recognition can be improved.

[0128] Fig. 17 is a diagram showing another configuration of the event sensor 1. Compared to the configuration shown in Fig. 2, the configuration shown in Fig. 17 differs in that it further includes a histogram generation unit 14. The histogram generation unit 14 counts the number of detected events for each event type based on the event signals output from each event detection pixel 11A of the light receiving unit 10, generates a histogram, and outputs this histogrammed event signal to the readout unit 12.

[0129] For example, the histogram generator 14 includes a counter provided for each p×q pixel. Each counter counts directional events for each direction of movement over a certain period of time based on the event signals output from the corresponding p×q pixel, and generates a histogram. This reduces the number of reads performed by the reader 12, thereby reducing the burden on subsequent processing.

[0130] Furthermore, based on this histogram, tasks such as classification and identification can be directly performed using machine learning, such as neural networks, reservoirs, Histogram of Optical Flow (HOF), and Motion Boundary Histograms (MBH).

[0131] A counter may be provided for each event type. Alternatively, a counter may be provided in common for both upward and downward direction events, in which case the count value is decremented by 1 for an upward direction event and incremented by 1 for a downward direction event.

[0132] The histogram generator 14 may include multiple counters, each corresponding to p×q pixels, that perform different encoding. The encoding here refers to, for example, determining that for a given pixel, counter A and counter B count the number of events, while counter C does not count the number of events. Weighted encoding may also be used. This makes it possible to restore the directional events of each pixel using numerical calculations such as compressed sensing based on the histogram.

[0133] In the histogram generation unit 14, one or more counters may be provided for each p×q pixel, or one pixel may be connected to multiple counters. The p×q pixel area may partially overlap with other areas. The combination of which pixels are connected to which counters may be arbitrarily selected. One counter may be provided for all pixels.

[0134] The histogram generation unit 14 may output only the most frequent direction of the counter, only the directions with higher count values, or only these counter values.The histogram generation unit 14 may output only the directions whose counter values ​​exceed a predetermined threshold, only the most frequent direction among them, or only these count values.The histogram generation unit 14 may use the histogram values ​​as weights to calculate the directions of the entire p×q pixel region (by averaging the directions) and output them.

[0135] Fig. 18 is a diagram illustrating an example of the use of the event sensor having the configuration example of Fig. 17. In this diagram, two unit blocks, each consisting of 4 x 4 pixels, are shown, and directional events detected at each pixel during a certain period are indicated by hatched arrows.

[0136] In the left half unit block 1, three pixels detect an upward direction event, one pixel detects a downward direction event, four pixels detect a leftward direction event, and one pixel detects a rightward direction event. In this case, the direction weights are -2 for the up-down direction and -3 for the left-right direction, resulting in an overall upward-left direction. The most frequent value is the leftward direction.

[0137] In the right half unit block 2, one pixel detects an upward direction event, three pixels detect a downward direction event, one pixel detects a leftward direction event, and three pixels detect a rightward direction event. In this case, the direction weights are +2 for the up / down direction and +2 for the left / right direction, resulting in a downward / rightward direction overall. The most frequent values ​​are the downward and rightward directions.

[0138] 19 is a diagram showing the configuration of an event detection device including an event sensor 1. The event detection device shown in this figure includes an inference unit 2 in addition to the event sensor 1. The inference unit 2 performs inference regarding an image formed on the light-receiving surface of the light-receiving unit 10 based on an event signal read out by a readout unit 12 of the event sensor 1. The inference unit 2 performs tasks such as recognition, classification, and prediction using an inference model such as machine learning or a rule-based algorithm based on the event signal that includes information on any of a motion event, a directional event, and an edge event.

[0139] The inference unit 2 accumulates one or more types of information included in the event signal for a certain period of time and inputs the accumulated information as a single image with each pixel having the information to an inference model, etc. Examples of inference models include a convolutional neural network (CNN) and a transformer model that receive non-time-series data as input, and a recurrent neural network (RNN) and a reservoir computing (RC) that receive time-series data as input. The transformer model can also handle time-series data.

[0140] In these inference models, an inference result may be obtained for each image, or the most frequent result may be obtained from the inference results of multiple images. In particular, in the case of a time series inference model, the inference result after multiple images are input may be used.

[0141] Each time an event signal is output, the inference unit 2 inputs the information contained in the event signal into a time series inference model, and may obtain an inference result for each input, may obtain the most frequent value among multiple inference results, or may obtain an inference result after multiple pieces of information have been input.

[0142] 20 is a diagram showing another configuration of an event detection device including an event sensor 1. The event detection device shown in this figure includes an inference unit 2 and a movement detection unit 3 in addition to the event sensor 1.

[0143] In this configuration, the motion detection unit 3 acquires motion information (such as what kind of motion occurred at what position and in what direction) based on an event signal containing information on a motion event, a direction event, or an edge event. The inference unit 2 performs more detailed recognition and classification based on the motion information and information on edge events acquired by the motion detection unit 3.

[0144] The event sensor and the event detection device are not limited to the above-described embodiment and configuration examples, and various modifications are possible.

[0145] The event sensor of the first aspect according to the above embodiment includes: (1) a light receiving unit having a light receiving surface on which a plurality of pixels are arranged, where an image is formed by the incidence of light on this light receiving surface, and any one or more of the plurality of pixels is an event detection pixel; (2) a readout unit that reads out an event signal output from the light receiving unit; and (3) a control unit that controls the operation of each of the light receiving unit and the readout unit; and (4) the event detection pixel uses one or more pixels among the plurality of pixels that are in the vicinity of the event detection pixel as a comparison pixel, and detects a directional event related to the direction of movement in the image or an edge event related to an edge in the image based on a comparison of incident light intensity information of the event detection pixel at a first time and a second time after the first time, and a comparison of the incident light intensity information of the event detection pixel at the first time with the incident light intensity information of the comparison pixel at the second time, and outputs an event signal representing the detection of the directional event or the edge event.

[0146] In the event sensor of the second aspect, in the configuration of the first aspect, the event detection pixel may be configured to detect a directional event when a motion event indicating that motion has occurred in the image is detected based on a comparison of incident light intensity information of the event detection pixel at the first time and the second time.

[0147] In the event sensor of the third aspect, in the configuration of the first or second aspect, the event detection pixel may be configured to detect an edge event when it does not detect a motion event indicating that motion has occurred in the image based on a comparison of incident light intensity information of the event detection pixel at the first time and the second time.

[0148] In the event sensor of the fourth aspect, in the configuration of any of the first to third aspects, the event detection pixel may be configured to select a comparison pixel from a plurality of comparison pixels and detect a directional event or an edge event based on incident light intensity information of the selected comparison pixel.

[0149] In the event sensor of the fifth aspect, in the configuration of any one of the first to fourth aspects, when the event detection pixels detect multiple types of events, they may be configured to output event signals without distinguishing between these multiple types of events.

[0150] In the event sensor of the sixth aspect, in the configuration of any one of the first to fifth aspects, the event detection pixels may be configured to output coded event signals when detecting multiple types of events.

[0151] In the event sensor of the seventh aspect, in the configuration of any one of the first to sixth aspects, the control unit may be configured to permit or prohibit the output of the event signal from the event detection pixel or the readout of the event signal by the readout unit.

[0152] The event sensor of the eighth aspect may be configured in any of the configurations of the first to seventh aspects, further comprising a histogram generation unit that counts the number of detected events for each event type based on the event signal output from the light receiving unit, generates a histogram, and outputs this histogrammed event signal to the readout unit.

[0153] The event detection device according to the above embodiment includes the event sensor having the above configuration, and an inference unit that performs inference regarding an image based on the event signal read out by the readout unit of the event sensor.

[0154] Embodiments can be used as event sensors and event detectors that can detect other types of events (directional events or edge events) in real time.

[0155] 1...Event sensor, 2...Inference unit, 3...Motion detection unit, 10...Light receiving unit, 11...Pixel, 11A, 11C...Event detection pixel, 12...Readout unit, 13...Control unit, 14...Histogram generation unit, 21...Photodiode, 22...IV conversion unit, 23...Amplifier, 24, 24A to 24G...Event detection unit, 25...Output unit, 31, 32, 32 １ , 32 ２ ...Differentiator, 33, 34, 34 １ , 34 ２...Comparator, 35...Voltage holding unit, 36...Threshold value holding unit, 37, 38...Comparator, 39...Threshold value setting unit, 41...Avalanche photodiode (APD), 42...Quench element, 43...Waveform shaping unit, 44...Event detection unit, 45...Output unit, 51, 52...Pulse counting unit, 53, 54...Time measurement unit, 55...Determination unit, 56, 57...Comparator, 58...Threshold value setting unit, 61 to 63...Selection unit, 71 to 73...Comparators, 74 to 76...Selectors.

Claims

1. An event sensor comprising: a light receiving unit having a light receiving surface on which a plurality of pixels are arranged, wherein an image is formed by incident light on this light receiving surface, and at least one of the plurality of pixels being an event detection pixel; a readout unit that reads out an event signal output from the light receiving unit; and a control unit that controls the operation of the light receiving unit and the readout unit, wherein the event detection pixel uses one or more of the plurality of pixels that are in the vicinity of the event detection pixel as a comparison pixel, and detects a directional event related to the direction of movement in the image or an edge event related to an edge in the image based on a comparison of incident light intensity information of the event detection pixel at a first time and a second time that is later than the first time, and a comparison of the incident light intensity information of the event detection pixel at the first time with the incident light intensity information of the comparison pixel at the second time, and outputs an event signal representing the detection of the directional event or the edge event.

2. The event sensor of claim 1, wherein the event detection pixel detects the directional event when it detects a motion event indicating that a movement has occurred in the image based on a comparison of incident light intensity information of the event detection pixel at the first time and the second time.

3. An event sensor as described in claim 1 or 2, wherein the event detection pixel detects the edge event when it does not detect a motion event indicating that a movement has occurred in the image based on a comparison of incident light intensity information of the event detection pixel at the first time and the second time.

4. An event sensor according to any one of claims 1 to 3, wherein the event detection pixel selects one of the plurality of comparison pixels and detects the directional event or the edge event based on incident light intensity information of the selected comparison pixel.

5. The event sensor according to any one of claims 1 to 4, wherein when the event detection pixels detect multiple types of events, they output event signals without distinguishing between these multiple types of events.

6. The event sensor according to any one of claims 1 to 5, wherein the event detection pixels output coded event signals when detecting multiple types of events.

7. The event sensor according to any one of claims 1 to 6, wherein the control unit permits or prohibits the output of an event signal from the event detection pixel or the reading out of the event signal by the readout unit.

8. An event sensor as claimed in any one of claims 1 to 7, further comprising a histogram generation unit that counts the number of detected events for each event type based on the event signal output from the light receiving unit, generates a histogram, and outputs this histogrammed event signal to the readout unit.

9. An event detection device comprising: an event sensor according to any one of claims 1 to 8; and an inference unit that performs inference regarding the image based on an event signal read out by the readout unit of the event sensor.

Images