Event sensor and event detection device

Abstract

To provide an event sensor capable of detecting a direction event or an edge event in real time.SOLUTION: Any one or more event detection pixels 11A among the plurality of pixels arranged in the light receiving unit of the event sensor include a photodiode 21, an IV conversion unit 22, an amplifier 23, an event detection unit 24, and an outputting unit 25. The event detection pixel 11A sets one or more pixels in the vicinity as a comparison pixel, and performs, based on comparison of incident light intensity information of the event detection pixel at each of a first time and a second time after the first time and comparison of incident light intensity information of the event detection pixel at the first time and incident light intensity information of the comparison pixel at the second time, A direction event related to a direction of movement in an image on a sensor light receiving surface or an edge event related to an edge in the image is detected, and an event signal indicating detection of the direction event or the edge event is output.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an event sensor and an event detection device. [Background technology]

[0002] In an event sensor, which is an event-driven imaging element, when each of the multiple pixels arranged on the light-receiving surface detects a temporal change in the intensity of incident light as an event, it outputs an event signal indicating the detection of the event. Each of the multiple pixels does not output an event signal unless it detects an event. In this event sensor, only the pixel among the multiple pixels arranged on the light-receiving surface that detects an event asynchronously outputs a position signal indicating the pixel's position and an event signal to a downstream processing unit. The processing unit then receives the pixel's position signal and event signal and can determine the change in the image of the light incident on the light-receiving surface. This event sensor can reduce power consumption depending on the object being imaged.

[0003] Conventional event sensors can detect events in which motion occurs in an image on a light-receiving surface (hereinafter referred to as "motion events"). If event sensors could detect other types of events in addition to motion events, the application fields of event sensors would be expected to expand. Examples of other types of events include directional events, which are related to the direction of motion when there is motion in an image on the light-receiving surface, and edge events, which are related to spatial brightness changes (edges) in an image.

[0004] The imaging device disclosed in Patent Document 1 accumulates event signals output when a motion event is detected by an event sensor at regular intervals to generate a frame sequence, and performs template matching or block matching between the frames to obtain information about the direction of motion. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-111437 Summary of the Invention [Problem to be solved by the invention]

[0006] The information about the direction of movement acquired by the imaging device disclosed in Patent Document 1 is obtained by performing required processing on a frame sequence generated by accumulating event signals at regular intervals, and is not information for each event. In other words, this imaging device does not detect directional events in real time every time an event occurs in each pixel.

[0007] The present invention has been made to solve the above problems, and has an object to provide an event sensor that can detect other types of events (directional events or edge events) in real time. [Means for solving the problem]

[0008] A first aspect of the event sensor of the present invention includes: (1) a light-receiving unit having a light-receiving surface on which a plurality of pixels are arranged, where an image is formed by incident light on the light-receiving surface, and at least one of the plurality of pixels is an event detection pixel, (2) a readout unit that reads out an event signal output from the light-receiving unit, and (3) a control unit that controls the operation of each of the light-receiving unit and the readout unit. The event detection pixel uses one or more pixels among the plurality of pixels that are near the event detection pixel as comparison pixels, and detects a directional event related to the direction of movement in the image or an edge event related to an edge in the image based on a comparison of incident light intensity information of the event detection pixel at a first time and a second time after the first time, and a comparison of the incident light intensity information of the event detection pixel at the first time with the incident light intensity information of the comparison pixel at the second time, and outputs an event signal representing the detection of the directional event or the edge event.

[0009] In a second aspect of the event sensor of the present invention, in addition to the first aspect, the event detection pixel detects a directional event when it detects a motion event indicating that a motion has occurred in the image based on a comparison of incident light intensity information of the event detection pixel at a first time and a second time.

[0010] In a third aspect of the event sensor of the present invention, in addition to the first or second aspect, the event detection pixel detects an edge event when it does not detect a motion event indicating that motion has occurred in the image based on a comparison of incident light intensity information of the event detection pixel at the first time and the second time.

[0011] In a fourth aspect of the event sensor of the present invention, in addition to any one of the first to third aspects, the event detection pixel selects a comparison pixel from among a plurality of comparison pixels and detects a directional event or an edge event based on incident light intensity information of the selected comparison pixel.

[0012] In a fifth aspect of the event sensor of the present invention, in addition to any one of the first to fourth aspects, when the event detection pixels detect multiple types of events, they output event signals without distinguishing between these multiple types of events.

[0013] In a sixth aspect of the event sensor of the present invention, in addition to any one of the first to fifth aspects, the event detection pixels output coded event signals when detecting multiple types of events.

[0014] In a seventh aspect of the event sensor of the present invention, in addition to any one of the first to sixth aspects, the control unit permits or prohibits the output of the event signal from the event detection pixel or the readout of the event signal by the readout unit.

[0015] An eighth aspect of the event sensor of the present invention is, in addition to any one of the first to seventh aspects, further comprising a histogram generation unit that counts the number of detected events for each event type based on the event signal output from the light receiving unit, generates a histogram, and outputs this histogrammed event signal to the readout unit.

[0016] An event detection device of the present invention includes the above-described event sensor of the present invention, and an inference unit that performs inference regarding an image based on the event signal read out by the readout unit of the event sensor. [Effects of the Invention]

[0017] According to the present invention, directional or edge events can be detected in real time. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing the configuration of the event sensor 1. As shown in FIG. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of an event detection pixel. [Figure 3] FIG. 3 is a diagram illustrating an example of the arrangement of event detection pixels and comparison pixels. [Figure 4] FIG. 4 is a diagram illustrating another example of the arrangement of the event detection pixels and the comparison pixels. [Figure 5] FIG. 5 is a diagram showing a first configuration example of the event detection unit 24 of the event detection pixel 11A. [Figure 6] FIG. 6 is a diagram showing a second configuration example of the event detection unit 24 of the event detection pixel 11A. [Figure 7] FIG. 7 is a diagram showing another example of the configuration of the event detection pixel. [Figure 8] FIG. 8 is a diagram showing an example of the configuration of the event detection unit 44 of the event detection pixel 11C. [Figure 9] FIG. 9 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. [Figure 10]FIG. 10 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. [Figure 11] FIG. 11 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. [Figure 12] FIG. 12 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. [Figure 13] FIG. 13 is a diagram showing an example of the configuration of the selection unit 63 of the event detection unit 24G in FIG. [Figure 14] FIG. 14 is a diagram showing another configuration of the event sensor 1. In FIG. [Figure 15] FIG. 15 is a diagram illustrating an example of the use of the event sensor having the configuration example of FIG. [Figure 16] FIG. 16 is a diagram illustrating another example of the use of the event sensor having the configuration example of FIG. [Figure 17] FIG. 17 is a diagram showing another configuration of the event sensor 1. In FIG. [Figure 18] FIG. 18 is a diagram illustrating an example of the use of the event sensor having the configuration example of FIG. [Figure 19] FIG. 19 is a diagram showing the configuration of an event detection device including the event sensor 1. As shown in FIG. [Figure 20] FIG. 20 is a diagram showing another configuration of an event detection device including the event sensor 1. In FIG. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate explanations will be omitted. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0020] FIG. 1 is a diagram showing the configuration of an event sensor 1. The event sensor 1 includes a light receiving unit 10, a readout unit 12, and a control unit 13. The light receiving unit 10 has a light receiving surface on which a plurality of pixels 11 are arranged two-dimensionally (8 rows and 10 columns in the figure), and an image is formed on this light receiving surface by incident light. One or more of the plurality of pixels 11 are event detection pixels. The event detection pixels detect an event based on incident light intensity information and output an event signal representing the event detection. All of the pixels 11 may be event detection pixels. Alternatively, for example, 3×3 pixels may be defined as a unit block, and only the pixel at the center of each unit block may be an event detection pixel. The readout unit 12 reads out the event signal output from the light receiving unit 10. The control unit 13 controls the operation of both the light receiving unit 10 and the readout unit 12.

[0021] 2 is a diagram showing an example of the configuration of an event detection pixel. In addition to an event detection pixel 11A, this diagram also shows a readout unit 12 and a control unit 13. The event detection pixel 11A shown in this diagram includes a photodiode 21, an IV conversion unit 22, an amplifier 23, an event detection unit 24, and an output unit 25. Note that pixels that do not detect events also include a photodiode 21, an IV conversion unit 22, and an amplifier 23.

[0022] The photodiode 21 generates charges in response to incident light and outputs a current signal to the IV conversion unit 22. The magnitude of the current signal output from the photodiode 21 (the amount of charge generated per unit time) corresponds to the intensity of light incident on the photodiode 21. The IV conversion unit 22 receives the current signal output from the photodiode 21 and outputs a voltage signal having a value corresponding to the input current value to the amplifier 23. The amplifier 23 receives the voltage signal output from the IV conversion unit 22, amplifies the received voltage signal, and outputs the amplified voltage signal to the event detection unit 24. The magnitude of the voltage signal V1 output from the amplifier 23 corresponds to the intensity of light incident on the photodiode 21.

[0023] The event detection unit 24 receives a voltage signal V1 output from the amplifier 23 of the subject pixel 11A, as well as a voltage signal V2 output from the amplifier 23 of a comparison pixel located near the subject pixel 11A among the multiple pixels 11, and detects an event in the image on the light receiving surface based on these voltage signals (incident light intensity information) V1 and V2. The comparison pixel may be a pixel adjacent to the subject pixel 11A in a direction above, below, left, right, or diagonally. One or more pixels may exist between the subject pixel 11A and the comparison pixel. The comparison pixel may be a pixel that detects an event, or a pixel that does not detect an event.

[0024] The output unit 25 is connected to the control unit 13 via a Y request (Y-Req) line and a Y acknowledge (Y-Ack) line. The output unit 25 also outputs an event signal indicating an event detection by the event detection unit 24 to the readout unit 12. The output of the event signal from the output unit 25 to the readout unit 12 is performed, for example, as follows.

[0025] The output unit 25 of the pixel 11A that detected the event sets the Y-Req line connected to that output unit 25 to High to notify the control unit 13 of an event signal output request. Based on the information on the Y-Req line that has been set to High, the control unit 13 determines the Y address of the pixel 11A that has issued the event signal output request and sets the Y-Ack line corresponding to that Y address to High. Of the multiple pixels in the row whose Y-Ack line has been set to High, the output unit 25 of the pixel 11A that issued the event signal output request outputs an event signal to the readout unit 12 and sets the Y-Req line to Low. This allows arbitration even if multiple pixels simultaneously output event signal output requests, making it possible to read out the event signal while avoiding collisions in the event signal outputs. In addition to reading out the event signal, the readout unit 12 also acquires position information (X address and Y address) of the pixel 11A that output the event signal.

[0026] The request line and acknowledge line may be provided for each row, but may also be provided for each column or pixel, or may also be provided for each region including a plurality of pixels.

[0027] FIG. 3 is a diagram illustrating an example of the arrangement of event detection pixels and comparison pixels. This diagram shows 2×2 unit blocks, each consisting of 2×2 pixels A to D. For each pixel for detecting an event, the direction of the pixel to be used as the comparison pixel for comparison may be determined. For example, assuming that all pixels A to D are event detection pixels, in each unit block, the comparison pixel for event detection pixel A may be the pixel C below, the comparison pixel for event detection pixel C may be the pixel D to the right, the comparison pixel for event detection pixel D may be the pixel B above, and the comparison pixel for event detection pixel B may be the pixel A to the left. Furthermore, there may be multiple comparison pixels for one event detection pixel.

[0028] 4 is a diagram illustrating another example of the arrangement of event detection pixels and comparison pixels. This diagram shows one unit block including 3×3 pixels A to I. In this case, for example, the event detection pixel may be pixel E, which is located in the center of the 3×3 pixels, and the other eight pixels A to D and F to I may be comparison pixels for this event detection pixel. The eight pixels A to D and F to I may not be pixels that detect an event.

[0029] 5 is a diagram showing a first configuration example of the event detection unit 24 of the event detection pixel 11A. In addition to the event detection unit 24A of the first configuration example, this diagram also shows an output unit 25. The event detection unit 24A includes differentiators 31 and 32, comparators 33 and 34, a voltage holding unit 35, and a threshold value holding unit 36. The event detection unit 24A detects a voltage at each time t n It operates as follows:

[0030] The difference calculator 31 calculates the difference at time t nThe voltage V1 output from the amplifier 23 of the pixel 11A in question is input to the comparator 33, along with the voltage V0 held by the voltage holding unit 35. The differentiator 31 then calculates the difference ΔV1 (=V1−V0) between the voltage V1 and the voltage V0, and outputs this voltage difference ΔV1 to the comparator 33.

[0031] The comparator 33 receives the voltage difference ΔV1 output from the differentiator 31 and also receives the threshold value held by the threshold value holding unit 36. The comparator 33 then compares the voltage difference ΔV1 with the threshold value and notifies the output unit 25 of the comparison result.

[0032] The difference calculator 32 calculates the difference at time t n The voltage V2 output from the amplifier 23 of the comparison pixel is input to the comparator 34, as well as the voltage V0 held by the voltage holding unit 35. The differentiator 32 then calculates the difference ΔV2 (=V2−V0) between the voltage V2 and the voltage V0, and outputs this voltage difference ΔV2 to the comparator 34.

[0033] The comparator 34 receives the voltage difference ΔV2 output from the differentiator 32, as well as the threshold value held by the threshold value holding unit 36. The comparator 34 then compares the voltage difference ΔV2 with the threshold value and notifies the output unit 25 of the comparison result.

[0034] The voltage holding unit 35 n-1 At (first time), the voltage V1 output from the amplifier 23 of the pixel 11A is input, and this is n The voltage V0 is held until (second time), and the held voltage V0 is provided to each of the differentiators 31 and 32. The voltage V0 held by the voltage holding unit 35 may be updated when the pixel is reset, when an event is detected, when the Y-Ack line goes high, etc.

[0035] The threshold value holding unit 36 ​​holds a threshold value and provides the held threshold value to each of the comparators 33 and 34. The threshold values ​​provided to each of the comparators 33 and 34 may be the same or different from each other. The threshold value may be fixed or may be changed as appropriate in response to an instruction from the control unit 13 or the like.

[0036] The threshold value held by the threshold value holding unit 36 ​​may be both a positive and a negative value, or may be either one of the values. The comparator 33 may detect a positive event when the voltage difference ΔV1 exceeds a positive threshold value and a negative event when the voltage difference ΔV1 falls below a negative threshold value, or may detect an event without distinguishing between a positive event and a negative event. The same applies to the comparator 34.

[0037] The output unit 25 outputs the difference between the difference calculator 31 and the comparator 33 (time t n-1 and time t n Based on the result of the comparison of the incident light intensity information (voltages V0, V1) of the pixel 11A at each pixel, the output unit 25 can detect a motion event related to the motion of the image on the light receiving surface of the light receiving unit 10. n-1 The incident light intensity information (voltage V0) of the pixel 11A at time t n A directional event or an edge event can be detected based on the result of a comparison between the incident light intensity information (voltage V2) of the comparison pixel in (V1) and the incident light intensity information (voltage V3) of the comparison pixel in (V2). Specifically, the event detection mode includes the following modes.

[0038] The first event detection mode is a mode in which a direction event is further detected when a motion event is detected. That is, the output unit 25 uses the difference calculator 31 and the comparator 33 to calculate the time t n-1 and time t n When the difference ΔV1 between the incident light intensity information (voltages V0, V1) of the pixel 11A is outside the threshold range, the output unit 25 detects that a motion event has occurred. n-1 The incident light intensity information (voltage V0) of the pixel 11A at time tn When a difference ΔV2 between the incident light intensity information (voltage V2) of the comparison pixel at the pixel 11A falls within a threshold range, it is detected that a directional event has occurred, that is, movement has occurred in a direction from the pixel 11A to the comparison pixel.

[0039] In the first event detection mode, when the output unit 25 detects a direction event, it outputs an event signal indicating the detection of the direction event. The output unit 25 may output an event signal indicating the detection of both a motion event and a direction event. Furthermore, when a motion event is detected but no direction event is detected (when the voltage difference ΔV1 is outside the threshold range and the voltage difference ΔV2 is also outside the threshold range), the output unit 25 may output an event signal indicating only the detection of a motion event.

[0040] The second event detection mode is a mode in which an edge event is detected when no motion event is detected. That is, the output unit 25 detects an edge event at time t n-1 and time t n When the difference ΔV1 between the incident light intensity information (voltages V0, V1) of the pixel 11A is within the threshold range, the output unit 25 detects that no motion event has occurred. n-1 The incident light intensity information (voltage V0) of the pixel 11A at time t n When the difference ΔV2 between the incident light intensity information (voltage V2) of the comparison pixel at the pixel 10 and the incident light intensity information (voltage V1) of the comparison pixel at the pixel 10 is outside the threshold range, it is detected that an edge event has occurred.

[0041] In the second event detection mode, when a motion event is not detected but an edge event is detected, the output unit 25 outputs an event signal indicating the detection of an edge event. Alternatively, when a motion event is detected, the output unit 25 may output an event signal indicating the detection of a motion event.

[0042] The third event detection mode is a mode in which a directional event is further detected when a motion event is detected, and an edge event is detected when no motion event is detected. That is, the output unit 25 detects, using the difference calculator 31 and the comparator 33, that a motion event has occurred when the voltage difference ΔV1 is outside the threshold range, and conversely, detects that a motion event has not occurred when the voltage difference ΔV1 is within the threshold range. When a motion event has occurred, the output unit 25 detects, using the difference calculator 32 and the comparator 34, that a directional event has occurred, that is, movement from the subject pixel 11A toward the comparison pixel, when the voltage difference ΔV2 is within the threshold range. On the other hand, when no motion event has occurred, the output unit 25 detects, using the difference calculator 32 and the comparator 34, that an edge event has occurred when the voltage difference ΔV2 is outside the threshold range. Furthermore, when a motion event has been detected but no directional event has been detected (when both the voltage difference ΔV1 and the voltage difference ΔV2 are outside the threshold range), the output unit 25 may output an event signal indicating only the detection of a motion event.

[0043] In the third event detection mode, when a motion event and a directional event are detected, the output unit 25 outputs an event signal indicating the detection of the directional event. When no motion event is detected but an edge event is detected, the output unit 25 outputs an event signal indicating the detection of the edge event. In the third event detection mode, the threshold for detecting a directional event and the threshold for detecting an edge event may be the same or different.

[0044] In any of the first to third event detection modes, when outputting event signals representing multiple types of events from the output unit 25 to the readout unit 12, a signal line for each type of event may be provided between the output unit 25 and the readout unit 12, or an encoded event signal may be sent from the output unit 25 to the readout unit 12. For example, as described with reference to FIG. 4 , when comparing eight pixels adjacent to an event detection pixel on the top, bottom, left, right, and diagonal sides as comparison pixels, eight pairs of difference calculators 32 and comparators 34 are provided corresponding to the eight comparison pixels to detect directional events in each of the eight directions (up, down, left, right, and diagonal). In this case, eight signal lines may be provided between the output unit 25 and the readout unit 12 corresponding to the eight directional events, or an encoded 4-bit event signal may be sent over four signal lines, even in cases where no directional event is detected. Furthermore, if it is not necessary to distinguish between the directions of movement, fewer than eight signal lines may be provided for the eight directional events. The same applies to edge events. The same applies to the case where any two or more types of events among a motion event, a direction event, and an edge event are sent from the output unit 25 to the readout unit 12.

[0045] In any of the first to third event detection modes, it is preferable that the output unit 25 holds the event signal after detecting an event until the Y-Ack line goes High. If the voltage V0 held by the voltage holding unit 35 is updated when an event is detected, it is assumed that the event detection unit 24A will no longer output an event, so the event signal of the output unit 25 is not updated at that time.

[0046] In any of the first to third event detection modes, the output unit 25 may have a time measurement unit that counts clocks in case a new event is detected after an event is detected and before the Y-Ack line goes High, and may also be configured to hold timestamp information for each of multiple events, and output an event signal including the timestamp information.

[0047] In any of the first to third event detection modes, the output section 25 may output an event signal including incident light intensity information (such as voltage V1) when a motion event, a direction event, or an edge event is detected.

[0048] Furthermore, in any of the first to third event detection modes, the output unit 25 may output an event signal only when requested or permitted by the control unit 13. In other words, an event detection request line is connected from the control unit 13 to the output unit 25 of each pixel, and the Y request line is set to High only when the event detection request line is High and an event has occurred for each pixel.

[0049] When detecting an edge event in the second or third event detection mode, it is preferable to do the following. If there is no change in the brightness of the image on the light-receiving surface of the light-receiving unit 10, edge events may occur constantly, resulting in a large number of readouts and inefficiency. Therefore, after an edge event is detected and output, the Y request line may not be set high by the edge event until the edge event is no longer detected (until the voltage difference ΔV1 falls below a predetermined threshold). When the edge event is no longer detected, the Y request line may be set high and the edge event line may be set low to notify the readout unit 12 that the edge has disappeared. Alternatively, an edge event may be detected upon reset and then only once at certain time intervals. Alternatively, an edge event may be detected only in pixels where a motion event was detected, and once an edge event is detected, edge events may not be detected until a motion event is detected again. Alternatively, the comparator 34 may be enabled to detect an edge event when permission or request from the control unit 13 is received.

[0050] 6 is a diagram showing a second configuration example of the event detection unit 24 of the event detection pixel 11A. In addition to the event detection unit 24B of the second configuration example, an output unit 25 is also shown in this diagram. The event detection unit 24B includes comparators 37 and 38 and a threshold setting unit 39. The event detection unit 24B detects a time tn It operates as follows:

[0051] The comparator 37 detects the time t n The comparator 37 receives the voltage V1 output from the amplifier 23 of the pixel 11A, as well as the threshold value set and held by the threshold value setting unit 39. The comparator 37 then compares the voltage V1 with the threshold value and notifies the output unit 25 of the comparison result.

[0052] The comparator 38 detects the time t n The comparator 38 receives the voltage V2 output from the amplifier 23 of the comparison pixel, as well as the threshold value set and held by the threshold value setting unit 39. The comparator 38 then compares the voltage V2 with the threshold value and notifies the output unit 25 of the comparison result.

[0053] The threshold setting unit 39 sets the threshold value at time t n-1 At (first time), the voltage V1 output from the amplifier 23 of the pixel 11A is input, and the threshold value set based on the voltage V1 is set at time t n The threshold is held until (second time), and the held threshold is provided to each of the comparators 37 and 38. The threshold may be updated when the pixel is reset, when an event is detected, when the Y-Ack line goes high, etc.

[0054] The thresholds that the threshold setting unit 39 provides to the comparators 37 and 38 may be the same or different from each other. n The threshold value output from the threshold value setting unit 39 at (second time) is n-1 The voltage V1 output from the amplifier 23 of the pixel 11A at (the first time point) (i.e., at time t n (V0) is a voltage that is maintained until (the second time) H and a small value V L The comparator 37 determines whether the voltage V1 is equal to or lower than the threshold value V H A positive event occurs when the voltage V1 exceeds the threshold V LThe comparator 38 may detect events by distinguishing between positive and negative events and events when the output voltage falls below 0V, or may detect events without distinguishing between positive and negative events.

[0055] The output unit 25 receives the comparison signal from the comparator 37 (at time t n-1 and time t n Based on the result of the comparison of the incident light intensity information (voltages V0 and V1) of the pixel 11A at each pixel, the output unit 25 can detect a motion event related to the motion of the image on the light receiving surface of the light receiving unit 10. n-1 The incident light intensity information (voltage V0) of the pixel 11A at time t n A directional event or an edge event can be detected based on the result of a comparison with incident light intensity information (voltage V2) of a comparison pixel in the pixel A. This configuration also has aspects similar to the first to third event detection aspects described above.

[0056] The event detection pixel described above with reference to Figures 2 to 6 converts the current signal output from the photodiode 21 into a voltage signal using the IV converter 22 and amplifier 23, and detects an event by using the magnitude (voltage value) of the voltage signal as incident light intensity information. Alternatively, as will be described next with reference to Figures 7 and 8, the event detection pixel may count pulses output when the photodiode detects a photon, and detect an event by using the time it takes for the pulse count value to reach a predetermined value as incident light intensity information. In this case, the shorter the time it takes for the pulse count value to reach the predetermined value, the greater the incident light intensity.

[0057] 7 is a diagram showing another example of the configuration of an event detection pixel. In addition to an event detection pixel 11C, this diagram also shows a readout unit 12 and a control unit 13. The event detection pixel 11C shown in this diagram includes an avalanche photodiode (APD) 41, a quenching element 42, a waveform shaping unit 43, an event detection unit 44, and an output unit 45. Note that pixels that do not detect events also include the APD 41, quenching element 42, and waveform shaping unit 43.

[0058] A voltage V is applied to the anode of the APD41. DD is applied to the cathode of the APD 41 via the quench element 42, and a voltage V bd is applied between the anode and cathode. A voltage greater than the breakdown voltage of the APD 41 is applied between the anode and cathode. The APD 41 avalanche-multiplies electrons generated by the incidence of a single photon, allowing a large current (avalanche current) to flow. The quench element 42 converts the avalanche current into a voltage signal, discharges the electrons generated and accumulated by the avalanche multiplication, and returns the voltage to the initial voltage.

[0059] The waveform shaping unit 43 receives the voltage at the connection point between the APD 41 and the quench element 42, shapes the waveform of a pulse generated by the incidence of one photon on the APD 41, and outputs the shaped pulse P1 to the event detection unit 44. The waveform shaping unit 43 can be configured to include, for example, an inverter and a buffer.

[0060] The event detection unit 44 receives the pulse P1 output from the waveform shaping unit 43 of the pixel 11C in question, as well as the pulse P2 output from the waveform shaping unit 43 of a comparison pixel located near the pixel 11C in question among the multiple pixels 11, and detects an event in the image on the light receiving surface based on the counting results (incident light intensity information) of these pulses P1 and P2. The output unit 45 is similar to the output unit 25 described above.

[0061] 8 is a diagram showing an example of the configuration of the event detection unit 44 of the event detection pixel 11C. In addition to the event detection unit 44, this diagram also shows an output unit 45. The event detection unit 44 includes pulse counting units 51 and 52, time measurement units 53 and 54, a determination unit 55, comparators 56 and 57, and a threshold setting unit 58.

[0062] The pulse counter 51 receives and counts the pulse P1 output from the waveform shaping unit 43 of the pixel 11C in question, thereby counting the photons incident on the APD 41 of the pixel 11C in question. The pulse counter 52 receives and counts the pulse P2 output from the waveform shaping unit 43 of the comparison pixel, thereby counting the photons incident on the APD 41 of the comparison pixel. The time measurement units 53 and 54 count clock pulses of a clock with a constant frequency (for example, a clock provided from outside the sensor).

[0063] The determination unit 55 instructs the initialization of the count values ​​of the pulse counters 51 and 52 and the time measurement units 53 and 54. After the initialization, the pulse counters 51 and 52 and the time measurement units 53 and 54 start their counting operations.

[0064] Determination unit 55 receives the count values ​​of pulse counters 51 and 52 and time measurement units 53 and 54. Determination unit 55 acquires count value C1 of time measurement unit 53 when the count value of pulse counter 51 reaches a predetermined value, and acquires count value C2 of time measurement unit 54 when the count value of pulse counter 52 reaches a predetermined value. After the count values ​​of both pulse counters 51 and 52 reach the predetermined values, determination unit 55 instructs pulse counters 51 and 52 and time measurement units 53 and 54 to initialize their count values.

[0065] The comparator 56 receives the count value C1 from the determination unit 55, as well as the threshold value set and held by the threshold value setting unit 58. The comparator 56 then compares the count value C1 with the threshold value and notifies the output unit 45 of the comparison result.

[0066] Comparator 57 receives count value C2 from determination unit 55 and also receives the threshold value set and held by threshold setting unit 58. Comparator 57 then compares count value C2 with the threshold value and notifies output unit 45 of the comparison result.

[0067] The threshold setting unit 58 receives the count value C1 from the determination unit 55, holds this as the count value C0, and also holds a threshold value set based on the count value C0, and provides the held threshold value to each of the comparators 56 and 57. The held count value C0 may be updated when a pixel is reset, when an event is detected, when the Y-Ack line goes high, etc.

[0068] The threshold value set by the threshold setting unit 58 to the comparators 56 and 57 may be the same or different from each other. The threshold value output from the threshold setting unit 58 is a value C that is larger than the stored count value C0. H and a small value of C L The comparator 56 determines whether the count value C1 is equal to or exceeds the threshold value C H A positive event occurs when the count value C1 exceeds the threshold value C L The comparator 57 may detect events by distinguishing between positive and negative events, or may detect events without distinguishing between positive and negative events.

[0069] The output unit 45 can detect a motion event related to movement in the image on the light receiving surface of the light receiving unit 10 based on the result of the comparison by the comparator 56 (comparison of the incident light intensity information (count value C0, C1) of the pixel 11C itself). The output unit 45 can also detect a directional event or an edge event based on the result of the comparison by the comparator 57 (comparison of the incident light intensity information (count value C0) of the pixel 11C itself with the incident light intensity information (count value C2) of the comparison pixel). This configuration also has aspects similar to the first to third event detection aspects described above.

[0070] The configuration example of event detection unit 44 shown in Fig. 8 compares count values ​​C1 and C2 with a threshold value set based on count value C0, and corresponds to the second configuration example of event detection unit 24 shown in Fig. 6. As another configuration example of event detection unit 44, which corresponds to the first configuration example of event detection unit 24 shown in Fig. 5, the difference between count value C0 and count values ​​C1 and C2 may be found and this difference may be compared with a threshold value.

[0071] There are various configurations as modified examples. In any of the configurations, the event detection unit may use either a voltage value or a count value as incident light intensity information, and may correspond to either the first or second configuration example. The configuration example described below uses a voltage value as incident light intensity information and corresponds to the first configuration example. Event sensors of any of the configurations can detect not only motion events but also other types of events (directional events or edge events) in real time.

[0072] Fig. 9 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24A shown in Fig. 5, the event detection unit 24D shown in Fig. 9 differs in that it is provided with multiple sets (two sets in this figure) of differentiators 32 and comparators 34 for comparing the incident light intensities between the subject pixel 11A and multiple comparison pixels.

[0073] The difference calculator 321 calculates the difference at time t n , the voltage V output from the amplifier 23 of the first comparison pixel 21 , and also inputs the voltage V0 held by the voltage holding unit 35. Then, the difference calculator 321 calculates the voltage V 21 and the voltage V0 ΔV 21 (=V 21 -V0), and calculate this voltage difference ΔV 21 is output to the comparator 341.

[0074] The comparator 341 detects the voltage difference ΔV output from the differentiator 321. 21The comparator 341 also receives the threshold value held by the threshold value holding unit 36. The comparator 341 then receives the voltage difference ΔV 21 The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0075] The difference calculator 322 calculates the difference at time t n The voltage V output from the amplifier 23 of the second comparison pixel 22 , and also inputs the voltage V0 held by the voltage holding unit 35. Then, the difference calculator 322 calculates the voltage V 22 and the voltage V0 ΔV 22 (=V 22 -V0), and calculate this voltage difference ΔV 22 is output to the comparator 342.

[0076] The comparator 342 detects the voltage difference ΔV output from the differentiator 322. 22 The comparator 342 receives the voltage difference ΔV 22 The output unit 25 compares the calculated value with a threshold value and notifies the output unit 25 of the comparison result.

[0077] The output unit 25 can detect a more detailed directional event or edge event based on the comparison results for each of the multiple comparison pixels. The output unit 25 may output an event signal for each comparison pixel, or may output an event signal that represents the most likely directional event or edge event.

[0078] Fig. 10 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24A shown in Fig. 5, the event detection unit 24E shown in Fig. 10 differs in that it further includes a selection unit 61.

[0079] The selection unit 61 selects the voltage V output from the amplifier 23 of each of a plurality of comparison pixels (four in this figure). 21 ~V 24 Input the voltage V 21 ~V 24The selector 61 sequentially selects one of the voltages V 21 ~V 24 and the voltage V 21 ~V 24 The voltage V 21 ~V 24 The selection may be made by the selection unit 61 itself, or may be made in response to an instruction from the control unit 13.

[0080] The differentiator 32 calculates the voltage V 21 ~V 24 and the voltage V0 held by the voltage holding unit 35. The differentiator 32 then calculates the difference between the two input voltages and outputs this voltage difference ΔV2 to the comparator 34.

[0081] The output unit 25 can detect a more detailed directional event or edge event based on the comparison results for each of the multiple comparison pixels. The output unit 25 may output an event signal for each comparison pixel, or may output an event signal representing the most likely directional event or edge event. The output unit 25 may also output an event signal including position information of the comparison pixel (information indicating the direction of the comparison pixel relative to the target pixel 11A).

[0082] Fig. 11 is a diagram showing another example configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24E shown in Fig. 10, the event detection unit 24F shown in Fig. 11 differs in that it does not include the differentiator 31 and the comparator 33, and that it includes a selection unit 62 instead of the selection unit 61.

[0083] The selection unit 62 receives the voltage V1 output from the amplifier 23 of the pixel 11A, and also receives the voltage V2 output from the amplifier 23 of each of a plurality of (four in this figure) comparison pixels. 21 ~V 24 Then, the selection unit 62 inputs the voltages V1 and V21 ~V 24 The selector 62 sequentially selects one of the voltages V1, V2, V3, V4, V5, V6, V7, V8, V9, V10, V11, V12, V13, V14, V15, V16, V17, V18, V19, V20, V21, V22, V23, V24, V25, V26, V27, V28, V29, V30, V 21 ~V 24 and the voltage V1,V 21 ~V 24 Alternatively, one of the voltages V1, V2 may be selected in sequence. 21 ~V 24 The selection may be made by the selection unit 62 itself, or may be made in response to an instruction from the control unit 13.

[0084] The differencer 32 calculates the voltages V1 and V 21 ~V 24 and the voltage V0 held by the voltage holding unit 35. The differentiator 32 then calculates the difference between the two input voltages and outputs this voltage difference ΔV to the comparator 34.

[0085] The output unit 25 can detect a motion event when the voltage V1 output from the amplifier 23 of the pixel 11A is selected by the selection unit 62. The output unit 25 can also detect a motion event when the voltage V1 output from the amplifier 23 of each of the plurality of comparison pixels is selected by the selection unit 62. 21 ~V 24 When either of the voltages V1 and V2 is selected by the selection unit 62, a direction event or an edge event can be detected. When a motion event is detected while the voltage V1 is selected, the voltage holding unit 35 is not updated immediately, but the voltage V 21 ~V 24 A directional event can be detected by selecting one of the above. The output unit 25 may also output an event signal including information indicating which of the subject pixel 11A and the plurality of comparison pixels has been selected by the selector 62.

[0086] Fig. 12 is a diagram showing another example of the configuration of the event detection unit 24 of the event detection pixel 11A. Compared to the configuration of the event detection unit 24A shown in Fig. 5, the event detection unit 24G shown in Fig. 12 differs in that it includes multiple (two in this figure) differentiators 32 and further includes a selection unit 63.

[0087] The difference calculator 321 calculates the difference at time t n , the voltage V output from the amplifier 23 of the first comparison pixel 21 , and also inputs the voltage V0 held by the voltage holding unit 35. Then, the difference calculator 321 calculates the voltage V 21 and the voltage V0 ΔV 21 (=V 21 -V0), and calculate this voltage difference ΔV 21 to the selection unit 63.

[0088] The difference calculator 322 calculates the difference at time t n The voltage V output from the amplifier 23 of the second comparison pixel 22 , and also inputs the voltage V0 held by the voltage holding unit 35. Then, the difference calculator 322 calculates the voltage V 22 and the voltage V0 ΔV 22 (=V 22 -V0), and calculate this voltage difference ΔV 22 to the selection unit 63.

[0089] The selector 63 selects the voltage difference ΔV output from the differentiator 321. 21 is input, and the voltage difference ΔV output from the differentiator 322 is 22 Then, the selection unit 63 inputs the voltage difference ΔV 21 ,ΔV 22 The voltage difference ΔV2 is output to the comparator 34. 21 ,ΔV 22 The selection may be made by the selection unit 63 itself, or may be made in response to an instruction from the control unit 13.

[0090] The output unit 25 can detect a more detailed directional event or edge event based on the comparison results for each of the multiple comparison pixels. The output unit 25 may output an event signal for each comparison pixel, or may output an event signal representing the most likely directional event or edge event. The output unit 25 may also output an event signal including position information of the comparison pixel (information indicating the direction of the comparison pixel relative to the target pixel 11A).

[0091] The selector 63 also receives the voltage difference ΔV1 output from the differentiator 31 and calculates the voltage differences ΔV1, ΔV 21 ,ΔV 22 and ΔV1 are selected, the voltage difference is output to the comparator 34. In this case, the comparator 33 is not required. In this case, when a movement event is detected while the voltage difference ΔV1 is selected, the voltage holding unit 35 is not updated immediately, but the voltage difference ΔV 21 ,ΔV 22 Directional events can be detected by selecting one of the following.

[0092] The selector 63 also selects the voltage difference ΔV 21 ,ΔV 22 , as will be explained with reference to FIG. 13, the voltage difference ΔV 21 ,ΔV 22 The voltage difference with the smallest or largest absolute value may be selected.

[0093] 13 is a diagram showing an example of the configuration of the selection unit 63 of the event detection unit 24G in FIG. 12. In this diagram, the voltages V 21 ~V 24 and V0, the voltage difference ΔV 21 ~ΔV 24 is input to the selection unit 63. The four comparison pixels are, for example, four pixels adjacent to the event detection unit 24G on the top, bottom, left and right sides. The selection unit 63 shown in this diagram includes comparators 71 to 73 and selectors 74 to 76.

[0094] The comparator 71 and the selector 74 each detect a voltage difference ΔV 21 (=V 21 -V0) and voltage difference ΔV 22 (=V 22 The selector 74 inputs the voltage difference ΔV 21 ,ΔV 22 The voltage difference judged by the comparator 71 to have the smaller absolute value is selected and output.

[0095] The comparator 72 and the selector 75 each detect a voltage difference ΔV 23 (=V 23 -V0) and voltage difference ΔV 24 (=V 24 -V0) is input. The selector 75 selects the voltage difference ΔV 23 ,ΔV 24 The voltage difference judged by the comparator 72 to have the smaller absolute value is selected and output.

[0096] The comparator 73 and the selector 76 each receive the voltage difference output from the selectors 74 and 75. The selector 76 selects the voltage difference determined by the comparator 73 to have the smaller absolute value of the two input voltage differences, and outputs the selected voltage difference to the comparator 73.

[0097] The results of the comparisons made by the comparators 71 to 73 are sent to the output unit 25. The voltage difference ΔV2 output from the selector 76 to the comparator 34 is the voltage difference ΔV 21 ~ΔV 24 This allows the output unit 25 to grasp the direction of the movement in detail based on the result of the comparison by the comparator 34 when a movement event is detected based on the result of the comparison by the comparator 33.

[0098] In this figure, four comparison pixels are assumed to be adjacent in the vertical and horizontal directions of the event detection unit 24G, but even if eight comparison pixels are assumed to be adjacent in the vertical and horizontal and diagonal directions of the event detection unit 24G, the eight voltage differences ΔV 21 ~ΔV 28 The voltage difference with the smallest absolute value is selected and input to the comparator 34.

[0099] Conversely, depending on the selection method of each of the selectors 74 to 76, the voltage difference ΔV2 input from the selector 76 to the comparator 34 may be 21 ~ΔV 24 In this case, when the output unit 25 does not detect a motion event based on the comparison result by the comparator 33, the output unit 25 can grasp the edge direction in detail based on the comparison result by the comparator 34.

[0100] Fig. 14 is a diagram showing another configuration of the event sensor 1. Compared to the configuration shown in Fig. 2, the configuration shown in Fig. 14 differs in that a mask signal is provided from the control unit 13 to the output unit 25 of the event detection pixel 11A.

[0101] The mask signal permits or prohibits the output of an event signal from the event detection pixels 11A. The permitting or prohibiting of event signal output may be performed for each individual event detection pixel 11A, or for each region, row, or column. The permitting or prohibiting of event signal output may also be performed for each direction of movement in the case of a directional event, or for each edge direction in the case of an edge event. Based on the mask signal provided by the control unit 13, the output unit 25 of the event detection pixels 11A outputs an event signal to the readout unit 12 for permitted events, and does not set the Y request line high and does not output an event signal for prohibited events.

[0102] Generally, an event sensor detects a motion event not only when a subject is moving, but also when the subject is stationary and the event sensor is moving. In particular, the device described in Patent Document 1 cannot extract a specific event until it has performed the required processing on the event signal output from the event sensor. Therefore, even if the necessity of extraction for each event is determined in advance, event signals are output for events that do not require extraction, which is inefficient. In contrast, the present configuration is efficient because it allows the output unit 25 to selectively output events to be extracted.

[0103] In the above description, the mask signal is provided from the control unit 13 to the event detection pixel 11A, but the mask signal may be provided from the control unit 13 to the readout unit 12. In the latter case, the readout unit 12 outputs an event signal to the outside for a permitted event, and does not output an event signal to the outside for a prohibited event.

[0104] FIG. 15 is a diagram illustrating an example of an application of the event sensor of the configuration example of FIG. 14. This diagram shows an image formed on the light receiving surface of an event sensor mounted on an automobile, of a subject located ahead in the direction of travel of the automobile. The image shows as subjects other automobiles parked on the road, roadside trees off the road, and a pedestrian crossing the road from left to right. When an automobile equipped with an event sensor is traveling, it is highly important to detect other automobiles and pedestrians on the road, but it is less important to detect roadside trees.

[0105] If the event sensor installed in the vehicle is a conventional one, it will detect movement events, regardless of their importance, for other vehicles on the road, roadside trees outside the road, and pedestrians in the image formed on the light receiving surface of the event sensor.

[0106] In contrast, if the event sensor mounted on a vehicle is the one of this embodiment, in the image formed on the light-receiving surface of the event sensor, a downward direction event will occur for other vehicles on the road and roadside trees off the road, and a rightward or downward-right direction event will occur for pedestrians. The event sensor of the configuration example in Figure 14 can be set to allow or prohibit the output of an event signal depending on the area of ​​the image (the area corresponding to the road, the area off the road) and the direction of movement.

[0107] Specifically, in the region of the image formed on the light-receiving surface of the event sensor that corresponds to the left side of the road, the output of an event signal for a rightward direction event is permitted and the output of event signals for direction events in other directions is prohibited. In the region of the image formed on the light-receiving surface of the event sensor that corresponds to the right side of the road, the output of an event signal for a leftward direction event is permitted and the output of event signals for direction events in other directions is prohibited. In the region of the image formed on the light-receiving surface of the event sensor that corresponds to the road, the output of an event signal for a downward direction event is permitted and the output of event signals for direction events in other directions is prohibited.

[0108] Information about the movement of the event sensor may be received from an external source (for example, the vehicle or a passenger) or from a sensor that can obtain the direction of movement, such as a gyro sensor. This information may be used to set whether to permit or prohibit the output of an event signal.

[0109] In this way, in the event sensor of the configuration example shown in Figure 14, by setting the output of an event signal to be permitted or prohibited depending on the level of importance, it becomes possible to detect and recognize important events quickly and efficiently, and subsequent responses can be made quickly.

[0110] FIG. 16 is a diagram illustrating another example of the use of the event sensor of the configuration example of FIG. 14. This diagram shows a schematic image of a subject in front of the robot, formed on the light receiving surface of an event sensor mounted on a robot. The image shows a person's hand waving up and down and a walking person as the subject. The robot equipped with the event sensor recognizes the hand waving up and down as a signal to cancel its standby state and move on to the next action.

[0111] If the event sensor installed on the robot is a conventional one, the up and down movement that signals the end of the standby state cannot be extracted until the required processing is performed on the event signal output from the event sensor.

[0112] In contrast, if the event sensor mounted on the robot is the one of this embodiment, a directional event of up or down will be generated for a person's hand waving up and down in the image formed on the light-receiving surface of the event sensor, and a directional event of right or left will be generated for a person walking. The event sensor of the configuration example in Figure 14 can be set to allow or prohibit the output of an event signal depending on the direction of movement.

[0113] Specifically, in the image formed on the light receiving surface of the event sensor, the output of an event signal for a directional event in the upward or downward direction is permitted, and the output of an event signal for a directional event in any other direction is prohibited. By setting it in this way, it is possible to prevent the robot from misinterpreting a signal due to a person or object moving left or right.

[0114] 15 and 16, the event sensor of the configuration example in Fig. 14 can output event signals less frequently than conventional event sensors, thereby reducing power consumption. Furthermore, by detecting and using edge events in addition to directional events, the accuracy of event recognition can be improved.

[0115] Fig. 17 is a diagram showing another configuration of the event sensor 1. Compared to the configuration shown in Fig. 2, the configuration shown in Fig. 17 further includes a histogram generation unit 14. The histogram generation unit 14 counts the number of detected events for each event type based on the event signals output from each event detection pixel 11A of the light receiving unit 10, generates a histogram, and outputs the histogrammed event signals to the readout unit 12.

[0116] For example, the histogram generation unit 14 includes counters provided for each p×q pixel. Each counter counts directional events for each direction of motion over a certain period of time based on the event signals output from the corresponding p×q pixel, generating a histogram. This reduces the number of reads performed by the readout unit 12 and the burden on subsequent processing. Furthermore, based on this histogram, tasks such as classification and identification can be directly performed using machine learning. Examples of machine learning used here include neural networks, reservoirs, Histogram of Optical Flow (HOF), and Motion Boundary Histograms (MBH).

[0117] A counter may be provided for each event type. Alternatively, a counter may be provided in common for both upward and downward direction events, in which case the count value is decremented by 1 for an upward direction event and incremented by 1 for a downward direction event.

[0118] The histogram generator 14 may include multiple counters, each corresponding to p×q pixels, that perform different encoding. The encoding here refers to, for example, determining that for a given pixel, counter A and counter B count the number of events, while counter C does not count the number of events. Weighted encoding may also be used. This makes it possible to restore the directional events of each pixel using numerical calculations such as compressed sensing based on the histogram.

[0119] In the histogram generation unit 14, one or more counters may be provided for each p×q pixel, or one pixel may be connected to multiple counters. The p×q pixel area may partially overlap with other areas. The combination of which pixels are connected to which counters may be arbitrarily selected. One counter may be provided for all pixels.

[0120] The histogram generation unit 14 may output only the most frequent direction of the counter, only the directions with higher count values, or only these counter values.The histogram generation unit 14 may output only the directions whose counter values ​​exceed a predetermined threshold, only the most frequent direction among them, or only these count values.The histogram generation unit 14 may use the histogram values ​​as weights to calculate the directions of the entire p×q pixel region (by averaging the directions) and output them.

[0121] Fig. 18 is a diagram illustrating an example of the use of the event sensor having the configuration example of Fig. 17. In this diagram, two unit blocks, each consisting of 4x4 pixels, are shown, and directional events detected at each pixel during a certain period are indicated by hatched arrows.

[0122] In the left half unit block 1, three pixels detect an upward direction event, one pixel detects a downward direction event, four pixels detect a leftward direction event, and one pixel detects a rightward direction event. In this case, the direction weights are -2 for the up-down direction and -3 for the left-right direction, resulting in an overall upward-left direction. The most frequent value is the leftward direction.

[0123] In the right half of unit block 2, one pixel detects an upward direction event, three pixels detect a downward direction event, one pixel detects a leftward direction event, and three pixels detect a rightward direction event. In this case, the direction weights are +2 for the up / down direction and +2 for the left / right direction, resulting in a downward / rightward direction overall. The most frequent values ​​are the downward and rightward directions.

[0124] 19 is a diagram showing the configuration of an event detection device including an event sensor 1. The event detection device shown in this figure includes an inference unit 2 in addition to the event sensor 1. The inference unit 2 performs inference regarding the image formed on the light-receiving surface of the light-receiving unit 10 based on the event signal read by the readout unit 12 of the event sensor 1. The inference unit 2 performs tasks such as recognition, classification, and prediction using an inference model such as machine learning or a rule-based algorithm based on the event signal that includes information on any of a motion event, a directional event, and an edge event.

[0125] The inference unit 2 accumulates one or more types of information contained in the event signal for a certain period of time and inputs the accumulated information to an inference model or the like as a single image with each information per pixel. Examples of inference models include a convolutional neural network (CNN) and a transformer model that receive non-time-series data as input, and a recurrent neural network (RNN) and a reservoir computing (RC) model that receive time-series data as input. The transformer model can also handle time-series data. In these inference models, an inference result may be obtained for each image, or the most frequent result among inference results for multiple images may be obtained. In particular, in the case of a time-series inference model, the inference result after multiple images have been input may be used.

[0126] Each time an event signal is output, the inference unit 2 inputs the information contained in the event signal into the time series inference model, and may obtain an inference result for each input, or may obtain the most frequent value among multiple inference results, or may obtain an inference result after multiple pieces of information have been input.

[0127] 20 is a diagram showing another configuration of an event detection device including an event sensor 1. The event detection device shown in this figure includes an inference unit 2 and a movement detection unit 3 in addition to the event sensor 1.

[0128] In this configuration, the motion detection unit 3 acquires motion information (such as what kind of motion occurred at what position and in what direction) based on an event signal containing information on a motion event, a direction event, or an edge event. The inference unit 2 performs more detailed recognition and classification based on the motion information and edge event information acquired by the motion detection unit 3. [Explanation of symbols]

[0129] 1...Event sensor, 2...Inference unit, 3...Motion detection unit, 10...Light receiving unit, 11...Pixel, 11A, 11C...Event detection pixel, 12...Readout unit, 13...Control unit, 14...Histogram generation unit, 21...Photodiode, 22...IV conversion unit, 23...Amplifier, 24, 24A to 24G...Event detection unit, 25...Output unit, 31, 32, 321, 322...Differentiator, 33, 34, 341, 342 42...comparator, 35...voltage holding unit, 36...threshold holding unit, 37, 38...comparator, 39...threshold setting unit, 41...avalanche photodiode (APD), 42...quench element, 43...waveform shaping unit, 44...event detection unit, 45...output unit, 51, 52...pulse counting unit, 53, 54...time measurement unit, 55...judgment unit, 56, 57...comparator, 58...threshold setting unit, 61 to 63...selection units.

Claims

1. a light receiving unit having a light receiving surface on which a plurality of pixels are arranged, an image being formed on the light receiving surface by light incident thereon, and at least one pixel among the plurality of pixels being an event detection pixel; a readout unit that reads out the event signal output from the light receiving unit; a control unit that controls the operations of the light receiving unit and the readout unit; Equipped with The event detection pixel is One or more pixels among the plurality of pixels that are in the vicinity of the event detection pixel are set as comparison pixels, detecting a directional event related to a direction of movement in the image or an edge event related to an edge in the image based on a comparison of incident light intensity information of the event detection pixel at a first time point and a second time point that is later than the first time point, and a comparison of the incident light intensity information of the event detection pixel at the first time point and the incident light intensity information of the comparison pixel at the second time point; outputting an event signal indicative of the detection of the directional event or the edge event; Event sensors.

2. the event detection pixel detects the directional event when detecting a motion event indicating that a motion has occurred in the image based on a comparison of incident light intensity information of the event detection pixel at the first time point and the second time point. The event sensor of claim 1 .

3. the event detection pixel detects the edge event when it has not detected a motion event indicating that a motion has occurred in the image based on a comparison of incident light intensity information of the event detection pixel at the first time point and the second time point. The event sensor of claim 1 .

4. the event detection pixel selects one of the plurality of comparison pixels and detects the directional event or the edge event based on incident light intensity information of the selected comparison pixel; The event sensor of claim 1 .

5. When the event detection pixel detects a plurality of types of events, the event detection pixel outputs an event signal without distinguishing between the plurality of types of events. The event sensor of claim 1 .

6. When detecting a plurality of types of events, the event detection pixel outputs an encoded event signal. The event sensor of claim 1 .

7. the control unit permits or prohibits the output of an event signal from the event detection pixel or the readout of the event signal by the readout unit; The event sensor of claim 1 .

8. a histogram generating unit that counts the number of detected events for each event type based on the event signal output from the light receiving unit, generates a histogram, and outputs the histogrammed event signal to the readout unit; The event sensor of claim 1 .

9. An event sensor according to any one of claims 1 to 8; an inference unit that performs inference regarding the image based on the event signal read by the readout unit of the event sensor; An event detection device comprising:

Images