Light detection element and system

Abstract

[Problem] To more appropriately reduce a data amount. [Solution] A light detection element comprises: a pixel array unit that includes a first pixel group containing a plurality of first pixels, and a second pixel group containing a plurality of second pixels; a first event detection unit that detects, as a first event, a luminance change in incident light in the first pixel; and a second event detection unit that detects, as a second event, a luminance change in incident light in the second pixel, at a timing based on the detected first event.

Description

Photodetection element and system

[0001] SUMMARY OF THE INVENTION Embodiments according to the present disclosure relate to light detection elements and systems.

[0002] There is a known imaging device that acquires data of a portion of a scene where a luminance level has changed due to an event only when the event occurs in the scene. This type of imaging device is sometimes called an EVS (Event-based Vision Sensor).

[0003] Furthermore, in EVS, in order to reduce the amount of data, an event rate may be estimated based on the number of detected events, and the detected events may be filtered based on a comparison between the estimated event rate and a threshold value (see Patent Document 1).

[0004] JP 2021-508975 A JP 2022-76837 A

[0005] However, in dynamic scenes where the EVS sensor itself moves, a large number of events are detected, and a large amount of calculation is required for event processing.

[0006] Therefore, the present disclosure provides a photodetector element and a system that can more appropriately reduce the amount of data.

[0007] In order to solve the above problem, according to the present disclosure, there is provided a light detection element including: a pixel array unit having a first pixel group including a plurality of first pixels and a second pixel group including a plurality of second pixels; a first event detection unit that detects a change in luminance of incident light at the first pixels as a first event; and a second event detection unit that detects a change in luminance of incident light at the second pixels as a second event at a timing based on the detected first event.

[0008] The first event detection unit may further include a control unit that controls the second event detection unit to continue detecting the first event and to detect the second event at a timing based on the detected first event.

[0009] The control unit may determine a detection timing for causing the second event detection unit to detect the second event based on the detected first event, and control the second event detection unit based on the determined detection timing.

[0010] The control unit may calculate an entropy of the detected first event, and determine the detection timing based on the calculated entropy.

[0011] The light detecting element according to claim 3 , wherein the control unit calculates an event rate of the first event detected in a predetermined period, and determines the detection timing based on the calculated event rate.

[0012] The control unit may determine the detection timing by applying the detected first event to a rule-based model generated based on a correspondence between the first event and the detection timing.

[0013] The control unit may determine the detection timing by applying the detected first event to a machine learning model generated based on a correspondence between the first event and the detection timing.

[0014] The control unit may determine the detection timing by applying the detected first event to a rule-based model generated based on a correspondence between the first event and the detection timing, and applying a machine learning model generated based on the correspondence between the first event and the detection timing to an output of the rule-based model, or by applying the detected first event to the machine learning model and applying the rule-based model to an output of the machine learning model.

[0015] The pixel array unit may be divided into a plurality of regions, each region including the first pixel group and the second pixel group, and the second event detection unit may detect the second event for each of the divided regions.

[0016] The second event detection unit may detect the second event within the area at a timing based on the first event detected within the area.

[0017] The second event detection unit may detect the second event in the first area at a timing based on the first event detected in a plurality of areas including an area different from the first area.

[0018] The second event detection unit may detect the second event of an on-event at a timing based on the first event of the detected on-event, and may detect the second event of an off-event at a timing based on the first event of the detected off-event.

[0019] The second pixel may have a photoelectric conversion element that photoelectrically converts incident light to generate a photocurrent, and the second event detection unit may have a current-voltage conversion unit that converts the photocurrent into a voltage, a subtractor that calculates and outputs an amount of change in the voltage, and a quantizer that detects whether the amount of change exceeds a threshold, and the quantizer may have a switching unit that switches the threshold depending on whether the detected first event is an on event or an off event, and a comparator that compares the amount of change with the threshold switched by the switching unit.

[0020] the second pixel has a photoelectric conversion element that photoelectrically converts incident light to generate a photocurrent, the second event detection unit has: a current-voltage conversion unit that converts the photocurrent into a voltage; a subtractor that calculates and outputs a change in the voltage; and a quantizer that detects whether the change exceeds a threshold, the subtractor has: a first capacitor; an inverter whose input is electrically connected to the first capacitor; a first switch connected between a node between the first capacitor and the input of the inverter and an output of the inverter, the first switch initializing an output of the subtractor; a second capacitor connected between the node and the output of the inverter; and a second switch connected between the node and the output of the inverter, and the second switch may be turned on or off at a timing based on the detected first event.

[0021] the second pixel has a photoelectric conversion element that photoelectrically converts incident light to generate a photocurrent, the second event detection unit has: a current-voltage conversion unit that converts the photocurrent into a voltage; a subtractor that calculates and outputs a change in the voltage; and a quantizer that detects whether the change exceeds a threshold, the subtractor having: a first capacitor; an inverter whose input is electrically connected to the first capacitor; a first switch connected between a node between the first capacitor and the input of the inverter and an output of the inverter, the first switch initializing an output of the subtractor; and a second capacitor connected between the node and the output of the inverter, and the second event detection unit may further include a switch control unit that controls the first switch based on the detected first event.

[0022] The sensitivity of the first pixel may be different from the sensitivity of the second pixel.

[0023] The second event detection unit may detect the second event at a timing based on the first event detected in the first pixel group.

[0024] According to the present disclosure, there is provided a system comprising: a pixel array unit having a first pixel group including a plurality of first pixels and a second pixel group including a plurality of second pixels; a first event detection unit that detects a change in luminance of incident light at the first pixel as a first event; a second event detection unit that detects a change in luminance of incident light at the second pixel as a second event at a timing based on the detected first event; and an emission control unit that controls a light source to emit light in a first emission pattern and a second emission pattern that follows the first emission pattern, wherein the emission control unit causes the second event detection unit to detect the second event that corresponds to the second emission pattern, and causes the light source to emit light in the first emission pattern.

[0025] 1 is a block diagram showing an example of a system configuration of an imaging system to which the technology according to the present disclosure is applied. FIG. 2 is a block diagram showing an example of a configuration of an imaging device according to a first configuration example of the present disclosure. FIG. 3 is a block diagram showing an example of a configuration of a pixel array unit. FIG. 4 is a circuit diagram showing an example of a circuit configuration of a pixel. FIG. 5 is a block diagram showing a first configuration example of an address event detection unit. FIG. 6 is a circuit diagram showing an example of a configuration of a current-voltage conversion unit in the address event detection unit. FIG. 7 is a circuit diagram showing an example of a configuration of a subtractor and a quantizer in the address event detection unit. FIG. 8 is a block diagram showing a second configuration example of the address event detection unit. FIG. 9 is a block diagram showing an example of a configuration of an imaging device according to a second configuration example of the present disclosure. FIG. 10 is an exploded perspective view showing an outline of a stacked chip structure of an imaging device. FIG. 11 is a block diagram showing an example of a configuration of a column processing unit of an imaging device according to the first configuration example. FIG. 12 is a block diagram showing an example of a configuration of an imaging device according to a first embodiment. FIG. 13 is a block diagram showing an example of a configuration of an imaging device according to the first embodiment. FIG. 14 is a diagram showing an example of a configuration of a first pixel group and a second pixel group according to the first embodiment. FIG. 15 is a timing chart showing an example of the operation of a first pixel, a second pixel, and a control unit according to the first embodiment. FIG. 16 is a diagram showing an example of a configuration of a storage unit according to the first embodiment. FIG. 17 is a flowchart showing an example of a calculation method of a control unit according to the first embodiment. FIG. 18 is a diagram showing an example of a configuration of a first pixel group and a second pixel group according to a second embodiment. 13 is a diagram showing an example of the configuration of a first pixel group and a second pixel group according to a second embodiment. FIG. 14 is a diagram showing an example of the configuration of a first pixel group and a second pixel group according to a third embodiment. FIG. 15 is a flowchart showing an example of a calculation method of a control unit according to a fourth embodiment. FIG. 16 is a block diagram showing an example of control of a control unit according to a fifth embodiment. FIG. 17 is a block diagram showing an example of control of a control unit according to a second modified example of the fifth embodiment. FIG. 18 is a block diagram showing an example of the configuration of an imaging device according to a sixth embodiment. FIG. 19 is a diagram showing an example of the positions of a light source and an imaging system according to a tenth embodiment. FIG. 20 is a block diagram showing an example of the configuration of a system according to a tenth embodiment. FIG. 21 is a timing chart showing an example of the operation of a first pixel, a second pixel, and a control unit according to a tenth embodiment.Fig. 20 is a diagram showing an example of the number of events in a pixel array unit according to a tenth embodiment. Fig. 21 is a schematic diagram showing an example of the overall configuration of an electronic device. Fig. 22 is a block diagram showing an example of the schematic configuration of a vehicle control system. Fig. 23 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit.

[0026] Hereinafter, embodiments of a light detection element and a system will be described with reference to the drawings. The following description will focus on the main components of the light detection element and the system, but the light detection element and the system may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0027] FIG. 1 is a block diagram showing an example of the system configuration of an imaging system to which the technology according to the present disclosure is applied.

[0028] 1, an imaging system 10 to which the technology according to the present disclosure is applied includes an imaging lens 11, an imaging device 20, a recording unit 12, and a control unit 13. The imaging system 10 is an example of an electronic device according to the present disclosure, and examples of such electronic devices include a camera system mounted on an industrial robot and an in-vehicle camera system.

[0029] In the imaging system 10 configured as described above, the imaging lens 11 captures incident light from a subject and forms an image on the imaging surface of the imaging device 20. The imaging device 20 obtains imaging data by photoelectrically converting the incident light captured by the imaging lens 11 on a pixel-by-pixel basis. An imaging device (photodetector element) according to the present disclosure, which will be described later, is used as this imaging device 20.

[0030] The imaging device 20 performs predetermined signal processing such as image recognition processing on the captured image data, and outputs data indicating the processing result and an address event detection signal (hereinafter simply referred to as a "detection signal") to the recording unit 12. A method for generating the address event detection signal will be described later. The recording unit 12 stores the data supplied from the imaging device 20 via a signal line 14. The control unit 13 is, for example, configured by a microcomputer, and controls the imaging operation of the imaging device 20.

[0031] [Imaging Device According to First Configuration Example (Arbiter System)] FIG. 2 is a block diagram showing an example of the configuration of an imaging device according to a first configuration example, which is used as the imaging device 20 in the imaging system 10 to which the technology according to the present disclosure is applied.

[0032] As shown in FIG. 2 , the imaging device 20 according to the first configuration example of the imaging device of the present disclosure is an asynchronous imaging device called EVS, and is configured to include a pixel array section 21, a drive section 22, an arbiter section (arbitration section) 23, a column processing section 24, and a signal processing section 25.

[0033] In the imaging device 20 having the above configuration, a plurality of pixels 30 are two-dimensionally arranged in a matrix (array) in the pixel array unit 21. Vertical signal lines VSL (described later) are wired for each pixel column in this matrix-like pixel arrangement.

[0034] Each of the plurality of pixels 30 generates a pixel signal, which is an analog signal of a voltage corresponding to the photocurrent. Each of the plurality of pixels 30 detects the presence or absence of an address event based on whether or not the amount of change in the photocurrent exceeds a predetermined threshold. When an address event occurs, the pixel 30 outputs a request to the arbiter unit 23.

[0035] The driving section 22 drives each of the plurality of pixels 30 to output a pixel signal generated in each pixel 30 to the column processing section 24 .

[0036] The arbiter unit 23 arbitrates requests from each of the multiple pixels 30 and transmits a response based on the arbitration result to the pixel 30. Upon receiving the response from the arbiter unit 23, the pixel 30 supplies a detection signal indicating the detection result (address event detection signal) to the drive unit 22 and the signal processing unit 25. The detection signal from the pixel 30 can also be read out from multiple rows.

[0037] The column processing unit 24 is composed of, for example, an analog-to-digital converter, and performs processing to convert, into digital signals, analog pixel signals output from the pixels 30 in each pixel column of the pixel array unit 21. Then, the column processing unit 24 supplies the digital signals after analog-to-digital conversion to the signal processing unit 25.

[0038] The signal processing unit 25 performs predetermined signal processing such as CDS (Correlated Double Sampling) processing and image recognition processing on the digital signal supplied from the column processing unit 24. Then, the signal processing unit 25 supplies data indicating the processing result and the detection signal supplied from the arbiter unit 23 to the recording unit 12 (see FIG. 1 ) via the signal line 14.

[0039] [Configuration Example of Pixel Array Unit] FIG. 3 is a block diagram showing an example of the configuration of the pixel array unit 21. As shown in FIG.

[0040] In the pixel array section 21, which is made up of a plurality of pixels 30 arranged two-dimensionally in a matrix, each of the plurality of pixels 30 has a light receiving section 31, a pixel signal generating section 32, and an address event detecting section 33.

[0041] In the pixel 30 having the above configuration, the light receiving unit 31 performs photoelectric conversion on incident light to generate a photocurrent. Then, under the control of the drive unit 22 (see FIG. 2 ), the light receiving unit 31 supplies the photocurrent generated by photoelectric conversion to either the pixel signal generation unit 32 or the address event detection unit 33.

[0042] The pixel signal generation unit 32 generates a pixel signal SIG, which is a voltage signal corresponding to the photocurrent supplied from the light receiving unit 31, and supplies this generated pixel signal SIG to the column processing unit 24 (see Figure 2) via the vertical signal line VSL.

[0043] The address event detector 33 detects whether an address event has occurred based on whether the amount of change in photocurrent from each of the light receiving units 31 has exceeded a predetermined threshold. An address event is, for example, an ON event indicating that the amount of change in photocurrent has exceeded an upper threshold, and an OFF event indicating that the amount of change has fallen below a lower threshold. The address event detection signal is, for example, one bit indicating the detection result of an ON event and one bit indicating the detection result of an OFF event. The address event detector 33 can also be configured to detect only ON events.

[0044] When an address event occurs, the address event detection unit 33 supplies a request for transmission of an address event detection signal to the arbiter unit 23 (see FIG. 2 ). Then, upon receiving a response to the request from the arbiter unit 23, the address event detection unit 33 supplies an address event detection signal to the drive unit 22 and the signal processing unit 25.

[0045] 4 is a circuit diagram showing an example of the circuit configuration of the pixel 30. As described above, each of the plurality of pixels 30 includes a light receiving unit 31, a pixel signal generating unit 32, and an address event detecting unit 33.

[0046] In the pixel 30 having the above configuration, the light receiving unit 31 includes a light receiving element (photoelectric conversion element) 311, a transfer transistor 312, and an OFG (Over Flow Gate) transistor 313. For example, an N-type MOS (Metal Oxide Semiconductor) transistor is used as the transfer transistor 312 and the OFG transistor 313. The transfer transistor 312 and the OFG transistor 313 are connected in series with each other.

[0047] The light receiving element 311 is connected between a common connection node N1 of the transfer transistor 312 and the OFG transistor 313 and the ground, and performs photoelectric conversion on incident light to generate charges of an amount corresponding to the amount of incident light.

[0048] 2 is supplied to the gate electrode of the transfer transistor 312. In response to the transfer signal TRG, the transfer transistor 312 supplies the charge photoelectrically converted by the light receiving element 311 to the pixel signal generation unit 32.

[0049] A control signal OFG is supplied from the driving unit 22 to the gate electrode of the OFG transistor 313. In response to the control signal OFG, the OFG transistor 313 supplies the electrical signal generated by the light receiving element 311 to the address event detection unit 33. The electrical signal supplied to the address event detection unit 33 is a photocurrent made up of electric charges.

[0050] The pixel signal generation unit 32 includes a reset transistor 321, an amplification transistor 322, a selection transistor 323, and a floating diffusion layer 324. The reset transistor 321, the amplification transistor 322, and the selection transistor 323 are, for example, N-type MOS transistors.

[0051] The pixel signal generation unit 32 receives charges photoelectrically converted by the light receiving element 311 from the light receiving unit 31 via the transfer transistor 312. The charges supplied from the light receiving unit 31 are accumulated in the floating diffusion layer 324. The floating diffusion layer 324 generates a voltage signal having a voltage value corresponding to the amount of accumulated charge. In other words, the floating diffusion layer 324 converts the charges into a voltage.

[0052] The reset transistor 321 is connected between the power supply line of the power supply voltage VDD and the floating diffusion layer 324. A reset signal RST is supplied to the gate electrode of the reset transistor 321 from the drive unit 22. The reset transistor 321 initializes (resets) the amount of charge in the floating diffusion layer 324 in response to the reset signal RST.

[0053] The amplifier transistor 322 is connected in series with the selection transistor 323 between the power supply line of the power supply voltage VDD and the vertical signal line VSL. The amplifier transistor 322 amplifies the voltage signal converted from charge to voltage by the floating diffusion layer 324.

[0054] A selection signal SEL is supplied to the gate electrode of the selection transistor 323 from the drive unit 22. In response to the selection signal SEL, the selection transistor 323 outputs the voltage signal amplified by the amplification transistor 322 as a pixel signal SIG to the column processing unit 24 (see FIG. 2) via the vertical signal line VSL.

[0055] In an imaging device 20 having a pixel array section 21 in which pixels 30 having the above-described configuration are arranged two-dimensionally, when the control section 13 shown in FIG. 1 instructs the drive section 22 to start detecting an address event, the drive section 22 supplies a control signal OFG to the OFG transistor 313 of the light receiving section 31, thereby driving the OFG transistor 313 and causing it to supply a photocurrent to the address event detection section 33.

[0056] When an address event is detected in a certain pixel 30, the drive unit 22 turns off the OFG transistor 313 of that pixel 30 to stop the supply of photocurrent to the address event detection unit 33. Next, the drive unit 22 drives the transfer transistor 312 by supplying a transfer signal TRG to the transfer transistor 312, causing the charge photoelectrically converted by the light receiving element 311 to be transferred to the floating diffusion layer 324.

[0057] In this way, the imaging device 20 having the pixel array unit 21 in which the pixels 30 having the above configuration are arranged two-dimensionally outputs only pixel signals of the pixels 30 in which an address event is detected to the column processing unit 24. This makes it possible to reduce the power consumption of the imaging device 20 and the amount of image processing compared to when pixel signals of all pixels are output regardless of whether an address event occurs.

[0058] Note that the configuration of the pixel 30 illustrated here is merely an example, and is not limited to this configuration example. For example, a pixel configuration not including the pixel signal generation unit 32 is also possible. In this pixel configuration, the OFG transistor 313 is omitted from the light receiving unit 31, and the function of the OFG transistor 313 is given to the transfer transistor 312.

[0059] [First Configuration Example of Address Event Detector] Fig. 5 is a block diagram showing a first configuration example of the address event detector 33. As shown in Fig. 5, the address event detector 33 according to this configuration example includes a current-voltage converter 331, a buffer 332, a subtractor 333, a quantizer 334, and a transfer unit 335.

[0060] The current-voltage converter 331 converts the photocurrent from the light receiving unit 31 of the pixel 30 into a logarithmic voltage signal. The current-voltage converter 331 supplies the converted voltage signal to the buffer 332. The buffer 332 buffers the voltage signal supplied from the current-voltage converter 331 and supplies it to the subtractor 333.

[0061] The subtractor 333 receives a row drive signal from the driver 22. The subtractor 333 reduces the level of the voltage signal supplied from the buffer 332 in accordance with the row drive signal. The subtractor 333 then supplies the voltage signal after the reduction in level to the quantizer 334. The quantizer 334 quantizes the voltage signal supplied from the subtractor 333 into a digital signal and outputs it to the transfer unit 335 as an address event detection signal.

[0062] The transfer unit 335 transfers the address event detection signal supplied from the quantizer 334 to the arbiter unit 23 etc. When an address event is detected, the transfer unit 335 supplies a request for transmission of the address event detection signal to the arbiter unit 23. Then, when the transfer unit 335 receives a response to the request from the arbiter unit 23, it supplies the address event detection signal to the drive unit 22 and the signal processing unit 25.

[0063] Next, an example of the configuration of the current-voltage converter 331, the subtractor 333, and the quantizer 334 in the address event detector 33 will be described.

[0064] 6 is a circuit diagram showing an example of the configuration of the current-voltage converter 331 in the address event detector 33. As shown in Fig. 6, the current-voltage converter 331 according to this example has a circuit configuration including an N-type transistor 3311, a P-type transistor 3312, and an N-type transistor 3313. These transistors 3311 to 3313 may be, for example, MOS transistors.

[0065] The N-type transistor 3311 is connected between the power supply line of the power supply voltage VDD and a signal input line 3314. The P-type transistor 3312 and the N-type transistor 3313 are connected in series between the power supply line of the power supply voltage VDD and ground. A common connection node N2 of the P-type transistor 3312 and the N-type transistor 3313 is connected to the gate electrode of the N-type transistor 3311 and the input terminal of the buffer 332 shown in FIG. 5.

[0066] A predetermined bias voltage Vbias is applied to the gate electrode of the P-type transistor 3312. This causes the P-type transistor 3312 to supply a constant current to the N-type transistor 3313. A photocurrent is input to the gate electrode of the N-type transistor 3313 from the light receiving unit 31 through the signal input line 3314.

[0067] The drain electrodes of the N-type transistor 3311 and the N-type transistor 3313 are connected to the power supply side, and this type of circuit is called a source follower. These two source followers connected in a loop convert the photocurrent from the light receiving unit 31 into a logarithmic voltage signal.

[0068] (Configuration Example of Subtractor and Quantizer) FIG. 7 is a circuit diagram showing an example of the configuration of the subtractor 333 and the quantizer 334 in the address event detector 33. As shown in FIG.

[0069] The subtractor 333 according to this example includes a capacitance element 3331 , an inverter circuit 3332 , a capacitance element 3333 , and a switch element 3334 .

[0070] 5, and the other end is connected to the input terminal of the inverter circuit 3332. The capacitor 3333 is connected in parallel to the inverter circuit 3332. The switch element 3334 is connected across the capacitor 3333. A row drive signal is supplied from the drive unit 22 to the switch element 3334 as an opening / closing control signal. The switch element 3334 opens and closes a path connecting both ends of the capacitor 3333 in response to the row drive signal. The inverter circuit 3332 inverts the polarity of the voltage signal input via the capacitor 3331.

[0071] In the subtractor 333 configured as described above, when the switch element 3334 is turned on (closed), a voltage signal Vinit is input to the terminal of the capacitance element 3331 on the buffer 332 side, and the terminal on the opposite side serves as a virtual ground terminal. For convenience, the potential of this virtual ground terminal is set to zero. At this time, the charge Qinit accumulated in the capacitance element 3331 is expressed by the following equation (1), where the capacitance value of the capacitance element 3331 is C1. On the other hand, since both ends of the capacitance element 3333 are short-circuited, the accumulated charge is zero. Qinit = C1 × Vinit (1)

[0072] Next, when the switch element 3334 is turned off (open) and the voltage at the terminal of the capacitance element 3331 on the buffer 332 side changes to Vafter, the charge Qafter stored in the capacitance element 3331 is expressed by the following equation (2): Qafter=C1×Vafter (2)

[0073] On the other hand, the charge Q2 stored in the capacitance element 3333 is expressed by the following equation (3), where C2 is the capacitance of the capacitance element 3333 and Vout is the output voltage. Q2=-C2×Vout (3)

[0074] At this time, the total amount of charge of the capacitance elements 3331 and 3333 does not change, so the following equation (4) holds: Qinit=Qafter+Q2 (4)

[0075] By substituting equations (1) to (3) into equation (4) and rearranging it, the following equation (5) is obtained: Vout=-(C1 / C2)×(Vafter-Vinit) (5)

[0076] Equation (5) represents the subtraction operation of the voltage signal, and the gain of the subtraction result is C1 / C2. Since it is usually desirable to maximize the gain, it is preferable to design C1 large and C2 small. On the other hand, if C2 is too small, kTC noise increases and noise characteristics may deteriorate, so the reduction in the capacitance of C2 is limited to a range that allows for noise. Furthermore, since an address event detection unit 33 including a subtractor 333 is mounted on each pixel 30, there are area restrictions on the capacitance elements 3331 and 3333. The capacitance values ​​C1 and C2 of the capacitance elements 3331 and 3333 are determined taking these factors into consideration.

[0077] 7, the quantizer 334 has a configuration including a comparator 3341. The comparator 3341 receives the output signal of the inverter circuit 3332, i.e., the voltage signal from the subtractor 333, as its non-inverting (+) input, and receives a predetermined threshold voltage Vth as its inverting (-) input. The comparator 3341 compares the voltage signal from the subtractor 333 with the predetermined threshold voltage Vth, and outputs a signal indicating the comparison result to the transfer unit 335 as an address event detection signal.

[0078] [Second Configuration Example of Address Event Detector] Fig. 8 is a block diagram showing a second configuration example of the address event detector 33. As shown in Fig. 8, the address event detector 33 according to this configuration example includes a current-voltage converter 331, a buffer 332, a subtractor 333, a quantizer 334, and a transfer unit 335, as well as a storage unit 336 and a control unit 337.

[0079] The storage unit 336 is provided between the quantizer 334 and the transfer unit 335, and stores the output of the quantizer 334, i.e., the comparison result of the comparator 3341, based on a sample signal supplied from the control unit 337. The storage unit 336 may be a sampling circuit such as a switch, plastic, or capacitor, or may be a digital memory circuit such as a latch or flip-flop.

[0080] The control unit 337 supplies a predetermined threshold voltage Vth to the inverting (−) input terminal of the comparator 3341. The threshold voltage Vth supplied from the control unit 337 to the comparator 3341 may have different voltage values ​​in a time-division manner. For example, the control unit 337 supplies, at different timings, a threshold voltage Vth1 corresponding to an ON event indicating that the amount of change in photocurrent has exceeded an upper threshold, and a threshold voltage Vth2 corresponding to an OFF event indicating that the amount of change has fallen below a lower threshold, thereby enabling a single comparator 3341 to detect multiple types of address events.

[0081] For example, the storage unit 336 may store the comparison result of the comparator 3341 using the threshold voltage Vth1 corresponding to an ON event during a period in which the control unit 337 supplies the threshold voltage Vth2 corresponding to an OFF event to the inverting (−) input terminal of the comparator 3341. The storage unit 336 may be located inside the pixel 30 or outside the pixel 30. The storage unit 336 is not an essential component of the address event detection unit 33. In other words, the storage unit 336 may be omitted.

[0082] [Imaging Device According to Second Configuration Example (Scanning Method)] The imaging device 20 according to the first configuration example described above is an asynchronous imaging device that reads events using an asynchronous reading method. However, the event reading method is not limited to the asynchronous reading method and may be a synchronous reading method. An imaging device to which the synchronous reading method is applied is a scanning imaging device, the same as a normal imaging device that captures images at a predetermined frame rate.

[0083] FIG. 9 is a block diagram showing an example of the configuration of an imaging device according to the second configuration example, that is, a scanning type imaging device, which is used as the imaging device 20 in the imaging system 10 to which the technology according to the present disclosure is applied.

[0084] As shown in Figure 9, the imaging device 20 according to the second configuration example of the imaging device of the present disclosure is configured to include a pixel array section 21, a drive section 22, a signal processing section 25, a readout area selection section 27, and a signal generation section 28.

[0085] The pixel array unit 21 includes a plurality of pixels 30. The plurality of pixels 30 output an output signal in response to a selection signal from the readout region selection unit 27. Each of the plurality of pixels 30 may be configured to have a quantizer within the pixel, as shown in FIG. 7, for example. The plurality of pixels 30 output an output signal corresponding to the amount of change in light intensity. The plurality of pixels 30 may be arranged two-dimensionally in a matrix, as shown in FIG. 9.

[0086] The driver 22 drives each of the plurality of pixels 30 to output a pixel signal generated by each pixel 30 to the signal processor 25. The driver 22 and the signal processor 25 are circuit units for acquiring gradation information. Therefore, when only event information is to be acquired, the driver 22 and the signal processor 25 may be omitted.

[0087] The readout region selection unit 27 selects a portion of the plurality of pixels 30 included in the pixel array unit 21. For example, the readout region selection unit 27 selects one or more rows from among the rows included in the two-dimensional matrix structure corresponding to the pixel array unit 21. The readout region selection unit 27 sequentially selects one or more rows according to a preset cycle. The readout region selection unit 27 may also determine the selection region in response to a request from each pixel 30 in the pixel array unit 21.

[0088] The signal generation unit 28 generates an event signal corresponding to an active pixel that has detected an event among the selected pixels, based on the output signals of the pixels selected by the readout region selection unit 27. An event is an event in which the intensity of light changes. An active pixel is a pixel in which the amount of change in the intensity of light corresponding to the output signal exceeds or falls below a predetermined threshold. For example, the signal generation unit 28 compares the output signal of the pixel with a reference signal, detects an active pixel that outputs an output signal when the output signal is greater or smaller than the reference signal, and generates an event signal corresponding to the active pixel.

[0089] The signal generation unit 28 may be configured to include, for example, a column selection circuit that arbitrates signals coming into the signal generation unit 28. Furthermore, the signal generation unit 28 may be configured to output not only information about active pixels that have detected an event, but also information about inactive pixels that have not detected an event.

[0090] The signal generation unit 28 outputs address information and timestamp information (e.g., (X, Y, T)) of the active pixel in which an event is detected through the output line 15. However, the data output from the signal generation unit 28 may be not only address information and timestamp information but also frame format information (e.g., (0, 0, 1, 0, ...)).

[0091] [Configuration Example of Chip Structure] The chip (semiconductor integrated circuit) structure of the imaging device 20 according to the first or second configuration example described above can be, for example, a stacked chip structure. Fig. 10 is an exploded perspective view showing an outline of the stacked chip structure of the imaging device 20.

[0092] 10, the stacked chip structure, or so-called stacked structure, is a structure in which at least two chips, a light-receiving chip 201 which is a first chip, and a detection chip 202 which is a second chip, are stacked. In the circuit configuration of the pixel 30 shown in FIG. 4, each of the light-receiving elements 311 is disposed on the light-receiving chip 201, and all elements other than the light-receiving elements 311 and elements of other circuit portions of the pixel 30 are disposed on the detection chip 202. The light-receiving chip 201 and the detection chip 202 are electrically connected via connecting portions such as vias, Cu-Cu joints, and bumps.

[0093] Here, an example of a configuration has been given in which the light receiving element 311 is arranged on the light receiving chip 201, and elements other than the light receiving element 311 and elements of other circuit parts of the pixel 30 are arranged on the detection chip 202, but this configuration example is not limited to this.

[0094] 4, it is possible to arrange the elements of the light receiving unit 31 on the light receiving chip 201, and elements other than the light receiving unit 31 and elements of other circuit parts of the pixel 30 on the detection chip 202. It is also possible to arrange the elements of the light receiving unit 31, the reset transistor 321 of the pixel signal generation unit 32, and the floating diffusion layer 324 on the light receiving chip 201, and the other elements on the detection chip 202. It is also possible to arrange some of the elements that make up the address event detection unit 33 on the light receiving chip 201 together with the elements of the light receiving unit 31.

[0095] 11 is a block diagram showing an example of the configuration of the column processing unit 24 of the imaging device 20 according to the first configuration example. As shown in Fig. 11, the column processing unit 24 according to this example has a configuration including a plurality of analog-to-digital converters (ADCs) 241 arranged for each pixel column of the pixel array unit 21.

[0096] Although the configuration example in which the analog-to-digital converters 241 are arranged in a one-to-one correspondence with the pixel columns of the pixel array unit 21 has been exemplified here, the present invention is not limited to this configuration example. For example, a configuration in which the analog-to-digital converters 241 are arranged for each of a plurality of pixel columns and the analog-to-digital converters 241 are used in a time-division manner among the plurality of pixel columns may also be adopted.

[0097] The analog-to-digital converter 241 converts the analog pixel signal SIG supplied via the vertical signal line VSL into a digital signal having a larger number of bits than the address event detection signal described above. For example, if the address event detection signal is 2 bits, the pixel signal is converted into a digital signal having 3 or more bits (e.g., 16 bits). The analog-to-digital converter 241 supplies the digital signal generated by the analog-to-digital conversion to the signal processing unit 25.

[0098] First Embodiment FIG. 12 is a block diagram showing an example of the configuration of a system 1 according to a first embodiment.

[0099] The system 1 is a system that emits and receives light and performs processing based on the light reception results. The system 1 is, for example, a visible light communication (VLC) or an event-based structured light (ESL) system. The system 1 includes a light source 2 and an imaging system 10.

[0100] The imaging system 10 corresponds to the imaging system 10 shown in FIG.

[0101] The light source 2 emits light that can be received by the imaging system 10. The light source 2 transmits information by driving light pulses at different frequencies. The light source 2 emits, for example, visible light. The light source 2 is, for example, an LED (Light Emitting Diode).

[0102] FIG. 13 is a block diagram showing an example of the configuration of the imaging device 20 according to the first embodiment.

[0103] The imaging device 20 includes an event detection device 411 and a timing control device 420 .

[0104] The event detection device 411 includes a pixel array unit 21 , a latch circuit 412 , a row arbiter 413 , a row AER circuit 414 , a row address encoder 415 , an event readout unit 416 , a current detection unit 417 , and a threshold control unit 418 .

[0105] The pixel array section 21 and the pixels 30 in the pixel array section 21 correspond to the pixel array section 21 and the pixels 30 shown in Figures 2 and 3. Note that the pixels 30 do not necessarily need to be provided with the pixel signal generation section 32 shown in Figure 3.

[0106] The pixels 30 include first pixels 30A and second pixels 30B. The multiple pixels 30 are divided into a first pixel group GA including multiple first pixels 30A and a second pixel group GB including multiple second pixels 30B. The first pixels 30A are, for example, dispersedly arranged in the pixel array section 21. The second pixels 30B are, for example, arranged around the first pixels 30A. The arrangement of the first pixels 30A and the second pixels 30B shown in FIG. 13 is one example.

[0107] The address event detector 33 in the first pixel 30A is called a first event detector 33A, and the address event detector 33 in the second pixel 30B is called a second event detector 33B.

[0108] The first event detector 33A detects a change in the luminance of incident light at the first pixel 30A as a first event. The first event detector 33A continues to detect the first event. That is, the first event detector 33A constantly detects the first event.

[0109] The second event detector 33B detects a change in the luminance of the incident light at the second pixel 30B as a second event at a timing based on the detected first event.

[0110] The latch circuit 412 temporarily holds the states of all pixels in a row in which an address event has occurred, based on a request from the event readout unit 416 .

[0111] Furthermore, the latch circuit 412 transmits the temporarily held data based on a request from the event reading unit 416 requesting external output of the address event.

[0112] The row arbiter 413 arbitrates requests from the row AER circuit 414 and transmits a response to the row AER circuit 414 based on the arbitration result.

[0113] The row AER circuit 414 transmits and receives (handshakes) requests and responses requesting external output of address event detection results between each row, the row arbiter 413, and the event readout unit 416.

[0114] The row address encoder 415 encodes the address of the row in which the address event occurred and transmits it to the event readout unit 416 .

[0115] The event readout unit 416 performs handshake between the latch circuit 412 and the row AER circuit 414. When the event readout unit 416 receives a request from the row AER circuit 414, it decodes the data from the latch circuit 412 and the row address encoder 415 and identifies the address where the address event was detected. Image data is generated by arranging the address event detection results for each pixel in a two-dimensional grid.

[0116] The event readout unit 416 also performs a handshake with the timing control device 420. The event readout unit 416 transmits, for example, an event signal detected in the pixel 30 to the timing control device 420. The event readout unit 416 receives a detection start signal and a detection end signal from the timing control device 420. The detection start signal is a signal that causes the second pixel 30B to start detecting the second event. The detection end signal is a signal that causes the second pixel 30B to end detecting the second event.

[0117] The current detection section 417 detects the sum of the photocurrents of all the pixels as a total current, and supplies this to a threshold control section 418 .

[0118] The threshold control unit 418 controls the upper and lower thresholds to values ​​according to the total current. For example, the higher the total current, the higher the thresholds are controlled to be. The upper and lower thresholds are each controlled by a threshold voltage V ｏｎ , V ｏｆｆ Corresponds to.

[0119] The timing control device 420 controls the timing of event detection for the second pixel 30 B. The timing control device 420 includes a control unit 421 and a storage unit 422 .

[0120] The control unit 421 controls the second event detection unit 33B to detect a second event at a timing based on the detected first event. The control unit 421 receives an event signal (first event) from the event reading unit 416. The control unit 421 determines the detection timing at which the second event detection unit 33B detects the second event based on the detected first event. The control unit 421 controls the second event detection unit 33B based on the determined detection timing. The control unit 421 generates, for example, signals related to the detection timing (a detection start signal and a detection end signal) and outputs them to the second event detection unit 33B. The control unit 421 receives the event signal (second event) from the event reading unit 416. The control unit 421 may be configured with a single processor, such as a single CPU (Central Processing Unit) or multiple CPUs (Central Processing Units).

[0121] More specifically, the control unit 421 calculates the entropy (amount of information) of the detected first event. The control unit 421 determines the detection timing based on the calculated entropy. Details of the entropy calculation will be described later with reference to FIGS. 17 and 18.

[0122] Furthermore, the control unit 421 outputs the first event and the second event to the outside of the imaging device 20 .

[0123] The storage unit 422 stores information necessary for the processing of the control unit 421 .

[0124] 14 and 15 are diagrams showing examples of the configurations of the first pixel group GA and the second pixel group GB according to the first embodiment. Fig. 14 shows a case where the second event is not detected in the second pixel group GB. Fig. 15 shows a case where the second event is detected in the second pixel group GB.

[0125] The first pixel group GA includes M first pixels 30 A. One row AER circuit 414 is provided for the M first pixels 30 A.

[0126] 14 and 15 includes a light receiving element 311, a current-voltage converter 331, a subtractor 333, and a quantizer 334. ｐ is the output voltage of the current-voltage converter 331. ｄｉｆｆ is the output voltage of the subtractor 333.

[0127] 14 and 15 correspond to the configurations described with reference to FIGS. 5 to 7. The quantizer 334 has two comparators 3341. The two comparators 3341 have threshold voltages Vth corresponding to an ON event and an OFF event, respectively. ｏｎ , V ｏｆｆ It has.

[0128] The row AER circuit 414 receives the output of the quantizer 334, i.e., the comparison result of the comparator 3341. The row AER circuit 414 receives the auto-zero signal Φ ＡＺ is sent to the switch element 3334. The auto-zero signal Φ ＡＺ is an open / close control signal for the switch element 3334.

[0129] The second pixel group GB includes N second pixels 30 B. One row AER circuit 414 is provided for the N second pixels 30 B.

[0130] The switch element 3334 of the second pixel 30B is a switch that initializes the output of the subtractor 333. The switch element 3334 turns on or off at a timing based on the detected first event. The switch element 3334 turns on when the second event detection unit 33B does not detect the second event, and turns off when the second event detection unit 33B detects the second event.

[0131] Compared to the subtractor 333 of the first pixel 30A, the subtractor 333 of the second pixel 30B further includes a switch element 3335. The switch element 3335 is connected between the output part of the inverter circuit 3332 and a node between the capacitance element 3331 and the input part of the inverter circuit 3332. The switch element 3335 is connected in parallel with the switch element 3334. The switch element 3335 opens and closes in response to opening and closing control signals (entropy trigger, detection start signal, and detection end signal) output from the control unit 421.

[0132] The first events detected in the M first pixels 30A are shown on the same time axis in the graph of Fig. 14. The vertical axis of the graph indicates ON events and OFF events, and the horizontal axis of the graph indicates time t.

[0133] In the first pixel 30A shown in FIG. 14, the first event is constantly detected.

[0134] 14 turns on the switch element 3335 based on the entropy of the first event. This causes the second pixel 30B to stop detecting the second event. As shown in the graph in FIG. 14, the second event is not detected in the second pixel group GB.

[0135] The first events detected at the M first pixels 30A are shown in the graph of FIG. 15 on the same time axis.

[0136] In the first pixel 30A shown in FIG. 15, the first event is constantly detected.

[0137] 15 turns off the switch element 3335 based on the entropy of the first event, causing the second pixel 30B to start detecting the second event.

[0138] The second events detected at the N second pixels 30B are shown in the graph of FIG. 15 on the same time axis.

[0139] In the second pixel group GB shown in FIG. 15, a second event is detected.

[0140] 14 and 15, the control unit 421 sets the second event detection unit 33B in a state where it cannot detect the second event, depending on the entropy of the first event, thereby making it possible to more appropriately reduce the amount of data.

[0141] Next, the operations of the first pixel 30A, the second pixel 30B, and the control unit 421 regarding event detection control will be described in detail.

[0142] 16 is a timing chart showing an example of the operation of the first pixel 30A, the second pixel 30B, and the control unit 421 according to the first embodiment. The left side of FIG. 16 shows the light emission intensity of the light source 2, the voltage V ｐ , V ｄｉｆｆ , auto-zero signal Φ ＡＺ , and a timing chart of events. The right side of FIG. 16 shows the light emission intensity of the light source 2, the voltage V ｐ , V ｄｉｆｆ , auto-zero signal Φ ＡＺ , an entropy trigger, an entropy, and an event timing chart are shown. Also, the light intensity and voltage V of the light source 2 are shown. Ｐ The behavior of is common between the left and right sides of FIG.

[0143] In addition, in the event timing chart shown on the left side of Fig. 16, first events detected in multiple first pixels 30A in the first pixel group GA are shown on the same time axis, and in the event timing chart shown on the right side of Fig. 16, second events detected in multiple second pixels 30B in the second pixel group GB are shown on the same time axis.

[0144] First, the operation of the first pixel 30A will be described.

[0145] First, at time t1, the light source 2 is turned on and the light emission intensity of the light source 2 increases up to a predetermined light emission intensity.

[0146] The light receiving element 311 receives light from the light source 2 and generates a photocurrent. The current-voltage converter 331 converts the generated photocurrent into a logarithmic voltage signal. As a result, the voltage V ｐ will rise.

[0147] The subtractor 333 subtracts the voltage V ｐ and voltage V ｒｅｆ The difference voltage V ｄｉｆｆ Output.

[0148] The quantizer 334 converts the voltage V ｄｉｆｆ and the threshold voltage V ｏｎ , V ｏｆｆ Compare with the voltage V ｄｉｆｆ is the threshold voltage V ｏｎ When the first event detector 33A detects the first on-event, the row AER circuit 414 receives the detected first on-event and outputs a high auto-zero signal Φ ＡＺ (pulse signal) is transmitted. This causes the voltage V ｄｉｆｆ will be reset.

[0149] The event reader 416 outputs the first event of the detected on-event.

[0150] Next, at time t2, the light source 2 is turned off, and the light emission intensity of the light source 2 decreases. The light emission intensity of the light source 2 decreases to zero.

[0151] Since the amount of light received by the light receiving element 311 decreases, the voltage V ｐ decreases.

[0152] Voltage V ｄｉｆｆ is the threshold voltage V ｏｆｆ When the first event detector 33A detects the first off event, the row AER circuit 414 receives the detected first off event and outputs a high auto-zero signal Φ ＡＺ (pulse signal) is transmitted. This causes the voltage V ｄｉｆｆ will be reset.

[0153] The event reader 416 outputs the first event of the detected OFF event.

[0154] Next, at time t3, the voltage V ｐ returns to the initial voltage.

[0155] Next, the operation of the second pixel 30B and the control unit 421 will be described. Note that times t1, t2, and t3 are common between the left and right sides of FIG. 16. In addition, the light source 2 and the voltage V ｐ The change in is the same between the first pixel 30A and the second pixel 30B.

[0156] Initially, the entropy trigger is high. Therefore, the voltage V ｄｉｆｆ is approximately zero, and no second event is detected at the second pixel 30B.

[0157] The control unit 421 calculates the entropy based on the first event detected in the first pixel 30A.

[0158] The control unit 421 calculates the entropy of the detected first event at intervals of the period T. The control unit 421 calculates the entropy of the detected first event at intervals of the period T. ｏｎ , H ｏｆｆ Determine whether it is below

[0159] The entropy on the ON side is at threshold H ｏｎ When the value falls below t Ｓ１ ), the entropy trigger goes from high to low, which initiates detection of a second event at the second pixel 30B.

[0160] During the period when the entropy trigger is low, the voltage V ｄｉｆｆ is the threshold voltage V ｏｎ When the second event detector 33B detects the second on-event, the row AER circuit 414 receives the detected second on-event and outputs a high auto-zero signal Φ ＡＺ (pulse signal) is transmitted. This causes the voltage V ｄｉｆｆ will be reset.

[0161] The event reading unit 416 outputs a second event of the detected ON event.

[0162] The entropy on the ON side is at threshold H ｏｎ When the time exceeds the start time of the next period T (time t Ｅ１ ), the entropy trigger goes from low to high, which ends the detection of the second event at the second pixel 30B.

[0163] The entropy on the off side is at threshold H ｏｆｆ When the value falls below t Ｓ２ ), the entropy trigger goes from high to low, which initiates detection of a second event at the second pixel 30B.

[0164] During the period when the entropy trigger is low, the voltage V ｄｉｆｆ is the threshold voltage V ｏｆｆ When the second event detector 33B detects the second off event, the row AER circuit 414 receives the detected second off event and outputs a high auto-zero signal Φ ＡＺ (pulse signal).

[0165] The event reader 416 outputs a second event of the detected OFF event.

[0166] Next, the entropy calculation method will be described.

[0167] Probability P for any variable j ｊ is the counter value C ｊ , C ＭＡＸ It is expressed by the formula (6) using P ｊ =C ｊ / C ＭＡＸ ...(6) Here, the counter value C ｊ is the number of times the event is detected. ＭＡＸ is the maximum number of events that can be detected within a period T (time frame).

[0168] Entropy H Ａ is the probability P for any variable n ｎ is expressed by equation (7). Here, N is the unit in which entropy is calculated.

[0169] In addition, the entropy H Ａ In order to reduce the amount of calculation required for the calculation of (1), the following formula is transformed:

[0170] Entropy H Ａ is expressed by equation (8). log C in the second term on the right side of Equation 8 ＭＡＸ is the coefficient A = log C ＭＡＸ The log C in the third term on the right side of Equation 8 is a value calculated in advance. ｎ It is possible to speed up the calculation by using a look-up table (LUT) or approximating it with a quadratic function.

[0171] Entropy H for time t-1 ｔ－１ , and the entropy H for time t ｔ are expressed using Equations 9 and 10, respectively.

[0172] The first and second terms on the third side are common between Equation 4 and Equation 5. Therefore, Equation (11) can be obtained from Equation (9) and Equation (10). H ｔ = H ｔ－１ + (AC Ｎ -C Ｎ log C Ｎ )-(AC ０ -C ０ log C ０ ) ... (11) The second term on the right side of equation 11 is Z Ｎ (Z Ｎ =AC Ｎ -C Ｎ log C Ｎ The third term on the right side of Equation 11 is expressed as Z ０ (Z ０ =AC ０ -C ０ log C ０ )

[0173] By using the recursive update formula for the entropy sequence shown in Equation (11), Z Ｎ and Z ０ The entropy can be calculated with a reduced amount of calculation using

[0174] FIG. 17 is a diagram showing an example of the configuration of the storage unit 422 according to the first embodiment.

[0175] The storage unit 422 is a first-in first-out (FIFO) buffer that stores N pieces of data.

[0176] Z calculated by the control unit 421 Ｎ The new Z is stored in the FIFO buffer. Ｎ When the old Z ０ is retrieved from the storage unit 422. The control unit 421 Ｎ and Z ０ The entropy is calculated by substituting into equation (11).

[0177] 18 is a flowchart showing an example of a calculation method of the control unit 421 according to the first embodiment. The control unit 421 controls the timing of detecting the second event by clock-driven control.

[0178] First, the control unit 421 initializes the reference time Ts (Ts=0), the counter value C (C=0), the entropy H (H=0), the FIFO buffer, and the coefficient A=log C. ＭＡＸ is calculated (S10).

[0179] Next, the control unit 421 waits until a certain period of time T has elapsed (S20).

[0180] Next, the control unit 421 counts the number of events detected in the first pixel group GA within a period (time frame) Ts to Ts+T (S30). The control unit 421 also assigns the number of detected events to a counter value C, and assigns Ts+T to a reference time Ts (Ts=Ts+T).

[0181] Next, the control unit 421 calculates the entropy using equation (11) (S40). Ｎ Substitute A, C-C, log C into (Z Ｎ =A・C−C・logC), Z Ｎ is stored in the FIFO buffer and retrieved. ０ Obtain the entropy H and add H+ZＮ +Z ０ Substitute (H = H + Z Ｎ +Z ０ ).

[0182] Next, the control unit 421 determines whether the entropy H is H<C ＭＡＸ ・H Ｔｈ It is determined whether or not the following is satisfied (S50). ＭＡＸ ・H Ｔｈ is the threshold of entropy H.

[0183] Entropy H is H<C ＭＡＸ ・H Ｔｈ If the condition (NO in step S50) is not satisfied, the control unit 421 assigns 1 to the entropy trigger (S60). After that, step S20 is executed again.

[0184] Entropy H is H<C ＭＡＸ ・H Ｔｈ If the answer is YES in step S50, the control unit 421 assigns 0 to the entropy trigger (S70). The time when the entropy trigger changes from 1 to 0 is defined as time t Ｓ Then, step S20 is executed again.

[0185] Next, when step S60 is executed, the control unit 421 assigns 1 to the entropy trigger (S60). The time when the entropy trigger changes from 0 to 1 is defined as time t Ｅ As shown in FIG. Ｓ From time t Ｅ In the period up to, an event is detected in the second pixel group GB.

[0186] As described above, according to the first embodiment, the first event detection unit 33A detects a change in the luminance of incident light at the first pixel 30A as a first event. The second event detection unit 33B detects a change in the luminance of incident light at the second pixel 30B as a second event at a timing based on the detected first event. This allows for more appropriate reduction in the amount of data. As a result, power consumption can be reduced.

[0187] In the first embodiment, the timing of detecting the second event is determined by the entropy of the first event. When the entropy is high, the detected first event is highly random. Therefore, the detected first event may be meaningless. On the other hand, when the entropy is low, the detected first event is highly regular. Therefore, the detected first event may be meaningful. By using the entropy of the first event, the amount of data can be reduced more appropriately.

[0188] The control unit 421 also calculates the entropy based on, for example, the sum of the first events in the first pixel group GA. Therefore, the second event detection unit 33B detects the second event at a timing based on the first event detected in the first pixel group GA.

[0189] In the first embodiment, both on-events and off-events are used to calculate entropy and control the second pixel group GB, but this is not limiting, and either on-events or off-events may be used.

[0190] Furthermore, the plurality of pixels 30 may include three or more types of pixel groups, without being limited to the first pixel group GA and the second pixel group GB.

[0191] Alternatively, an event rate may be used instead of entropy. The control unit 421 calculates an event rate of the first events detected in a predetermined period. The control unit 421 determines the detection timing based on the calculated event rate.

[0192] Furthermore, the imaging system 10 and imaging device 20 (photodetector element) according to the first embodiment may be incorporated into the system 1 as described above, or may be an independent imaging system and imaging device.

[0193] As a comparative example, an event rate may be estimated based on the number of detected events, and the detected events may be filtered based on a comparison between the estimated event rate and a threshold. The event rate is the number of events detected within a predetermined period. The filter discards some of the detected events without outputting them. This reduces the amount of data. However, because all pixels are always in a state where events can be detected, a large number of events may be detected in dynamic scenes where the EVS sensor itself is moving, requiring a large amount of calculation for event processing. Furthermore, if events are discarded randomly, informative events may also be discarded. An informative event is, for example, an event (meaningful event) required for communication such as visible light communication (VLC) or any processing using the detected events.

[0194] In contrast, in the first embodiment, the timing for detecting a second event is determined based on the entropy of the first event. This prevents detection of a second event in the second pixel group GB when the entropy of the first event is high and highly random. This enables detection of few, meaningful events even in dynamic scenes where the EVS sensor itself moves at high speed. Furthermore, detection of a second event in the second pixel group GB is performed when the entropy of the first event is low and highly regular. As a result, informative events are more appropriately detected and output, enabling more appropriate reduction in data volume.

[0195] 19 and 20 are diagrams showing an example of the configuration of a first pixel group GA and a second pixel group GB according to a second embodiment. Fig. 19 shows a case where detection of a second event is not performed in the second pixel group GB. Fig. 20 shows a case where detection of a second event is performed in the second pixel group GB.

[0196] In the second embodiment, the auto-zero signal Φ output by the row AER circuit 414 is ＡＺ This embodiment differs from the first embodiment in that the timing of detecting the second event is controlled by the

[0197] The row AER circuit (switch control unit) 414 controls the switch element 3334 based on the detected first event. The row AER circuit 414 receives the entropy trigger from the control unit 421 and outputs an auto-zero signal Φ ＡＺ Output.

[0198] The second pixel 30B according to the second embodiment does not include the switch element 3335. That is, the switch element 3335 can be omitted.

[0199] As in the second embodiment, the auto-zero signal Φ output by the row AER circuit 414 ＡＺ The timing of detecting the second event may be controlled by: In this case as well, the same effects as in the first embodiment can be obtained.

[0200] 21 is a diagram showing an example of the configuration of a first pixel group GA and a second pixel group GB according to a third embodiment. The third embodiment differs from the first embodiment in that the polarity of an event is output from the control unit 421.

[0201] The second event detector 33B detects a second ON event at a timing based on the detected first ON event, and a second OFF event at a timing based on the detected first OFF event.

[0202] 21 does not include the switch element 3335 as described in the second embodiment. However, the switch element 3335 may be provided as in the first embodiment.

[0203] The quantizer 334 includes one comparator 3341 .

[0204] The quantizer 334 further includes a switching unit 3342. The switching unit 3342 switches a threshold (threshold voltage V ｏｎ , V ｏｆｆThe switching unit 3342 is electrically connected to the inverting (−) input terminal of the comparator 3341. The switching unit 3342 switches the connection destination of the inverting (−) input terminal of the comparator 3341 between the threshold voltage V ｏｎ or threshold voltage V ｏｆｆ The polarity of the event represents an increase or decrease in brightness. If the brightness is increasing, the polarity is +1 (on), and if the brightness is decreasing, the polarity is -1 (off).

[0205] In the first embodiment, when the entropy of the first event of an ON event is low, not only the second event of an ON event but also the second event of an OFF event may be detected. Similarly, when the entropy of the first event of an OFF event is low, not only the second event of an OFF event but also the second event of an ON event may be detected.

[0206] In contrast, in the third embodiment, the polarity of an event detected as a second event can be limited. This prevents detection of a second OFF event when the entropy of a first ON event is low and the entropy of a first OFF event is high. Similarly, prevents detection of a second ON event when the entropy of a first ON event is high and the entropy of a first OFF event is low.

[0207] As in the third embodiment, the polarity of the event may be output from the control unit 421. In this case, the same effects as in the first embodiment can be obtained.

[0208] <Fourth Embodiment> Fig. 22 is a flowchart showing an example of a calculation method of the control unit 421 according to the fourth embodiment. In the third embodiment, the entropy calculation method is different from that in the first embodiment. The control unit 421 controls the timing of detecting the second event by event-driven control. Note that the main differences between the flowchart shown in Fig. 22 and the flowchart shown in Fig. 18 are steps S120 to S150 and S200.

[0209] First, similarly to step S10, the control unit 421 initializes the reference time Ts (Ts=0), the counter value C (C=0), the entropy H (H=0), the FIFO buffer, and the coefficient A=log C. ＭＡＸ is calculated (S110).

[0210] Next, the control unit 421 waits for an event to be detected in the first pixel group GA (S120).

[0211] Next, the control unit 421 detects the event e ｊ Event e is acquired (S130). ｊ (e ｊ = (x ｊ , y ｊ , p ｊ , t ｊ )) is the position of the event x ｊ , y ｊ , polarity of the event p ｊ , and the occurrence time (timestamp) of the event t ｊ Includes.

[0212] Next, the control unit 421 determines whether Ts+T<t ｊ It is determined whether the following is satisfied (S140).

[0213] Ts+T<t ｊ If the condition is satisfied (YES in step S140), the control unit 421 counts the number of events (S150). The control unit 421 assigns C+1 to the counter value C. Thereafter, step S120 is executed again.

[0214] Ts+T<t ｊIf Z is not satisfied (NO in step S140), the control unit 421 calculates the entropy using equation (11) in the same manner as in step S40. ｎ Substitute A, C-C, log C into (Z Ｎ =A・C−C・logC), Z Ｎ is stored in the FIFO buffer and retrieved. ０ Obtain the entropy H and add H+Z Ｎ +Z ０ Substitute (H = H + Z Ｎ +Z ０ ).

[0215] Next, similarly to step S50, the control unit 421 determines whether the entropy H is H<C ＭＡＸ ・H Ｔｈ It is determined whether or not the following is satisfied (S170). ＭＡＸ ・H Ｔｈ is the threshold of entropy H.

[0216] Entropy H is H<C ＭＡＸ ・H Ｔｈ If the condition (NO in step S170) is not satisfied, the control unit 421 assigns 1 to the entropy trigger (S180), similarly to step S60.

[0217] Entropy H is H<C ＭＡＸ ・H Ｔｈ If the above condition is met (YES in step S170), the control unit 421 assigns 0 to the entropy trigger (S190), similarly to step S70.

[0218] After step S180 or step S190, the control unit 421 assigns Ts+T to the reference time Ts and assigns 1 to the counter value C (S200). Thereafter, step S120 is executed again.

[0219] As in the fourth embodiment, the entropy calculation method may be changed. In this case, the same effects as those of the first embodiment can be obtained. Note that the fourth embodiment may be combined with at least one of the second and third embodiments.

[0220] 23 is a block diagram showing an example of control by a control unit 421 according to a fifth embodiment. The fifth embodiment differs from the first embodiment in the method of determining the detection timing of a second event detection unit 33B.

[0221] The control unit 421 controls the second event detection unit 33B by rule-based control (domain knowledge). As the rule-based control, for example, counter-based control or a statistical method is used.

[0222] More specifically, the control unit 421 determines the detection timing by applying the detected first event to a rule-based model generated based on the correspondence between the first event and the detection timing.

[0223] As in the fifth embodiment, the method for determining the detection timing of the second event detection unit 33B may be changed. In this case, the same effects as those of the first embodiment can be obtained. Note that the fifth embodiment may be combined with at least one of the second and third embodiments. Furthermore, the detection timing of the second event detection unit 33B may be determined by combining the calculation of entropy or event rate with rule-based control. In this case, the output result of one determination method is used as input, and the output result of the other determination method is used.

[0224] First Modification of Fifth Embodiment In the first modification of the fifth embodiment, the method of determining the detection timing of the second event detector 33B is different from that of the fifth embodiment.

[0225] The control unit 421 controls the second event detection unit 33B by data-driven control. For example, a neural network (NN), a graph neural network (GNN), or a spiking neural network (SNN) is used as the data-driven control.

[0226] More specifically, the control unit 421 determines the detection timing by applying the detected first event to a machine learning model generated based on the correspondence between the first event and the detection timing.

[0227] Alternatively, a contrastive loss function for entropy maximization or entropy minimization may be used. For example, learning is performed so that entropy is minimized when a meaningful event is input and is maximized when an input includes a noise event.

[0228] As in the first modification of the fifth embodiment, the method for determining the detection timing of the second event detection unit 33B may be changed. In this case, the same effects as those of the fifth embodiment can be obtained. Furthermore, the detection timing of the second event detection unit 33B may be determined by combining the calculation of entropy or event rate with data-driven control. In this case, the output result of one determination method is used as input, and the output result of the other determination method is used.

[0229] 24 is a diagram showing an example of control by the control unit 421 according to a second modification of the fifth embodiment. The second modification of the fifth embodiment differs from the fifth embodiment in the method of determining the detection timing of the second event detection unit 33B.

[0230] The control unit 421 controls the second event detection unit 33B by combining a plurality of control methods. The control unit 421 may also control the second event detection unit 33B by combining rule-based control and data-driven control. This can make the control more efficient.

[0231] More specifically, the control unit 421 applies the detected first event to a rule-based model generated based on the correspondence between the first event and the detection timing, and determines the detection timing by applying the machine learning model generated based on the correspondence between the first event and the detection timing to the output of the rule-based model.

[0232] In the example shown in Figure 24, the control unit 421 inputs the first event in the first pixel group GA to the counter-based control, inputs the output of the counter-based control to the NN, GNN, or SNN, and outputs the output of the NN, GNN, or SNN to the second pixel group GB.

[0233] Furthermore, the second event detection unit 33B may be controlled by combining a plurality of data-driven controls.

[0234] In the example shown in Figure 24, the control unit 421 inputs the first event in the first pixel group GA to the SNN, inputs the output of the SNN to the NN or GNN, and outputs the output of the NN or GNN to the second pixel group GB.

[0235] The control unit 421 may apply the detected first event to a machine learning model, and apply a rule-based model to the output of the machine learning model to determine the detection timing.

[0236] As in the second modification of the fourth embodiment, the method of determining the detection timing of the second event detector 33B may be changed. In this case, the same effects as those of the fourth embodiment can be obtained.

[0237] 25 is a block diagram showing an example of the configuration of an image pickup device 20 according to a sixth embodiment. The fourth embodiment differs from the first embodiment in that the pixel array unit 21 is divided into regions and the detection of the second event is controlled.

[0238] The pixel array unit 21 is divided into a plurality of non-overlapping regions Ar. Each region Ar includes a plurality of first pixels 30A (first pixel groups GA) and a plurality of second pixels 30B (second pixel groups GB).

[0239] The second event detection unit 33B detects a second event for each divided area Ar. The second event detection unit 33B detects a second event in the area Ar (the same area Ar) at a timing based on the first event detected in the area Ar.

[0240] The control unit 421 detects a region of interest. The region of interest is a region of interest that is a region of interest where the entropy based on the first event in the first pixel 30A is equal to or exceeds a threshold H ｔｈ That is, the region of interest is the region where the second event is detected.

[0241] The control unit 421 outputs the attention area address to the event reading unit 416 along with the event trigger (detection start signal and detection end signal). The attention area address is the address of the pixel 30 within the attention area. This allows the second event in the second pixel 30B within the attention area to be detected. Note that the second event in the second pixel 30B is not detected in the area Ar that is not the attention area.

[0242] In the first embodiment, all of the second pixels 30B in the pixel array unit 21 are in a state in which they can detect the second event.

[0243] In contrast, in the sixth embodiment, only the second pixel 30B in the region of interest is in a state in which the second event can be detected, thereby enabling more efficient control of the detection of the second event.

[0244] The second event detection unit 33B may detect the second event in the first area at a timing based on the first event detected in a plurality of areas Ar, including an area Ar different from the first area. The first area is any area among the areas Ar.

[0245] As in the sixth embodiment, the pixel array unit 21 may be divided into regions to control the detection of the second event. In this case, the same effects as in the first embodiment can be obtained. Note that the sixth embodiment may be combined with at least one of the second to fifth embodiments.

[0246] Seventh Embodiment The seventh embodiment differs from the first embodiment in that the reset signal supplied to the pixel circuit of the second pixel 30B is turned on during a period when detection of the second pixel 30B is not performed (non-detection period).

[0247] The reset signal that is turned on during the undetectable period short-circuits, for example, the path connecting both ends of the capacitive element 3333 .

[0248] As in the seventh embodiment, the reset signal supplied to the pixel circuit of the second pixel 30B may be turned on during the undetectable period. In this case, the same effect as in the first embodiment can be obtained. Note that the seventh embodiment may be combined with at least one of the second to sixth embodiments.

[0249] Eighth Embodiment The seventh embodiment differs from the first embodiment in that the power supplied to the pixel circuit of the second pixel 30B is turned off by power gating during the undetectable period.

[0250] The power supply that is turned off during the undetectable period is, for example, the power supply that is supplied to the quantizer 334 by power gating.

[0251] As in the eighth embodiment, the power supplied to the pixel circuit of the second pixel 30B may be turned off during the undetectable period. In this case, the same effect as in the first embodiment can be obtained. Note that the eighth embodiment may be combined with at least one of the second to seventh embodiments.

[0252] Ninth Embodiment The ninth embodiment differs from the first embodiment in that the sensitivity differs between the first pixel 30A and the second pixel 30B.

[0253] The sensitivity of the first pixel 30A is different from the sensitivity of the second pixel 30B. The first pixel 30A may be more sensitive or less sensitive than the second pixel 30B.

[0254] Either the first pixel 30A or the second pixel 30B performs event detection. If the entropy of either the first pixel 30A or the second pixel 30B falls below a predetermined value, the other may stop detecting events. If the entropy of either the first pixel 30A or the second pixel 30B exceeds a predetermined value, the other may resume detecting events.

[0255] As in the ninth embodiment, the sensitivity may be different between the first pixel 30A and the second pixel 30B. In this case, the same effect as in the first embodiment can be obtained. Note that the ninth embodiment may be combined with at least one of the second to eighth embodiments.

[0256] 26 is a diagram showing an example of the positions of a light source 2 and an imaging system 10 according to a tenth embodiment. The tenth embodiment differs from the first embodiment in that a light emission control unit 3 is provided.

[0257] The system 1 is, for example, a SLAM (Simultaneous Localization and Mapping) system using VLC.

[0258] The light source 2 is, for example, a light marker provided at the tip of a pointer that is held by a person's hand. The light source 2 is an active light source.

[0259] The imaging system 10 is movable and is mounted on, for example, a virtual reality (VR) device or an augmented reality (AR) device.

[0260] FIG. 27 is a block diagram showing an example of the configuration of the system 1 according to the tenth embodiment.

[0261] The system 100 further includes a light emission control unit 3. The light emission control unit 3 adds a light emission pattern in front of the data signal, which causes the second event detection unit 33B to enter a state in which the second event is detected.

[0262] The light-emission control unit 3 controls the light source 2. More specifically, the light-emission control unit 3 controls the light source 2 to emit light in a first light-emission pattern and a second light-emission pattern that follows the first light-emission pattern. The light-emission control unit 3 causes the light source 2 to emit light in a first light-emission pattern that causes the second event detection unit 33B to detect a second event corresponding to the second light-emission pattern. The first light-emission pattern is a light-emission pattern that starts detection of a second event, and is, for example, a light-emission pattern designed so that the calculated entropy falls below a threshold. The second light-emission pattern is a light-emission pattern that is to be detected as a second event. In other words, the light-emission control unit 3 controls the light source 2, thereby controlling the events that are output (see FIG. 29 ).

[0263] 28 is a timing chart showing an example of the operations of the first pixel 30A, the second pixel 30B, and the control unit 421 according to the tenth embodiment. Fig. 28 shows a timing chart of the emission intensity of the light source 2, an event in the first pixel 30A, an entropy trigger, entropy, and an event in the second pixel 30B.

[0264] First, at time t11, the light source 2 is turned on, and the light emission intensity of the light source 2 increases to a first light emission intensity Int1.

[0265] The first event detection unit 33A detects a change in light emission intensity from zero to the first light emission intensity Int1 (first light emission pattern) as a first event of an ON event. Note that, because the entropy trigger is high, the second event detection unit 33B does not detect a second event.

[0266] The entropy on the ON side is at threshold H ｏｎ When the value falls below t Ｓ１１ ), the entropy trigger goes from high to low, which initiates detection of a second event at the second pixel 30B.

[0267] Next, at time t12, the light source 2 is turned on, and the light emission intensity of the light source 2 further increases to a second light emission intensity Int2. The second light emission intensity Int2 is higher than the first light emission intensity Int1.

[0268] The first event detection unit 33A and the second event detection unit 33B detect a change in light emission intensity from the first light emission intensity Int1 to the second light emission intensity Int2 (second light emission pattern) as a first event and a second event of an ON event, respectively. Therefore, a change in the light emission intensity of the light source 2 from low intensity to high intensity is detected as a second event.

[0269] The entropy on the ON side is at threshold H ｏｎ When the time exceeds the start time of the next period T (time t Ｅ１１ ), the entropy trigger goes from low to high, which ends the detection of the second event at the second pixel 30B.

[0270] Next, at time t13, the light source 2 is turned off, and the light emission intensity of the light source 2 decreases to zero.

[0271] The first event detection unit 33A detects a change in light emission intensity from the second light emission intensity Int2 to zero as a first event of an off event. Note that, because the entropy trigger is high, the second event detection unit 33B does not detect a second event.

[0272] 28, the light source 2 emits light up to the first light emission intensity Int1 (first light emission pattern), and then emits light up to the second light emission intensity Int2 (second light emission pattern). This allows the second event detection unit 33B to detect the second light emission pattern as the second event.

[0273] Fig. 29 is a diagram showing an example of the number of events in the pixel array unit 21 according to the tenth embodiment. Fig. 29 shows on events and off events (event stream) detected in the pixel array unit 21 over time. The upper part of Fig. 29 shows the number of events according to a comparative example in which the light-emission control unit 3 is not provided. The lower part of Fig. 29 shows the number of events according to the tenth embodiment.

[0274] If the imaging system 10 is movable as shown in FIG. 26, there is a possibility that a large number of unnecessary events will be detected, as shown in the upper part of FIG.

[0275] By controlling the light source 2 with the light emission control unit 3, it is possible to prevent unnecessary events from being output, as shown in the lower part of Fig. 29. The events to be output from the imaging system 10 can be controlled by the light emission pattern of the light source 2. For example, it is possible to output only events caused by the light source 2, and also to prevent events caused by egomotion from being output.

[0276] As in the tenth embodiment, a light emission control unit 3 may be provided. In this case, the same effects as in the first embodiment can be obtained. Note that the tenth embodiment may be combined with at least one of the second to ninth embodiments.

[0277] (Application Example to Electronic Device) FIG. 30 is a block diagram showing a configuration example of a camera 2000 as an electronic device to which the present technology is applied.

[0278] The camera 2000 includes an optical unit 2001 including a group of lenses, an imaging apparatus (imaging device) 2002 to which the imaging system 10 described above or the like (hereinafter referred to as the imaging system 10, etc.) is applied, and a DSP (Digital Signal Processor) circuit 2003, which is a camera signal processing circuit. The camera 2000 also includes a frame memory 2004, a display unit 2005, a recording unit 2006, an operation unit 2007, and a power supply unit 2008. The DSP circuit 2003, the frame memory 2004, the display unit 2005, the recording unit 2006, the operation unit 2007, and the power supply unit 2008 are connected to one another via a bus line 2009.

[0279] The optical unit 2001 takes in incident light (image light) from a subject and forms an image on the imaging surface of the imaging device 2002. The imaging device 2002 converts the amount of incident light formed on the imaging surface by the optical unit 2001 into an electrical signal on a pixel-by-pixel basis and outputs the signal as a pixel signal.

[0280] The display unit 2005 is formed of a panel display device such as a liquid crystal panel or an organic EL panel, and displays moving images or still images captured by the imaging device 2002. The recording unit 2006 records the moving images or still images captured by the imaging device 2002 on a recording medium such as a hard disk or semiconductor memory.

[0281] An operation unit 2007, under the operation of a user, issues operation commands for various functions of the camera 2000. A power supply unit 2008 appropriately supplies various types of power to the DSP circuit 2003, frame memory 2004, display unit 2005, recording unit 2006, and operation unit 2007 as operating power sources.

[0282] As described above, by using the imaging system 10 or the like as the imaging device 2002, it is possible to expect to obtain good images.

[0283] <Application to a Mobile Body> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0284] FIG. 31 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0285] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 31 , the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0286] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0287] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0288] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0289] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0290] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0291] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0292] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0293] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12030 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0294] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying information to vehicle occupants or the outside of the vehicle. In the example of Fig. 31, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0295] FIG. 32 is a diagram showing an example of the installation position of the imaging unit 12031.

[0296] In FIG. 32, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0297] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0298] 32 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0299] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0300] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0301] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0302] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0303] The above describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to, for example, the image capture units 12031, 12101, 12102, 12103, 12104, and 12105 of the above-described configuration. Specifically, for example, the image capture system 10 shown in FIG. 1 can be applied to these image capture units. By applying the technology according to the present disclosure to these image capture units, the amount of data can be more appropriately reduced, thereby enabling highly accurate control using captured images in a mobile object control system.

[0304] The present technology may be configured as follows: (1) A photodetector including: a pixel array unit having a first pixel group including a plurality of first pixels and a second pixel group including a plurality of second pixels; a first event detection unit that detects a change in luminance of incident light at the first pixel as a first event; and a second event detection unit that detects a change in luminance of incident light at the second pixel as a second event at a timing based on the detected first event. (2) The photodetector according to (1), wherein the first event detection unit continues to detect the first event, and further includes a control unit that controls the second event detection unit to detect the second event at a timing based on the detected first event. (3) The photodetector according to (2), wherein the control unit determines a detection timing at which the second event detection unit detects the second event based on the detected first event, and controls the second event detection unit based on the determined detection timing. (4) The light detecting element according to (3), wherein the control unit calculates an entropy of the detected first event and determines the detection timing based on the calculated entropy. (5) The light detecting element according to (3), wherein the control unit calculates an event rate of the detected first event over a predetermined period and determines the detection timing based on the calculated event rate. (6) The light detecting element according to (3), wherein the control unit determines the detection timing by applying the detected first event to a rule-based model generated based on a correspondence between the first event and the detection timing. (7) The light detecting element according to (3), wherein the control unit determines the detection timing by applying the detected first event to a machine learning model generated based on a correspondence between the first event and the detection timing.(8) The photodetector element according to (3), wherein the control unit determines the detection timing by applying the detected first event to a rule-based model generated based on a correspondence between the first event and the detection timing and applying a machine learning model generated based on the correspondence between the first event and the detection timing to an output of the rule-based model, or by applying the detected first event to the machine learning model and applying the rule-based model to an output of the machine learning model. (9) The photodetector element according to any one of (1) to (8), wherein the pixel array unit is divided into a plurality of regions, each including the first pixel group and the second pixel group, and the second event detection unit detects the second event in each of the divided regions. (10) The photodetector element according to (9), wherein the second event detection unit detects the second event in each of the regions at a timing based on the first event detected in the region. (11) The light detection element according to (9), wherein the second event detection unit detects the second event in the first region at a timing based on the first event detected in a plurality of regions including the region different from the first region. (12) The light detection element according to any one of claims (1) to (11), wherein the second event detection unit detects the second event of an on-event at a timing based on the first event of an on-event detected, and detects the second event of an off-event at a timing based on the first event of an off-event detected. (13) The photodetector element according to (12), wherein the second pixel has a photoelectric conversion element that photoelectrically converts incident light to generate a photocurrent, and the second event detection unit has: a current-voltage conversion unit that converts the photocurrent into a voltage; a subtractor that calculates and outputs an amount of change in the voltage; and a quantizer that detects whether the amount of change exceeds a threshold, and the quantizer has: a switching unit that switches the threshold depending on whether the detected first event is an on event or an off event; and a comparator that compares the amount of change with the threshold switched by the switching unit.(14) The photodetector element according to any one of (1) to (13), wherein the second pixel has a photoelectric conversion element that photoelectrically converts incident light to generate a photocurrent, the second event detection unit has: a current-voltage conversion unit that converts the photocurrent into a voltage; a subtractor that calculates and outputs a change in the voltage; and a quantizer that detects whether the change exceeds a threshold, the subtractor has: a first capacitor; an inverter whose input is electrically connected to the first capacitor; a first switch connected between a node between the first capacitor and the input of the inverter and an output of the inverter, the first switch initializing an output of the subtractor; a second capacitor connected between the node and the output of the inverter; and a second switch connected between the node and the output of the inverter, the second switch being turned on or off at a timing based on the detected first event. (15) The photodetector element according to any one of (1) to (13), wherein the second pixel includes a photoelectric conversion element that photoelectrically converts incident light to generate a photocurrent, the second event detection unit includes a current-voltage conversion unit that converts the photocurrent into a voltage, a subtractor that calculates and outputs a change in the voltage, and a quantizer that detects whether the change exceeds a threshold, the subtractor including: a first capacitor, an inverter whose input is electrically connected to the first capacitor, a first switch connected between a node between the first capacitor and the input of the inverter and an output of the inverter, the first switch being configured to initialize an output of the subtractor, and a second capacitor connected between the node and the output of the inverter, and further including a switch control unit that controls the first switch based on the detected first event. (16) The photodetector element according to any one of (1) to (15), wherein a sensitivity of the first pixel is different from a sensitivity of the second pixel. (17) The light detection element according to any one of (1) to (16), wherein the second event detection unit detects the second event at a timing based on the first event detected in the first pixel group.(18) A system comprising: a pixel array unit having a first pixel group including a plurality of first pixels and a second pixel group including a plurality of second pixels; a first event detection unit that detects a change in luminance of incident light at the first pixel as a first event; a second event detection unit that detects a change in luminance of incident light at the second pixel as a second event at a timing based on the detected first event; and an emission control unit that controls a light source to emit light in a first emission pattern and a second emission pattern that follows the first emission pattern, wherein the emission control unit causes the second event detection unit to detect the second event corresponding to the second emission pattern, and causes the light source to emit light in the first emission pattern.

[0305] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0306] 1 System, 2 Light source, 3 Light emission control unit, 10 Imaging system, 11 Imaging lens, 12 Recording unit, 13 Control unit, 20 Imaging device, 21 Pixel array unit, 22 Drive unit, 23 Arbiter unit, 24 Column processing unit, 25 Signal processing unit, 27 Readout area selection unit, 28 Signal generation unit, 30 Pixel, 30A First pixel, 30B Second pixel, 31 Light receiving unit, 32 Pixel signal generation unit, 33 Address event detection unit, 331 Current-voltage conversion unit, 332 Buffer, 333 Subtractor, 334 Quantizer, 414 Row AER circuit, 416 Event readout unit, 420 Timing control device, 421 Control unit, 422 Memory unit

Claims

1. A light detection element comprising: a pixel array section having a first pixel group including a plurality of first pixels and a second pixel group including a plurality of second pixels; a first event detection section that detects a change in luminance of incident light at the first pixels as a first event; and a second event detection section that detects a change in luminance of incident light at the second pixels as a second event at a timing based on the detected first event.

2. The light detection element according to claim 1, further comprising a control unit that controls the second event detection unit so as to continue to detect the first event and to detect the second event at a timing based on the detected first event.

3. The optical detection element described in claim 2, wherein the control unit determines a detection timing for causing the second event detection unit to detect the second event based on the detected first event, and controls the second event detection unit based on the determined detection timing.

4. The light detection element according to claim 3, wherein the control unit calculates an entropy of the detected first event, and determines the detection timing based on the calculated entropy.

5. The light detection element according to claim 3, wherein the control unit calculates an event rate of the first event detected in a predetermined period, and determines the detection timing based on the calculated event rate.

6. The light detection element of claim 3, wherein the control unit determines the detection timing by applying the detected first event to a rule-based model generated based on a correspondence between the first event and the detection timing.

7. The optical detection element described in claim 3, wherein the control unit determines the detection timing by applying the detected first event to a machine learning model generated based on a correspondence between the first event and the detection timing.

8. The light detection element described in claim 3, wherein the control unit determines the detection timing by applying the detected first event to a rule-based model generated based on the correspondence between the first event and the detection timing and applying a machine learning model generated based on the correspondence between the first event and the detection timing to an output of the rule-based model, or by applying the detected first event to the machine learning model and applying the rule-based model to an output of the machine learning model.

9. The light detection element according to claim 1, wherein the pixel array section is divided into a plurality of regions, each region including the first pixel group and the second pixel group, and the second event detection section detects the second event for each of the divided regions.

10. The light detection element according to claim 9, wherein the second event detection unit detects the second event within the region at a timing based on the first event detected within the region.

11. The light detection element described in claim 9, wherein the second event detection unit detects the second event in the first region at a timing based on the first event detected in a plurality of regions including a region different from the first region.

12. The light detection element of claim 1, wherein the second event detection unit detects the second event of an on-event at a timing based on the first event of a detected on-event, and detects the second event of an off-event at a timing based on the first event of a detected off-event.

13. The photodetector element of claim 12, wherein the second pixel has a photoelectric conversion element which performs photoelectric conversion on incident light to generate a photocurrent, and the second event detection unit has: a current-to-voltage conversion unit which converts the photocurrent into a voltage, a subtractor which finds and outputs an amount of change in the voltage, and a quantizer which detects whether the amount of change exceeds a threshold, and the quantizer has: a switching unit which switches the threshold depending on whether the detected first event is an on-event or an off-event, and a comparator which compares the amount of change with the threshold switched by the switching unit.

14. The photodetector element of claim 1, wherein the second pixel has a photoelectric conversion element that photoelectrically converts incident light to generate a photocurrent, the second event detection unit has: a current-voltage conversion unit that converts the photocurrent into a voltage, a subtractor that finds and outputs an amount of change in the voltage, and a quantizer that detects whether the amount of change has exceeded a threshold, the subtractor having: a first capacitor; an inverter whose input unit is electrically connected to the first capacitor; a first switch connected between a node between the first capacitor and the input unit of the inverter and an output unit of the inverter and that initializes an output of the subtractor; a second capacitor connected between the node and the output unit of the inverter; and a second switch connected between the node and the output unit of the inverter, the second switch being turned on or off at a timing based on the detected first event.

15. The photodetector element of claim 1, further comprising a switch control section for controlling the first switch based on the detected first event.

15. The photodetector element of claim 1, further comprising: a photoelectric conversion element for generating a photocurrent by photoelectrically converting incident light; and a second event detection section for detecting a change in the voltage by a current-voltage conversion section for converting the photocurrent into a voltage; a subtractor for determining and outputting an amount of change in the voltage; and a quantizer for detecting whether the amount of change has exceeded a threshold value; and the subtractor has: a first capacitor; an inverter having an input section electrically connected to the first capacitor; a first switch connected between a node between the first capacitor and the input section of the inverter and an output section of the inverter, the first switch being connected between the node and the output section of the inverter and initializing an output of the subtractor; and a second capacitor connected between the node and the output section of the inverter.

16. The photodetector of claim 1, wherein the sensitivity of the first pixel is different from the sensitivity of the second pixel.

17. The light detection element according to claim 1, wherein the second event detection section detects the second event at a timing based on the first event detected in the first pixel group.

18. A system comprising: a pixel array unit having a first pixel group including a plurality of first pixels and a second pixel group including a plurality of second pixels; a first event detection unit that detects a change in luminance of incident light at the first pixel as a first event; a second event detection unit that detects a change in luminance of incident light at the second pixel as a second event at a timing based on the detected first event; and an emission control unit that controls a light source to emit light in a first emission pattern and a second emission pattern subsequent to the first emission pattern, wherein the emission control unit causes the second event detection unit to detect the second event corresponding to the second emission pattern, and causes the light source to emit light in the first emission pattern.

Images