Light detection device and electronic apparatus

Abstract

[Problem] To perform automatic focusing with high accuracy and at high speed with a simple configuration. [Solution] This light detection device comprises: a plurality of pixels that output an event signal when a luminance change of incident light incident through an optical system exceeds a predetermined threshold; and a defocus amount detection circuit that detects a defocus amount of the incident light on the basis of the event signal. At least some pixels among the plurality of pixels have at least one among a first photoelectric conversion region and a second photoelectric conversion region that detects a luminance change in a divided pixel region and outputs the event signal. The defocus amount detection circuit detects the defocus amount on the basis of two of the event signals output from the first photoelectric conversion region and the second photoelectric conversion region which are paired.

Description

Photodetector and electronic equipment

[0001] The present disclosure relates to photodetection devices and electronic devices.

[0002] A camera that performs high-precision autofocus by tracking a fast-moving subject has been proposed (see Patent Document 1). In Patent Document 1, autofocus is performed by providing a moving object detection unit that detects a fast-moving moving object OB and a prediction calculation unit that predicts the future position of the fast-moving moving object OB.

[0003] Furthermore, a technology for performing autofocus in a photodetector that outputs an event signal has been proposed (see Patent Document 2). In Patent Document 2, the pixel that outputs the event signal is divided into two groups, a first pixel and a second pixel, and the defocus amount is calculated from the relative direction and amount of deviation of the digital signals output from each of these pixels to perform autofocus.

[0004] JP 2010-8507 A International Publication No. 2023 / 132129

[0005] However, with the technology of Patent Document 1, due to delays in processing by the moving object detection unit and the prediction calculation unit, it is not possible to perform autofocus quickly, and there is a risk that a fast-moving moving object OB cannot be tracked accurately. In addition, the configuration of the camera control system becomes complicated, which may increase power consumption.

[0006] Furthermore, in Patent Document 2, a circuit that compares a voltage corresponding to the accumulated charge of a photodiode with a reference voltage to generate a binary digital signal, separate from a circuit that detects an event signal, and detects the defocus amount based on this digital signal. In Patent Document 2, a circuit that generates a digital signal must be provided for each pixel, and processing must be performed to adjust the voltage level of the reference voltage, which may result in a long time being required to detect the defocus amount.

[0007] Therefore, the present disclosure provides a photodetector and electronic equipment that are capable of performing autofocusing with high accuracy and high speed with a simple configuration.

[0008] In order to solve the above problems, according to the present disclosure, there is provided a photodetection device comprising: a plurality of pixels that output an event signal when a change in luminance of incident light incident via an optical system exceeds a predetermined threshold; and a defocus amount detection circuit that detects a defocus amount of the incident light based on the event signal, wherein at least some of the plurality of pixels have at least one of a first photoelectric conversion region and a second photoelectric conversion region that detects a change in luminance within a divided pixel region and outputs the event signal, and the defocus amount detection circuit detects the defocus amount based on the two event signals output from the paired first photoelectric conversion region and second photoelectric conversion region.

[0009] The plurality of pixels may include a plurality of pairs of the first photoelectric conversion region and the second photoelectric conversion region, and the defocus amount detection circuit may detect the defocus amount based on the amount and direction of deviation between the pixel position of the first photoelectric conversion region that outputs the event signal and the pixel position of the second photoelectric conversion region that outputs the event signal, among the plurality of pairs of the first photoelectric conversion region and the second photoelectric conversion region.

[0010] The defocus amount detection circuit may detect, based on the pixel position of the first photoelectric conversion region that output the event signal and the pixel position of the second photoelectric conversion region that output the event signal, whether the focal position of the optical system is located closer to the subject than the light-receiving surfaces of the plurality of pixels, whether the focal position is located on the light-receiving surface, or whether the focal position is located farther from the subject than the light-receiving surface.

[0011] At least some of the pixels may have a pair of the first photoelectric conversion region and the second photoelectric conversion region.

[0012] Some of the pixels may have a pair of the first photoelectric conversion region and the second photoelectric conversion region, and other than the some of the pixels, the pixels may output the event signal without having the first photoelectric conversion region and the second photoelectric conversion region.

[0013] The pair of the first photoelectric conversion region and the second photoelectric conversion region may be arranged in different pixels, and a pixel having the first photoelectric conversion region or the second photoelectric conversion region may have a light-shielding member that covers an area other than the first photoelectric conversion region or the second photoelectric conversion region.

[0014] At least some of the pixels may have a correction unit that corrects and integrates the two event signals output from the paired first and second photoelectric conversion regions to output a new event signal.

[0015] The correction unit may control the threshold value so that the total number of two event signals output from a pair of the first photoelectric conversion region and the second photoelectric conversion region is equal to the total number of event signals output from pixels that do not have the first photoelectric conversion region and the second photoelectric conversion region and are located around a pixel that has a pair of the first photoelectric conversion region and the second photoelectric conversion region.

[0016] The correction unit may correct the two event signals output from the first photoelectric conversion region and the second photoelectric conversion region based on the occurrence rate of a plurality of the event signals output from a plurality of pixels that do not have the first photoelectric conversion region and the second photoelectric conversion region and are located around a pixel that has a pair of the first photoelectric conversion region and the second photoelectric conversion region.

[0017] The plurality of pixels may be eight pixels surrounding a pixel having the first photoelectric conversion region and the second photoelectric conversion region.

[0018] The defocus amount detection circuit may include an information processing unit that uses two event signals output from a pair of the first photoelectric conversion region and the second photoelectric conversion region as input parameters to construct an information processing model by machine learning that outputs the defocus situation notification event corresponding to the input parameters, and inputs new input parameters into the constructed, trained information processing model, and outputs the defocus situation notification event corresponding to the new input parameters from the information processing model.

[0019] The information processing model may have an input layer to which the two event signals are input and an output layer that outputs the defocus state notification event, and the output layer may output a plurality of event signals that represent the defocus amount.

[0020] The defocus state notification event may include a front focus detection event signal, a correct focus detection event signal, or a back focus detection event signal.

[0021] The information processing model may be a Spiking Neural Network (SNN).

[0022] The image sensor may include a first substrate disposed on the light incident surface side, and a second substrate disposed on the opposite side of the light incident surface and stacked on the first substrate, wherein at least some of the plurality of pixels are disposed on the first substrate, and at least some of the defocus amount detection circuit are disposed on the second substrate.

[0023] Each of the plurality of pixels may have an event detection circuit that generates the event signal, and at least a portion of the event detection circuit may be disposed on the second substrate.

[0024] The plurality of pixels may output an on-event signal when a change in luminance in a direction in which the luminance of the incident light increases exceeds the threshold, and output an off-event signal when a change in luminance in a direction in which the luminance of the incident light decreases exceeds the threshold, and the defocus amount detection circuit may detect the defocus amount based on the on-event signal and the off-event signal.

[0025] The plurality of pixels may include: a plurality of light-receiving pixels that output a signal indicating whether or not an incident photon has been detected; a counter for each of the plurality of light-receiving pixels that measures the number of times that a photon has been detected; and a comparator that outputs the event signal when the number of times measured by the counter exceeds the threshold value.

[0026] The image pickup device may further include a vibration suppression unit that suppresses vibration of a pixel array unit having the plurality of pixels, and the defocus amount detection circuit may detect the defocus amount based on the event signal output from the pixel array unit after the vibration suppression unit has suppressed vibration of the pixel array unit.

[0027] According to the present disclosure, there is provided an electronic device including: the above-described light detection device; the optical system; and an optical system driving unit that moves the optical system along the optical axis based on the defocus amount to perform focus adjustment.

[0028] 11A is a block diagram of an electronic device according to a first embodiment of the present disclosure; FIG. 11B is a schematic perspective view showing an example of a photodetector according to the present disclosure having a two-layer stacked structure; FIG. 11C is a schematic perspective view showing an example of a photodetector according to the present disclosure having a three-layer structure; FIG. 11D is a plan view showing an example of a light receiving chip; FIG. 11E is a plan view showing an example of a detection chip; FIG. 11F is a circuit diagram of a photoelectric conversion element and an event detection circuit according to the first embodiment; FIG. 11G is a diagram showing a pixel configuration of a pixel array section of the photodetector according to the first embodiment; FIG. 11H is a block diagram of a portion related to AF operation of the photodetector according to the first embodiment; FIG. 11H is a schematic timing chart of the photodetector according to the first embodiment; FIG. 11H is a schematic timing chart of a photodetector according to a comparative example; FIG. 11F is a diagram showing an example of front focus where the focal position of the lens is located in front of the imaging plane; FIG. 11H is a diagram showing an example of an imaging plane located at the focal position of the lens; FIG. 11B is a diagram showing an example of rear focus where the focal position of the lens is located behind the imaging plane; FIG. 11C is a diagram showing the detection position of an ON event signal when a moving object is in front focus; FIG. 11B is a diagram showing the detection position of an ON event signal following FIG. 12B. A diagram showing the detection position of an off-event signal when in front focus while a moving object is moving. 12C. A diagram showing the detection position of an off-event signal following FIG. 12A. A diagram showing the detection position of an off-event signal following FIG. 12B. A diagram showing the detection position of an on-event signal when in back focus while a moving object is moving. 13A. A diagram showing the detection position of an on-event signal following FIG. 13B. A diagram showing the detection position of an off-event signal when in back focus while a moving object is moving. 14A. A diagram showing the detection position of an off-event signal following FIG. 14B. A diagram explaining the processing of the defocus amount calculation unit 4. A diagram showing the change over time in pixel positions of event signals detected by multiple pairs of L-phase EVS pixels and R-phase EVS pixels. A diagram showing the distribution of pixel positions of event signals output from multiple pairs of L-phase EVS pixels and R-phase EVS pixels. A diagram explaining calculation of the amount of deviation. A diagram showing an example in which one EVS pixel is divided into two and an L-phase EVS pixel and an R-phase EVS pixel are provided within one EVS pixel. 1 is a diagram showing an example in which half of one EVS pixel is covered with a light-shielding member, and L-phase EVS pixels or R-phase EVS pixels are arranged in the remaining half. 2 is a diagram showing an example in which all EVS pixels in a pixel array section are divided into two to provide L-phase EVS pixels and R-phase EVS pixels. 3 is a diagram showing a first example of the correspondence between pixels and pixel circuits.27. A circuit diagram showing an example in which a portion of two pixel circuits (readout circuits) is shared. A diagram showing a second example of a pixel and a pixel circuit (readout circuit). A diagram showing a third example of a pixel and a pixel circuit (readout circuit). A diagram showing a fourth example of a pixel and a pixel circuit (readout circuit). A diagram showing a fifth example of a pixel and a pixel circuit. A circuit diagram showing a first example of the pixel circuit (readout circuit) of FIGS. 22 to 24. A circuit diagram showing a second example of the pixel circuit (readout circuit) of FIGS. 22 to 24. A circuit diagram showing a third example of the pixel circuit (readout circuit) of FIGS. 22 to 24. A diagram explaining the processing operation of the LR phase correction unit shown in FIG. 7. A block diagram showing the internal configuration of the LR phase correction unit of FIG. 27. A diagram showing a modified example of the processing operation of the LR phase correction unit shown in FIG. 7. A diagram explaining a photodetector according to a second embodiment. A diagram showing the basic structure of an SNN. A diagram showing a pixel array unit of a photodetector according to a third embodiment. A circuit diagram of each pixel according to the third embodiment. A timing chart of the photodetector according to the third embodiment. 10 is a block diagram of a portion related to an AF operation of a photodetector according to a fourth embodiment. 20 is a timing chart of the photodetector according to the fourth embodiment. 21 is a block diagram showing an example of a schematic configuration of a vehicle control system. 22 is an explanatory diagram showing an example of the installation positions of an outside-of-vehicle information detection unit and an imaging unit.

[0029] Hereinafter, an embodiment of a photodetector will be described with reference to the drawings. The following description will focus on the main components of the photodetector, but the photodetector may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0030] 1 is a block diagram of an electronic device 30 according to a first embodiment of the present disclosure. The electronic device 30 has a function of generating an image according to the luminance of incident light. The electronic device 30 in FIG. 1 includes a light detection device 1, an imaging lens 31, an image processing unit 32, a recording unit 33, and a control unit 34. The electronic device 30 can be applied to, for example, a surveillance camera or a camera mounted on an industrial robot, or a camera for general use, but the specific use and configuration of the electronic device 30 are arbitrary.

[0031] The imaging lens 31 collects incident light and guides it to the photodetector 1. The photodetector 1 captures an image of the incident light. The photodetector 1 causes light within a predetermined wavelength range, such as visible light or infrared light, to be incident on multiple pixels, and outputs an event signal from a pixel whose luminance change exceeds a predetermined threshold. The event signal output from the photodetector 1 is sent to an image processor 32 and a recorder 33. In this specification, each of the multiple pixels included in the photodetector 1 may be referred to as an EVS (Event-based Vision Sensor) pixel. The photodetector 1 may include a pixel that outputs a pixel signal containing gradation information corresponding to the amount of incident light. In this specification, a pixel that outputs a pixel signal containing gradation information may be referred to as a gradation pixel.

[0032] The image processing unit 32 performs predetermined image processing on the captured image, such as color or brightness adjustment, image compression, image recognition, tracking, or analysis. The image processed by the image processing unit 32 is output to the recording unit 33, for example.

[0033] The recording unit 33 records the image output from the light detection device 1 or the image processing unit 32. The recording unit 33 may be disposed in a server connected via a network. In the electronic device 30 according to this embodiment, at least one of the image processing unit 32 and the recording unit 33 in FIG. 1 can be omitted.

[0034] The control unit 34 controls the operation of the light detection device 1. Furthermore, although not explicitly shown in FIG.

[0035] (Two-Layer Stacking) The photodetector 1 according to the present disclosure can be realized as a stacked chip. FIG. 2A is a schematic perspective view illustrating an example of a two-layer stacked photodetector 1 according to the present disclosure. The photodetector 1 in FIG. 2A includes a first substrate SB1 disposed on the light incident surface side and a second substrate SB2 stacked on the first substrate SB1. For example, a photoelectric conversion element for each pixel is disposed on the first substrate SB1. This specification mainly describes an example in which the photoelectric conversion element is a photodiode. Peripheral circuits (e.g., transfer transistors, etc.) for the photodiode may also be disposed on the first substrate SB1. Multiple transistors for generating event signals are disposed on the second substrate SB2. The first substrate SB1 and the second substrate SB2 are bonded and transmit signals, for example, via a copper-copper connection (CCC). Alternatively, the first substrate SB1 and the second substrate SB2 may be bonded using vias or bumps other than CCC. In this specification, the first substrate SB1 may be referred to as the upper chip SB1, and the second substrate SB2 may be referred to as the lower chip SB2.

[0036] (Three-Layer Stacking) The photodetector 1 according to the present disclosure can be configured by stacking three or more substrates. FIG. 2B is a schematic perspective view showing an example of a three-layer structure of the photodetector 1 according to the present disclosure. FIG. 2B shows an example of a stacked structure of the photodetector 1 including a first substrate SB1, a second substrate SB2, and a third substrate SB3. The first substrate SB1 is provided with a photoelectric conversion element for each pixel. The second substrate SB2 is provided with, for example, a current-voltage conversion unit and an event detection unit. The third substrate SB2 is provided with a signal processing circuit.

[0037] The circuit elements and the like arranged on the first substrate SB1, the second substrate SB2, and the third substrate SB3 are arbitrary and may be combined in any desired manner. Below, an example in which the photodetector 1 according to the present disclosure has the two-layer stacked structure shown in FIG. 2A will be mainly described. Hereinafter, the first substrate SB1 may be referred to as the light-receiving chip 21, and the second substrate SB2 may be referred to as the detection chip 22.

[0038] 3 is a plan view showing an example of the light-receiving chip 21. The light-receiving chip 21 is provided with a light-receiving section 24 and a plurality of via arrangement sections 23.

[0039] Vias for transmitting and receiving various signals to and from the detection chip 22 are arranged in the via arrangement section 23. The light receiving section 24 also includes a pixel array section 25. The pixel array section 25 has a plurality of photoelectric conversion elements 26 arranged in a first direction X and a second direction Y. In this specification, the left-right (horizontal) direction in FIG. 3 is referred to as the first direction X, and the up-down (vertical) direction in FIG. 3 is referred to as the second direction Y. The photoelectric conversion elements 26 photoelectrically convert incident light and accumulate electric charges (hereinafter referred to as photocharges) according to the amount of incident light.

[0040] 4 is a plan view showing an example of the detection chip 22. This detection chip 22 has a via arrangement section 23, an event detection section 36, a row driving circuit 37, a column driving circuit 38, and a signal processing circuit 39. One or more vias are arranged in the via arrangement section 23 to transmit and receive various signals to and from the photosensor chip 21.

[0041] The event detection unit 36 ​​has an event detection circuit 41 for each pixel. The event detection circuit 41 generates an event detection signal based on the amount of change in the amount of light incident on the multiple photoelectric conversion elements 26, and outputs the signal to the signal processing circuit 39. A part of the event detection circuit 41 is arranged on the photosensor chip 21 side, for example.

[0042] The event detection circuits 41 quantize voltage signals corresponding to photocharges from the corresponding photoelectric conversion elements 26 and output the quantized voltage signals as detection signals. Each event detection circuit 41 is assigned a pixel address and connected to the photoelectric conversion elements 26 having the same address. In this specification, the photoelectric conversion elements 26 are sometimes referred to as pixels, and the event detection circuits 41 are sometimes referred to as pixel circuits or readout circuits. In some cases, the photoelectric conversion elements 26 and the event detection circuits 41 are sometimes collectively referred to as pixels. The multiple photoelectric conversion elements 26 and the multiple event detection circuits 41 are each arranged in a two-dimensional plane to form a pixel array unit 25, which will be described later. As will be described later, at least some of the multiple pixels in the pixel array unit 25 according to this embodiment have at least one of a first photoelectric conversion region and a second photoelectric conversion region that detects luminance changes within the divided pixel region and outputs an event signal.

[0043] The row drive circuit 37 selects a row address and causes the event detection circuit 41 to output a detection signal corresponding to the selected row address.

[0044] The column drive circuit 38 selects a column address and causes the event detection circuit 41 to output a detection signal corresponding to the selected column address.

[0045] The signal processing circuit 39 generates image data by performing predetermined signal processing on the detection signal from the event detection circuit 41. The signal processing circuit 39 may perform any signal processing, such as image recognition processing or inference processing, on the generated image data.

[0046] 5 is a circuit diagram of the photoelectric conversion element 26 and event detection circuit 41 according to the first embodiment. The event detection circuit 41 includes a current-voltage conversion unit 42, a buffer 43, a differentiation circuit 44, and a quantizer 45. The current-voltage conversion unit 42 and the photoelectric conversion element 26 constitute a logarithmic response unit.

[0047] The current-voltage converter 42 generates a voltage signal Vlog by logarithmically converting the charges photoelectrically converted by the photoelectric conversion element 26. The reason for the logarithmic conversion is to widen the dynamic range of the pixel (photoelectric conversion element 26) that acquires luminance information.

[0048] The photoelectric conversion element 26 accumulates electric charges (photocharges) based on incident light that is incident on the corresponding pixel (photoelectric conversion element 26). For example, a photodiode is used as the photoelectric conversion element 26. The photoelectric conversion element 26 has an anode and a cathode. Either the anode or the cathode (for example, the cathode) is connected to an input node n1 of the current-voltage conversion unit 42, and the other (for example, the anode) is connected to a predetermined reference voltage node such as a ground voltage.

[0049] The current-voltage converter 42 converts the charge accumulated in the photoelectric conversion element 26 into a voltage. The current-voltage converter 42 includes a transistor Q1, a transistor Q2, a transistor Q3, a transistor Q4, and a transistor Q5. The transistors Q1 to Q4 are, for example, NMOS transistors. The transistor Q5 is, for example, a PMOS transistor.

[0050] The transistors Q1 and Q2 are cascode-connected between the power supply voltage node and a predetermined photoelectric conversion element 26. The source of the transistor Q1 is connected to the cathode of the photoelectric conversion element 26 and the gate of the transistor Q3, and the gate of the transistor Q1 is connected to the drain of the transistor Q3 and the source of the transistor Q4. The drain of the transistor Q2 is connected to the power supply voltage node, and the gate is connected to the output node n2 of the current-voltage conversion unit 42, the drain of the transistor Q4, the drain of the transistor Q5, and the input node of the buffer 43.

[0051] Transistors Q3 and Q4 are cascode-connected between node n2 and a reference voltage (ground) node. The source of transistor Q3 is connected to the reference voltage (ground) node, and the gate is connected to the source of transistor Q1 and the cathode of photoelectric conversion element 26. Transistor Q4 is disposed between transistors Q3 and Q5, and the gate of transistor Q4 is connected to the drain of transistor Q1 and the source of transistor Q2, and the drain of transistor Q4 is connected to output node n2.

[0052] The source of the transistor Q5 is connected to the power supply voltage node, and the bias voltage Vblog is applied to the gate of the transistor Q5, which adjusts the voltage level of the output node n2 according to the voltage level of the bias voltage Vblog.

[0053] The voltage signal Vlog logarithmically converted by the current-voltage converter 42 is input to a buffer 43. The buffer 43 includes a transistor Q7 and a transistor Q6 cascode-connected between a power supply voltage node and a reference voltage (e.g., ground) node. The transistors Q6 and Q7 are, for example, PMOS transistors.

[0054] The transistor Q6 in the buffer 43 forms a source follower circuit. A pixel voltage Vsf corresponding to the voltage signal Vlog output from the current-voltage conversion unit 42 is output from the buffer 43. The voltage signal Vlog is input to the gate of the transistor Q6 from the output node n2 of the current-voltage conversion unit 42. The source of the transistor Q6 is connected to the power supply voltage node, and the drain is connected to the differentiation circuit 44 via the output node n3 of the buffer 43.

[0055] The source of transistor Q7 is connected to the power supply voltage node, and the drain is connected to the source of transistor Q6. A bias voltage Vbsf is applied to the gate of transistor Q7. Transistor Q7 adjusts the voltage level of the source of transistor Q6 in accordance with the voltage level of bias voltage Vbsf.

[0056] The pixel voltage Vsf output from the buffer 43 is input to the differentiation circuit 44. The buffer 43 can improve the driving force of the pixel voltage Vsf. Furthermore, by providing the buffer 43, it is possible to ensure isolation so that noise generated when the differentiation circuit 44 in the subsequent stage performs a switching operation is not transmitted to the current-voltage conversion unit 42.

[0057] The differentiating circuit 44 generates a differentiated signal Vout according to changes in the voltage signal Vlog converted by the current-voltage converter 42. The differentiating circuit 44 includes a capacitor C1 and transistors Q8 to Q10. The transistor Q10 is, for example, an NMOS transistor, and the transistors Q8 and Q9 are, for example, PMOS transistors.

[0058] Capacitor C1 is disposed between a connection node n4 of the source of transistor Q8 and the gate of transistor Q9 and an output node n3 of buffer 43. Capacitor C1 supplies a current corresponding to the amount of change in pixel voltage Vsf, which is obtained by time-differentiating pixel voltage Vsf output from buffer 43, to the source of transistor Q8 and the gate of transistor Q9.

[0059] Transistor Q8 switches whether to short-circuit the gate and drain of transistor Q9 in accordance with auto-zero signal XAZ. Auto-zero signal XAZ is a signal that instructs initialization, and for example, changes from high level to low level every time an event detection signal (described later) is output from event detection circuit 41. When auto-zero signal XAZ changes to low level, transistor Q8 turns on, resetting differentiated signal Vout to its initial value and initializing the charge of capacitor C1.

[0060] The source of the transistor Q10 is connected to a reference voltage (for example, ground) node, and a bias voltage Vbdiff is applied to its gate. The transistor Q10 adjusts the voltage level of the output node n5 of the differentiation circuit 44 in accordance with the voltage level of the bias voltage Vbdiff.

[0061] The transistors Q9 and Q10 function as an inverting circuit with the connection node n4 on the gate side of the transistor Q9 as an input node and the connection node n5 of the transistors Q9 and Q10 as an output node.

[0062] As described above, the differentiation circuit 44 detects the amount of change in the pixel voltage Vsf by differential calculation. The amount of change in the pixel voltage Vsf indicates the amount of change in the amount of light incident on the photoelectric conversion element 26. The differentiation circuit 44 supplies the differentiated signal Vout to the quantizer 45 via the output node n5.

[0063] The quantizer 45 performs a comparison operation to compare the differential signal Vout with a threshold voltage. Based on the result of the comparison operation, the quantizer 45 detects an event indicating that the absolute value of the change in the amount of incident light has exceeded the threshold voltage, and outputs an event detection signal (on-event signal) COMP+ and an event detection signal (off-event signal) COMP-. The quantizer 45 includes transistors Q11 to Q14 and an inverter K1. For example, PMOS transistors are used as the transistors Q11 and Q13. For example, NMOS transistors are used as the transistors Q12 and Q14.

[0064] The transistors Q11 and Q12 are cascode-connected between a power supply voltage node and a reference voltage (e.g., ground) node. The output signal Vout of the differentiating circuit 44 is applied to the gate of the transistor Q11. The threshold voltage Vhigh is applied to the gate of the transistor Q12. The transistors Q11 and Q12 compare the output signal Vout with the threshold voltage Vhigh. Specifically, when the output signal Vout of the differentiating circuit 44 is lower than the threshold voltage Vhigh, the transistor Q11 turns on, and the event detection signal COMP+ output from the drain of the transistor Q11 via the inverter K1 goes low.

[0065] The transistors Q13 and Q14 are cascode-connected between a power supply voltage node and a reference voltage (e.g., ground) node. The output signal Vout of the differentiating circuit 44 is applied to the gate of the transistor Q13. The threshold voltage Vlow is applied to the gate of the transistor Q14. The transistors Q13 and Q14 compare the output signal Vout with the threshold voltage Vlow. Specifically, when the output signal Vout of the differentiating circuit 44 is higher than the threshold voltage Vlow, the transistor Q13 is turned off, and the event detection signal COMP- output from the drain of the transistor Q13 goes low.

[0066] First Embodiment FIG. 6 is a diagram illustrating a pixel configuration of a pixel array unit 25 of a photodetector 1 according to a first embodiment of the present disclosure. While FIG. 6 illustrates an example in which the pixel array unit 25 has 8×8 pixels, the pixel array unit 25 may have any number of pixels. The pixel array unit 25 according to the first embodiment includes a plurality of EVS pixels 27 arranged in a first direction X and a second direction Y. Some of the plurality of EVS pixels 27 are divided into two in the first direction X. The EVS pixels 27 are divided into two so that a phase difference can be detected by separately detecting an event signal in each of the divided pixel regions. In this specification, the left half of the divided pixel region is referred to as an L-phase EVS pixel 27L, and the right half of the divided pixel region is referred to as an R-phase EVS pixel 27R. An EVS pixel 27 that is not divided into two is referred to as a normal EVS pixel 27. Furthermore, the L-phase EVS pixel 27L and the R-phase EVS pixel 27R are collectively referred to as phase difference detection pixels, the pixel region of the L-phase EVS pixel 27L is referred to as the first photoelectric conversion region, and the pixel region of the R-phase EVS pixel 27R is referred to as the second photoelectric conversion region.

[0067] 6 , in the pixel array unit 25, a plurality of phase difference detection pixels are scattered among a plurality of normal EVS pixels 27. The pixel positions and the total number of the individual phase difference detection pixels are arbitrary. In this specification, the plurality of phase difference detection pixels scattered in the pixel array unit 25 are referred to as a plurality of pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R.

[0068] 7 is a block diagram of a portion related to an autofocus (hereinafter, referred to as AF) operation of the photodetector 1 according to the first embodiment of the present disclosure. As shown in FIG. 7 , the photodetector 1 according to the first embodiment includes a plurality of normal EVS pixels 27, a plurality of pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R, an event acquirer 2, an LR phase corrector 3, a defocus amount calculator 4, an event signal processor 5, and a sensor interface unit (sensor I / F) 6. Of these, the event acquirer 2, the LR phase corrector 3, the defocus amount calculator 4, the event signal processor 5, and the sensor I / F 6 are provided in, for example, the signal processing circuit 39 of FIG. 4 .

[0069] Each of the L-phase EVS pixel 27L, the R-phase EVS pixel 27R, and the normal EVS pixel 27 has a photoelectric conversion element 26 and an event detection circuit 41 shown in Figure 5, and outputs an event signal when a change in luminance exceeds a predetermined threshold.

[0070] The processing of the event acquisition unit 2 is performed by the row drive circuit 37, column drive circuit 38, and signal processing circuit 39 shown in FIG. 4 . The event acquisition unit 2 may have a scanner function that sequentially drives the plurality of L-phase EVS pixels 27L, the plurality of R-phase EVS pixels 27R, and the plurality of normal EVS pixels 27 using the row drive circuit 37 and the column drive circuit 38. The event acquisition unit 2 may also have an arbiter function that acquires event signals from the plurality of L-phase EVS pixels 27L, the plurality of R-phase EVS pixels 27R, and the plurality of normal EVS pixels 27, and arbitrates the acquired event signals. In this way, the event acquisition unit 2 has at least one of the functions of a scanner and an arbiter.

[0071] The LR phase correction unit 3 corrects and integrates two event signals output from the paired L-phase EVS pixel 27L and R-phase EVS pixel 27R to generate and output a new event signal.

[0072] The defocus amount calculation unit 4 calculates the defocus amount based on multiple event signals output from the multiple L-phase EVS pixels 27L and the multiple R-phase EVS pixels 27R. More specifically, the defocus amount calculation unit 4 detects the defocus amount based on the amount and direction of shift in pixel positions of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R pair that output the event signal among the multiple pairs of L-phase EVS pixel 27L and R-phase EVS pixel 27R. Furthermore, the defocus amount detection circuit detects whether the focal position of the optical system is located closer to the subject than the light-receiving surfaces of the multiple pixels, whether the focal position is located on the light-receiving surface, or whether the focal position is located farther from the subject than the light-receiving surface, based on the pixel positions of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R pair that output the event signal.

[0073] The event signal processing unit 5 performs various signal processing based on the new event signal output from the LR phase correction unit 3 and the event signal output from the normal EVS pixel 27. For example, the event signal processing unit 5 may generate an event image for each frame based on the new event signal output from the LR phase correction unit 3 and the event signal output from the normal EVS pixel 27. An event image is an image in which only pixels whose luminance change exceeds a threshold are emphasized, and is, for example, an image that includes only the outline of a moving object.

[0074] The sensor I / F 6 outputs the output signal of the event signal processing unit 5 to the outside.

[0075] The photodetector 1 according to the first embodiment includes a lens control unit 7. Alternatively, the lens control unit 7 is connected to the photodetector 1. The lens control unit 7 controls the lens module 8 based on the defocus amount output from the defocus amount calculation unit 4.

[0076] The lens module 8 has a lens 9 that constitutes an optical system, and a lens driver 10. The lens 9 may have a single lens structure or a multiple lens structure. The lens driver 10 moves the lens 9 along the optical axis based on the lens control amount from the lens controller 7, thereby performing automatic focus adjustment of the lens 9.

[0077] FIG. 8 is a schematic timing diagram of the photodetector 1 according to the first embodiment. The photoelectric conversion element 26 of each pixel in the pixel array section 25 continuously performs exposure operations over multiple frame periods. The signal processing circuit 39 generates an event image for each frame based on the event signal output from each pixel. In this embodiment, the exposure period and readout period overlap, and the entire period of one frame can be allocated to the exposure period and readout period. This allows the frame period to be shortened, enabling events to be detected at high speed.

[0078] The defocus amount calculation unit 4 calculates the defocus amount based on the difference between two event signals output from multiple pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R. The defocus amount calculation unit 4 according to this embodiment can calculate the defocus amount directly using the event signals output from the event detection circuit 41, and therefore can calculate the defocus amount quickly and with a smaller circuit scale than the method disclosed in Patent Document 2, which generates a digital signal for calculating the defocus amount separately from the event signals.

[0079] The lens driver 10 drives the lens 9 based on a lens control signal corresponding to the defocus amount to perform automatic focus adjustment. Because the frame cycle is short, AF processing can be performed at high speed. Even if the lens driver 10 drives the lens 9 beyond the appropriate position, the lens 9 can be quickly returned to the appropriate position in the next defocus calculation, allowing automatic focus adjustment to be performed quickly and with high precision.

[0080] 9 is a schematic timing diagram of a photodetector according to a comparative example. The photodetector according to the comparative example has L-phase pixels and R-phase pixels, which are obtained by dividing some of the gradation pixels in a pixel array section in which a plurality of gradation pixels are arranged in half. The gradation pixels are pixels that output pixel signals containing gradation information according to the amount of incident light.

[0081] The photodetector according to one comparative example has an exposure period, a readout period, a moving object detection period, a defocus amount calculation period, a defocus amount prediction period, and a lens drive period within one frame period. During the moving object detection period, a moving object OB is detected from a captured image. During the defocus amount calculation period, a defocus amount is calculated from the difference between pixel signals of L-phase pixels and R-phase pixels. During the defocus amount prediction period, two frames of captured images are compared to detect the direction and speed of movement of the moving object OB, and a future defocus amount is predicted. During the lens drive period, focus adjustment is performed by moving the lens 9 on the optical axis based on the predicted defocus amount.

[0082] As described above, in the comparative example, multiple periods must be provided within one frame period, making it difficult to shorten the frame period. Therefore, in the comparative example, it is difficult to shorten the latency for AF processing. In the present embodiment, as shown in FIG. 8 , an event signal can be output in a shorter time than a pixel signal containing gradation information, making it possible to shorten the frame period compared to the comparative example. Furthermore, in the present embodiment, the defocus amount is calculated from the event signal, so the defocus amount can be calculated more quickly than in the comparative example, which outputs a pixel signal, detects a moving object OB, and then calculates the defocus amount. As described above, in the present embodiment, the defocus amount can be calculated with a shorter latency than the comparative example.

[0083] 10A, 10B, and 10C are diagrams illustrating the defocus amount. Fig. 10A is a diagram illustrating an example of front focus, in which the focal position F of the lens 9 is located in front of the imaging surface. In this specification, the side above the optical axis 9x in Figs. 10A to 10C is defined as the right side of the pixel, and the side below the optical axis 9x is defined as the left side of the pixel, with the R-phase EVS pixel 27R arranged on the right side of the pixel and the L-phase EVS pixel 27L arranged on the left side.

[0084] 10A , in the case of front focus, light from subject OB (hereinafter referred to as subject light) that has passed through the right half of lens 9 (above optical axis 9x in FIG. 10A ) passes through focal position F and then enters L-phase EVS pixel 27L. Similarly, subject light that has passed through the left half of lens 9 (below optical axis 9x in FIG. 10A ) passes through focal position F and then enters R-phase EVS pixel 27R.

[0085] 10B is a diagram showing an example in which the imaging surface is located at the focal position F of the lens 9. In this case, light that has passed through the right and left sides of the lens 9 is incident on the center position of the boundary line between the L-phase EVS pixel 27L and the R-phase EVS pixel 27R.

[0086] 10C is a diagram showing an example of rear focus, in which the focal position F of the lens 9 is located behind the imaging surface. In the case of rear focus, as shown in FIG. 10C , subject light that has passed through the right half of the lens 9 is incident on the R-phase EVS pixel 27R, and subject light that has passed through the left half of the lens 9 is incident on the L-phase EVS pixel 27L. The focal position F of the subject light that has passed through the right and left sides of the lens 9 is located farther from the lens 9 than the R-phase EVS pixel 27R and the L-phase EVS pixel 27L.

[0087] Thus, during front focus, an event signal based on light that has passed through the right half of the lens 9 is output from the L-phase EVS pixel 27L, and an event signal based on light that has passed through the left half of the lens 9 is output from the R-phase EVS pixel 27R. On the other hand, during back focus, an event signal based on subject light that has passed through the right half of the lens 9 is output from the R-phase EVS pixel 27R, and an event signal based on subject light that has passed through the left half of the lens 9 is output from the L-phase EVS pixel 27L.

[0088] 11A, 11B, and 11C are diagrams showing the detection positions of ON-event signals when the moving object OB is moving from right to left (from top to bottom in the figure) in front focus. FIGS. 11A to 11C show an example in which the luminance of the moving object OB is brighter than the luminance of the surroundings, and the L-phase EVS pixel 27L and the R-phase EVS pixel 27R output ON-event signals upon receiving subject light from the moving object OB. The horizontal axis in the diagrams on the right side of FIGS. 11A to 11C represents time, and the vertical axis represents the pixel positions of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that output ON-event signals. The left side of the horizontal axis represents newer times, and the right side represents older times. The open circle plots in the diagrams on the right side of FIGS. 11A to 11C represent event signals based on subject light that has passed through the left half of the lens 9, and the triangular plots represent event signals based on subject light that has passed through the right half of the lens 9.

[0089] As described above, during front focus, subject light that has passed through the right half of the lens 9 passes through the focal position F and is then incident on the L-phase EVS pixel 27L. As shown in Figures 11A to 11C, when a moving object OB moves from right to left, as the moving object OB moves from the right to the left of the lens 9, subject light is incident on the L-phase EVS pixel 27L and R-phase EVS pixel 27R located further to the right, and an on-event signal is output. In this way, the direction and speed of movement of the moving object OB can be determined from the pixel positions of the L-phase EVS pixel 27L and R-phase EVS pixel 27R that output event signals.

[0090] 12A, 12B, and 12C are diagrams showing the detection positions of off-event signals when the moving object OB is moving from right to left and the focus is in front. Figures 12A to 12C show an example in which the luminance of the moving object OB is darker than the luminance of the surrounding area, and the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that receive subject light from the moving object OB output off-event signals. The horizontal axis in the right-hand diagrams of Figures 12A to 12C represents time, and the vertical axis represents the pixel positions of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that output off-event signals.

[0091] 12A to 12C , as in FIGS. 11A to 11C , when the focus is in front, the subject light that has passed through the right half of the lens 9 passes through the focal position F and then enters the L-phase EVS pixel 27L. As the moving object OB moves from the right to the left of the lens 9, the subject light enters the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that are located further to the right, and an off-event signal is output.

[0092] 13A, 13B, and 13C are diagrams showing the detection positions of ON-event signals during back focus when a moving object OB moves from right to left. Figures 13A to 13C show an example in which the luminance of the moving object OB is brighter than the luminance of the surrounding area, and the L-phase EVS pixel 27L and the R-phase EVS pixel 27R, which receive subject light from the moving object OB, output ON-event signals. The horizontal axis in the right-hand diagrams of Figures 13A to 13C represents time, and the vertical axis represents the pixel positions of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that output ON-event signals. The left side of the horizontal axis indicates a newer time, and the right side indicates an older time.

[0093] As described above, during back focus, subject light that has passed through the right half of the lens 9 is incident on the R-phase EVS pixel 27L. As shown in Figures 13A to 13C, as the moving object OB moves from the right to the left of the lens 9, subject light is incident on the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that are located further to the right, and an on-event signal is output.

[0094] 14A, 14B, and 14C are diagrams showing the detection positions of off-event signals during back focus when a moving object OB moves from right to left. Figures 14A to 14C show an example in which the luminance of the moving object OB is darker than the luminance of its surroundings, and the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that receive subject light from the moving object OB output off-event signals. The horizontal axis in the right-hand diagrams of Figures 14A to 14C represents time, and the vertical axis represents the pixel positions of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that output off-event signals.

[0095] 14A to 14C, during back focus, subject light that has passed through the right half of the lens 9 is incident on the R-phase EVS pixel 27L. As the moving object OB moves from the right to the left of the lens 9, subject light is incident on the L-phase EVS pixel 27L and the R-phase EVS pixel 27R that are positioned further to the right, and an off-event signal is output.

[0096] 15 is a diagram illustrating the processing of the defocus amount calculation unit 4. All on-event signals and off-event signals detected by multiple pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R in the pixel array unit 25 at multiple times are input to the defocus amount calculation unit 4. The defocus amount calculation unit 4 calculates the defocus amount based on the differences between the on-event signals detected by multiple pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R at each time and the differences between the off-event signals detected by multiple pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R at each time.

[0097] FIG. 16 is a diagram showing temporal changes in pixel positions of event signals detected by multiple pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R. The X-axis in FIG. 16 represents the first (horizontal) direction of the pixel array section 25, the Y-axis represents the second (vertical) direction, and the t-axis represents the time direction. In FIG. 16, the pixel positions of the event signals output from the L-phase EVS pixels 27L are represented by open circles, and the pixel positions of the event signals output from the R-phase EVS pixels 27R are represented by triangular plots. FIG. 16 shows an example in which ON-event signals and OFF-event signals are not distinguished. As can be seen from FIG. 16, the pixel positions of the event signals output from the L-phase EVS pixels 27L are shifted from the pixel positions of the event signals output from the R-phase EVS pixels 27R, and the amount of defocus is calculated based on the amount and direction of this shift.

[0098] Fig. 17 is a diagram showing the distribution of pixel positions of event signals output from multiple pairs of L-phase EVS pixels 27L and R-phase EVS pixels 27R. The horizontal axis of Fig. 17 represents the first direction X (X direction) on the imaging surface, and the vertical axis represents the second direction Y (Y direction). A dashed-line area 11L in Fig. 17 shows the distribution of pixel positions of event signals output from the L-phase EVS pixels 27L, and a dashed-dotted-line area 11R shows the distribution of pixel positions of event signals output from the R-phase EVS pixels 27R. The direction and amount of deviation between the dashed-line area 11L and the dashed-dotted-line area 11R in Fig. 17 represent the defocus amount.

[0099] To calculate the amount of shift, as shown in FIG. 18, the shift amount θ that maximizes the correlation with the R-phase event data is searched for while shifting the L-phase event data in the X direction. In the following, the R-phase side is fixed and the L-phase side is shifted, but L and R may be treated in reverse. First, as shown in FIG. 17, an L-phase count image I is generated from the L-phase event data and the R-phase event data. Ｌ and R phase count image I Ｒ This can be created by counting the number of events within a certain time period T for each pixel position (x, y) and polarity (p), as shown in the following equations (1) and (2). In equation (1), the L phase event data is calculated as (x Ｌ , y Ｌ , p Ｌ , t Ｌ), and in equation (2), the R phase event data is expressed as (x Ｒ , y Ｒ , p Ｒ , t Ｒ ) where δ represents the Dirac delta function.

[0100] Next, the L phase count image I Ｌ While shifting L by Δx in the X direction, the correlation r with the R phase count image I_R is calculated. (L and R may be handled in reverse.) The correlation r is calculated by the following equation (3): Ｌ and I Ｒ All we need to do is take the correlation of all elements (x, y, p) of I. Ｌ Bar and I Ｒ The bars are the average values ​​of all elements of I_L and I_R, respectively. Note that in this specification, a symbol in a formula with a bar above it will be referred to as a "symbol bar" for convenience.

[0101] As described above, the L-phase EVS pixel 27L and the R-phase EVS pixel 27R are provided as a pair, but it is not necessary to divide one EVS pixel 27 into two and provide the L-phase EVS pixel 27L and the R-phase EVS pixel 27R within one EVS pixel 27. Figures 19A, 19B, and 19C are diagrams showing variations in the arrangement of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R. Figure 19A shows an example in which one EVS pixel 27 is divided into two and provides the L-phase EVS pixel 27L and the R-phase EVS pixel 27R within one EVS pixel 27. In the case of Figure 19A, for the EVS pixel 27 divided into the L-phase EVS pixel 27L and the R-phase EVS pixel 27R, two pixel circuits (readout circuits) must be provided within one pixel region.

[0102] 19B shows an example in which half of one EVS pixel 27 is covered with a light-shielding member 28, and an L-phase EVS pixel 27L or an R-phase EVS pixel 27R is arranged in the remaining half. In the case of FIG. 19B , for example, two EVS pixels 27 arranged spaced apart in the first (horizontal) direction have an L-phase EVS pixel 27L arranged in one half and an R-phase EVS pixel 27R arranged in the other half. In this way, in FIG. 19B , two EVS pixels 27 are used to form a pair of an L-phase EVS pixel 27L and an R-phase EVS pixel 27R. In the case of FIG. 19B , one EVS pixel 27 has one pixel circuit (readout circuit) 29 for the L-phase EVS pixel 27L or the R-phase EVS pixel 27R.

[0103] 19C shows an example in which all EVS pixels 27 in the pixel array section 25 are divided into two, providing L-phase EVS pixels 27L and R-phase EVS pixels 27R. In the case of FIG. 19C , two pixel circuits (readout circuits) 29 need to be provided within the pixel region of each EVS pixel 27.

[0104] As shown in FIGS. 19A to 19C , in the pixel array unit 25 according to this embodiment, at least some of the pixels include a pair of a first photoelectric conversion region (L-phase EVS pixel 27L) and a second photoelectric conversion region (R-phase EVS pixel 27R). More specifically, as shown in FIG. 19A , some of the pixels include a pair of a first photoelectric conversion region and a second photoelectric conversion region, and the remaining pixels may output an event signal without including the first photoelectric conversion region and the second photoelectric conversion region. Alternatively, as shown in FIG. 19B , the pair of the first photoelectric conversion region and the second photoelectric conversion region may be disposed in different pixels, and the pixels including the first or second photoelectric conversion region may include a light-shielding member covering the region other than the first or second photoelectric conversion region. Alternatively, as shown in FIG. 19C , all of the EVS pixels 27 in the pixel array unit 25 may be divided into a first photoelectric conversion region and a second photoelectric conversion region.

[0105] 20 is a diagram showing a first example of the correspondence between pixels and pixel circuits 29. When one EVS pixel 27 is divided into an L-phase EVS pixel 27L and an R-phase EVS pixel 27R, two pixel circuits (readout circuits) 29 must be provided within the pixel region of one EVS pixel 27, and there is a risk that the two pixel circuits 29 will not fit within one pixel region. Therefore, it is conceivable to share a portion of the two pixel circuits 29 to reduce the circuit size.

[0106] Fig. 21 is a circuit diagram showing an example in which a part of two pixel circuits (readout circuits) 29 is shared. In Fig. 21, of the two pixel circuits 29, a pixel unit circuit 60 including a photoelectric conversion element 26 and a current-voltage converter 42 is provided for each pixel. Of the two pixel circuits 29, a shared circuit including a subtractor and a quantizer is shared by the two pixels.

[0107] A part of the pixel unit circuit is arranged on the first substrate, and the rest is arranged on the second substrate. In the example of Fig. 21, the photoelectric conversion element of the pixel unit circuit is arranged on the first substrate, and the rest is arranged on the second substrate, but circuit elements other than the photoelectric conversion element may be arranged on the first substrate.

[0108] Figure 20 shows an example in which one EVS pixel 27 is divided into an L-phase EVS pixel 27L and an R-phase EVS pixel 27R, and two pixel circuits (readout circuits) 29 are provided to share parts of the two pixel circuits 29, but it is also possible to provide only one pixel circuit 29.

[0109] 22 is a diagram showing a second example of a pixel and a pixel circuit (readout circuit) 29. In the second example, one EVS pixel 27 having an L-phase EVS pixel 27L and an R-phase EVS pixel 27R has one pixel circuit 29. The specific circuit configuration of the pixel circuit 29 will be described later.

[0110] 23 is a diagram showing a third example of pixels and pixel circuits (readout circuits) 29. In the third example, half of one EVS pixel 27 is covered with a light-shielding member 28, and an L-phase EVS pixel 27L or an R-phase EVS pixel 27R is disposed in the remaining half. In the third example, one pixel circuit 29 is provided for one EVS pixel 27 having an L-phase EVS pixel 27L or an R-phase EVS pixel 27R. The specific circuit configuration of the pixel circuit 29 will be described later.

[0111] 24 is a diagram showing a fourth example of pixels and pixel circuits (readout circuits) 29. In the fourth example, all EVS pixels 27 are divided into L-phase EVS pixels 27L and R-phase EVS pixels 27R. In the fourth example, one pixel circuit 29 is provided for one EVS pixel 27 having an L-phase EVS pixel 27L and an R-phase EVS pixel 27R.

[0112] 25 is a diagram showing a fifth example of pixels and pixel circuits 29. In the fifth example, all EVS pixels 27 are divided into L-phase EVS pixels 27L and R-phase EVS pixels 27R, and each of the L-phase EVS pixels 27L and the R-phase EVS pixels 27R has a pixel circuit 29. As a result, the fifth example has twice the number of pixel circuits 29 as the fourth example.

[0113] Figures 26A, 26B, and 26C are circuit diagrams showing first to third examples of pixel circuit (readout circuit) 29 of Figures 22 to 24. Pixel circuit 29 of Figures 26A to 26C has two photoelectric conversion elements (hereinafter referred to as a first photodiode and a second photodiode) 31a and 31b, two switches (hereinafter referred to as a first switch and a second switch) 35a and 35b, and a current-voltage conversion unit 42. The circuit configuration of current-voltage conversion unit 42 is not important.

[0114] A first switch 35a is connected between the cathode of the first photodiode 31a for the L-phase EVS pixel 27L and the input node of the current-voltage converter 42. A second switch 35b is connected between the cathode of the second photodiode 31b for the R-phase EVS pixel 27R and the input node of the current-voltage converter 42. In the pixel circuit 29 shown in FIG. 26A , one of the first switch 35a and the second switch 35b is on and the other is off. This allows circuit elements other than the photoelectric conversion elements 31a and 31b to be shared.

[0115] 26A , the pixel circuit 29 according to the first example turns on the first switch 35 a and turns off the second switch 35 b. In this case, the pixel circuit 29 detects a change in luminance of the L-phase EVS pixel 27L and outputs an ON event signal or an OFF event signal for the L-phase EVS pixel 27L.

[0116] 26B , the pixel circuit 29 according to the second example turns on the second switch 35 b and turns off the first switch 35 a. In this case, the pixel circuit 29 detects a change in luminance of the R-phase EVS pixel 27R and outputs an ON-event signal or an OFF-event signal for the R-phase EVS pixel 27R.

[0117] 26C , the pixel circuit 29 according to the third example turns on both the first switch 35 a and the second switch 35 b. In this case, the pixel circuit 29 detects a change in luminance of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R, and outputs a signal that combines the on-event signals or off-event signals of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R.

[0118] Although not shown in FIGS. 26A to 26C, the pixel circuit 29 shown in FIGS. 26A to 26C has a buffer 43, a differentiating circuit 44, and a quantizer 45, which have the same circuit configuration as those in FIG.

[0119] 27 and 28 are diagrams illustrating the processing operation of the LR phase correction unit 3 shown in FIG. The LR phase correction unit 3 receives an event signal output from the L-phase EVS pixel 27L in the pixel array unit 25, an event signal output from the R-phase EVS pixel 27R, and an event signal output from the non-halved EVS pixel 27. The LR phase correction unit 3 controls a threshold value so that the total number of two event signals output from the paired L-phase EVS pixel 27L and R-phase EVS pixel 27R is equal to the total number of event signals output from the surrounding normal EVS pixels 27. The threshold value is a threshold value set in the quantizer 45 shown in FIG. 5. Here, the event signals may be divided into on-event signals and off-event signals, or there may be no distinction between on-event signals and off-event signals.

[0120] The L-phase EVS pixel 27L and the R-phase EVS pixel 27R have different characteristics such as sensitivity because they have only half the aperture area of ​​the normal EVS pixel 27. If left as is, this could have a negative impact on downstream signal processing. However, by providing an LR phase correction unit 3 and controlling the threshold values ​​of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R, it is possible to align the event detection characteristics of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R with the characteristics of the normal EVS pixel 27.

[0121] More specifically, as shown in FIG. 28 , the LR phase correction unit 3 includes a first counter 12, a second counter 13, and a threshold control unit 14. The first counter 12 counts the total number of ON-event signals and OFF-event signals in the L-phase EVS pixel 27L and the R-phase EVS pixel 27R. The second counter 13 counts the total number of ON-event signals and OFF-event signals in the normal EVS pixel 27 that is not divided into two and is located near the L-phase EVS pixel 27L and the R-phase EVS pixel 27R. The threshold control unit 14 controls the thresholds of the quantizer 45 in FIG. 5 for the L-phase EVS pixel 27L and the R-phase EVS pixel 27R so that the count value of the first counter 12 matches the count value of the second counter 13.

[0122] As described above, the provision of the LR phase correction unit 3 makes it possible to align the event detection characteristics between the two-divided EVS pixels 27L and 27R and the non-divided EVS pixel 27. Therefore, when the AF operation is not performed, the two-divided EVS pixel 27 can be used as the normal EVS pixel 27, and event detection can be performed with all of the EVS pixels 27 in the pixel array unit 25.

[0123] In the above description, an example has been described in which the EVS pixel 27 outputs an on-event signal and an off-event signal, but the EVS pixel 27 may output an event signal without distinguishing between an on-event and an off-event. Alternatively, the EVS pixel 27 may output only either an on-event signal or an off-event signal as the event signal.

[0124] The LR phase correction unit 3 according to this embodiment may perform processing operations different from those shown in FIGS. 27 and 28 . FIG. 29 is a diagram showing a modified example of the processing operations of the LR phase correction unit 3 shown in FIG. 7 . In the modified example shown in FIG. 29 , the LR phase correction unit 3 adjusts the event occurrence rates of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R based on the event occurrence rates of the normal EVS pixels 27 located around the EVS pixel 27 divided into the L-phase EVS pixel 27L and the R-phase EVS pixel 27R. The event occurrence rate is adjusted by controlling the threshold value of the quantizer 45 shown in FIG. 5 . 29 , the normal EVS pixels 27 located around the EVS pixel 27 divided into the L-phase EVS pixel 27L and the R-phase EVS pixel 27R are, for example, eight surrounding normal EVS pixels 27, and the event occurrence rates of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R are adjusted so as to match the average value of the event occurrence rates of these normal EVS pixels 27. Alternatively, the event occurrence rates of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R are adjusted based on the average value of the event occurrence rates of the surrounding normal EVS pixels 27 for multiple frames.

[0125] As described above, in the first embodiment, the EVS pixels 27 divided into the L-phase EVS pixels 27L and the R-phase EVS pixels 27R are scattered within the pixel array unit 25, and the defocus amount is calculated from the difference between the distribution of the L-phase EVS pixels 27L and the distribution of the R-phase EVS pixels 27R that have detected an on-event signal or an off-event signal, thereby enabling automatic focus adjustment of the lens 9. In particular, in the present embodiment, the defocus amount is calculated using the event signal output from the event detection circuit 41 as is, which simplifies the circuit configuration for calculating the defocus amount and enables the defocus amount to be calculated quickly, thereby enabling accurate tracking of a moving object OB moving at high speed.

[0126] Furthermore, in this embodiment, the threshold values ​​of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R are controlled so that the total number of on-event signals and off-event signals detected by the EVS pixel 27 divided into the L-phase EVS pixel 27L and the R-phase EVS pixel 27R matches the total number of on-event signals and off-event signals detected by the normal EVS pixel 27 that is not divided into two. This makes it possible to match the event detection characteristics of the divided EVS pixel 27 and the normal EVS pixel 27 that is not divided into two, enabling AF processing and event detection to be performed with high precision and high speed.

[0127] 30 is a diagram illustrating a photodetector 1 according to a second embodiment. The photodetector 1 according to the second embodiment has the same hardware configuration as the photodetector 1 according to the first embodiment, but differs from the first embodiment in the processing operation of the signal processing circuit 39 in FIG.

[0128] The signal processing circuit 39 in the photodetector 1 according to the second embodiment includes an information processing unit 15. The information processing unit 15 uses two event signals output from a pair of the L-phase EVS pixel 27L and the R-phase EVS pixel 27R as input parameters to construct an information processing model through machine learning that outputs a defocus state notification event corresponding to the input parameters. Furthermore, by inputting two event signals output from the new L-phase EVS pixel 27L and the new R-phase EVS pixel 27R into the constructed, trained information processing model, the information processing model outputs a corresponding defocus state notification event. The defocus state notification events include a front-focus detection event signal, a positive-focus detection event signal, and a back-focus detection event signal. The defocus state notification events output from the information processing model are sent to the lens driver 10 shown in FIG. 7 .

[0129] The information processing model is, for example, an SNN (Spiking Neural Network) 50. The SNN 50 is a network that mimics the behavior of spike transmission between neurons in the human brain, and can describe input / output relationships using only addition operations.

[0130] 31 is a diagram showing the basic structure of an SNN 50. The SNN 50 has an input layer 17, an intermediate layer 18, and an output layer 19. Each layer has a plurality of nodes (neurons) 50n, and weights are set for the paths connecting the nodes in each layer, and the weights are updated during repeated machine learning processes.

[0131] The input layer 17 of the SNN 50 includes, for example, a group of neurons that input event information from the L-phase and R-phase EVS pixels. The output layer 19 of the SNN 50 includes, for example, neurons that detect front focus, positive focus, and back focus. When the front focus detection neuron outputs an event, a front focus detection event is sent to the lens control unit 7 in FIG. 7, and the lens control unit 7 outputs a signal to the lens driving unit 10 in FIG. 7 to drive the lens in a direction to correct the front focus. Conversely, when the back focus detection neuron outputs an event, a back focus detection event is sent to the lens control unit 7 in FIG. 7, and the lens control unit 7 outputs a signal to the lens driving unit 10 in FIG. 7 to drive the lens in a direction to correct the back focus.

[0132] In this way, in the second embodiment, by constructing in advance an information processing model that infers the defocus amount from event information, the defocus amount of a moving object OB moving at high speed can be quickly inferred and tracked with high accuracy.

[0133] Third Embodiment In the first and second embodiments, an example was described in which the EVS pixel 27 that outputs an event signal is divided into two halves, an L-phase EVS pixel 27L and an R-phase EVS pixel 27R, to calculate the defocus amount. However, the defocus amount may also be calculated using a SPAD (Single Photon Avalanche Diode) pixel.

[0134] 32 is a diagram showing a pixel array unit 25 of a photodetector 1 according to the third embodiment. The pixel array unit 25 according to the third embodiment includes a plurality of pixels arranged in a first direction X and a second direction Y. Each pixel has a light receiving element. The light receiving element includes, for example, a SPAD 27S. Some of the plurality of pixels arranged in the pixel array unit 25 are each divided into two in the first direction X. In this specification, the divided pixels are referred to as phase difference detection pixels, and the left half of the phase difference detection pixels are referred to as L-phase SPAD pixels 27SL and the right half as R-phase SPAD pixels 27SR.

[0135] 33 is a circuit diagram of each pixel according to Embodiment 3. Each of the non-halved SPAD pixel 27S, the L-phase SPAD pixel 27SL, and the R-phase SPAD pixel 27SR has the circuit configuration shown in FIG.

[0136] As shown in FIG. 33, the pixel circuit 29 according to the third embodiment includes a light receiving unit 51, a counter 52, and a comparator 53.

[0137] The light receiving unit 51 includes a light receiving element 26a, an inverter 54, a recharge transistor Q21, and detection transistors Q22 and Q23. In the example of FIG. 4, the transistors Q21 and Q22 are PMOS (P-channel Metal-Oxide-Semiconductor) transistors. The transistor Q23 is an NMOS (N-channel Metal-Oxide-Semiconductor) transistor. The conductivity types of the transistors Q21 to Q23 can be changed as desired. Either one or both of the transistors Q21 and Q22 may be an NMOS transistor, and the transistor Q23 may be a PMOS transistor.

[0138] The light-receiving element 26a is, for example, a SPAD. The light-receiving element 26a detects photons incident on the pixel 20 and generates an electric charge. One of the anode and cathode of the light-receiving element 26a (the cathode in the example of FIG. 4) is connected to the drain of the transistor Q21. The other of the anode and cathode of the light-receiving element 26a (the anode in the example of FIG. 4) is connected to a node having a voltage level lower than the power supply voltage node VDD.

[0139] The recharge transistor Q21 switches whether or not the light receiving element 26a is charged. The drain of the recharge transistor Q21 is connected to the detection node n6. The source of the recharge transistor Q21 is connected to the power supply voltage node VDD. A control signal XRST is input to the gate of the recharge transistor Q21. The recharge transistor Q21 is turned on when, for example, a low level control signal XRST is input.

[0140] The detection transistors Q22 and Q23 constitute a detection circuit 55. The detection circuit 55 outputs a pulse signal PLS based on whether or not the light-receiving element 26a detects a photon. The pulse signal PLS is input to the counter 52.

[0141] The drain of the detection transistor Q22 is connected to the output node N2. The source of the detection transistor Q23 is connected to the power supply voltage node VDD. The detection transistor Q22 receives a detection signal Vdt from a detection node n6. The detection transistor Q22 turns on when the voltage level of the detection signal Vdt falls below a predetermined threshold.

[0142] The source of the detection transistor Q23 is connected to the ground voltage node. The drain of the detection transistor Q23 is connected to the output node N2. An inverted signal of the control signal XRST is input to the gate of the detection transistor Q23 via an inverter 54. The detection transistor Q23 is turned off when, for example, an inverted signal of the high-level control signal XRST is input.

[0143] The detection circuit 55 may have an inverter or the like that inverts the voltage of the output node N2, but this is not shown in FIG.

[0144] The counter 52 updates its count value every time a high-level pulse signal PLS is output from the detection circuit 55. The counter 52 is an up / down counter 52 that operates as an up counter 52 until the count value reaches an upper limit, and then operates as a down counter 52 until the count value reaches an up / down counter 52.

[0145] The comparator 53 outputs a positive event signal when the count value of the counter 52 is equal to or less than a first threshold, and outputs a negative event signal when the count value is equal to or greater than a second threshold. The positive event signal is output when the luminance change is equal to or greater than a predetermined threshold, and the negative event signal is output when the luminance change is equal to or less than the predetermined threshold.

[0146] 34 is a timing diagram of the photodetector 1 according to the third embodiment, showing the timing of the subframe synchronization signal, the count value of the counter 52, the processing timing of the signal processing circuit 39, the overflow signal of the counter 52, and the subject motion determination signal.

[0147] The photodetector 1 determines the movement of the subject for each subframe. Fig. 34 shows the timing within the periods of subframes SF1, SF2, and SFn. One main frame includes multiple subframes SF1 to SFn. In Fig. 34, subframes SF1, SF2, and SFn start at times t1, t7, and t11, respectively.

[0148] During the period from time t1 to t2 in subframe SF1, counter 52 counts up each time light receiving element 26a detects a photon. When the count value of counter 52 reaches its upper limit at time t2, an overflow signal is output. During the period from time t2 to t3, counter 52 counts down each time light receiving element 26a detects a photon. Counter 52 stops counting in subframe SF1 at time t3.

[0149] Then, at time t4, the comparator 53 performs threshold determination processing to determine whether the count value is greater than the first threshold value Th1. In this case, the count value is equal to or less than the first threshold value Th1, so the comparator 53 outputs a positive event signal. At time t5, the signal processing circuit 39 starts calculation processing of the defocus amount. At time t6, the signal processing circuit 39 outputs a movement signal (defocus amount).

[0150] Subsequently, during the period from time t7 to t8 in subframe SF2, counter 52 counts up each time light receiving element 26a detects a photon. At time t8, the count value of counter 52 reaches its upper limit, and an overflow signal is output. During the period from time t8 to t9, counter 52 counts down each time light receiving element 26a detects a photon.

[0151] At time t9, the counter 52 stops counting in subframe SF1, and the comparator 53 determines whether the count value is greater than the second threshold value Th2. In this case, the count value is equal to the first threshold value Th1, so the comparator 53 outputs neither a positive event signal nor a negative event signal. At time t10, the signal processing circuit 39 starts calculation of the defocus amount. In this case, no movement signal is output.

[0152] Thereafter, during the period from time t11 to t12 in subframe SFn, the counter 52 counts up each time the light-receiving element 26a detects a photon. At time t12, the count value of the counter 52 reaches its upper limit, and an overflow signal is output. During the period from time t12 to t13, the counter 52 counts down each time the light-receiving element 26a detects a photon. At time t13, the counter 52 stops counting in subframe SFn, and the comparator 53 determines whether the count value is greater than the second threshold value Th2. In this case, because the count value is greater than or equal to the second threshold value Th2, the comparator 53 outputs a minus event signal. At time t14, the signal processing circuit 39 starts calculating the defocus amount. At time t15, a movement signal (defocus amount) is output.

[0153] In this way, in the third embodiment, two types of event signals (for example, a positive event signal and a negative event signal) can be generated by comparing the number of times that photons are detected by the SPAD pixel 27S with a threshold value. Therefore, the defocus amount can be calculated using the two types of event signals detected by the SPAD pixel 27S instead of the EVS pixel 27.

[0154] (Fourth Embodiment) The fourth embodiment is characterized in that the defocus amount is calculated after the egomotion cancellation process is performed.

[0155] Fig. 35 is a block diagram of a portion related to the AF operation of the photodetector 1 according to the fourth embodiment. The block diagram of Fig. 35 has a configuration in which an egomotion cancellation unit (vibration suppression unit) 56 is added to the configuration of Fig. 7. The egomotion cancellation unit 56 in the fourth embodiment can be added to the photodetector 1 according to the first to third embodiments described above.

[0156] Egomotion cancellation unit 56 is connected to event acquisition unit 2. Egomotion cancellation unit 56 detects the direction and amount of movement of photodetection device 1 using an acceleration sensor or the like attached to photodetection device 1, and generates on-event signals and off-event signals in which the direction and amount of movement of photodetection device 1 are cancelled from the on-event signals and off-event signals detected by photodetection device 1.

[0157] 36 is a timing diagram of the photodetector 1 according to the fourth embodiment. As shown in Fig. 36, the egomotion cancellation unit 56 performs processing (time t1 to t2), then calculates the defocus amount based on the on-event signal and the off-event signal (time t2 to t3), and the lens driver 10 drives the lens 9 based on the calculated defocus amount (time t3 to t4).

[0158] In this way, in the fourth embodiment, the defocus amount is calculated after canceling the movement of the photodetector 1 itself, so that the defocus amount can be calculated with high accuracy even if the photodetector 1 is installed in a place with large vibrations, such as a vehicle.

[0159] <Application to a Mobile Body> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0160] FIG. 37 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0161] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 37, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0162] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0163] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0164] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0165] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0166] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0167] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0168] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0169] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0170] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying information to vehicle occupants or the outside of the vehicle. In the example of Fig. 37, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0171] FIG. 38 is a diagram showing an example of the installation position of the imaging unit 12031.

[0172] In FIG. 38, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0173] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0174] 38 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0175] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0176] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0177] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0178] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0179] The above describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to, for example, the image capturing unit 12031 among the components described above. Specifically, by applying the technology according to the present disclosure to the image capturing unit 12031, it is possible to obtain a captured image that is easier to see, thereby reducing driver fatigue.

[0180] The present technology may be configured as follows: (1) A photodetector device comprising: a plurality of pixels that output an event signal when a change in luminance of incident light incident through an optical system exceeds a predetermined threshold; and a defocus amount detection circuit that detects a defocus amount of the incident light based on the event signal, wherein at least some of the plurality of pixels have at least one of a first photoelectric conversion region and a second photoelectric conversion region that detects a change in luminance within a divided pixel region and outputs the event signal, and the defocus amount detection circuit detects the defocus amount based on two event signals output from a pair of the first photoelectric conversion region and the second photoelectric conversion region. (2) The photodetector device described in (1), wherein the plurality of pixels include a plurality of pairs of the first photoelectric conversion region and the second photoelectric conversion region, and the defocus amount detection circuit detects the defocus amount based on a shift amount and a shift direction between a pixel position of the first photoelectric conversion region that outputs the event signal and a pixel position of the second photoelectric conversion region that outputs the event signal, among the plurality of pairs of the first photoelectric conversion region and the second photoelectric conversion region. (3) The photodetector according to (2), wherein the defocus amount detection circuit detects whether a focal position of the optical system is closer to the subject than the light-receiving surfaces of the plurality of pixels, whether the focal position is on the light-receiving surface, or whether the focal position is farther from the subject than the light-receiving surface, based on the pixel position of the first photoelectric conversion region that output the event signal and the pixel position of the second photoelectric conversion region that output the event signal. (4) The photodetector according to any one of (1) to (3), wherein at least some of the plurality of pixels have the first photoelectric conversion region and the second photoelectric conversion region that form a pair. (5) The photodetector according to (4), wherein some of the plurality of pixels have the first photoelectric conversion region and the second photoelectric conversion region that form a pair, and pixels other than the some of the plurality of pixels output the event signal without having the first photoelectric conversion region and the second photoelectric conversion region.(6) The photodetector according to any one of (1) to (3), wherein the paired first and second photoelectric conversion regions are arranged in different pixels, and a pixel having the first or second photoelectric conversion region has a light-shielding member covering an area other than the first or second photoelectric conversion region. (7) The photodetector according to any one of (1) to (6), wherein at least some of the pixels have a correction unit that corrects and integrates two event signals output from the paired first and second photoelectric conversion regions to output a new event signal. (8) The photodetector according to (7), wherein the correction unit controls the threshold so that a total number of the two event signals output from the paired first and second photoelectric conversion regions is equal to a total number of the event signals output from pixels that do not have the first and second photoelectric conversion regions and are located around the pixel having the paired first and second photoelectric conversion regions. (9) The photodetector according to (7), wherein the correction unit corrects the two event signals output from the first photoelectric conversion region and the second photoelectric conversion region based on an occurrence rate of the event signals output from a plurality of pixels that do not have the first photoelectric conversion region and the second photoelectric conversion region and are located around a pixel having a pair of the first photoelectric conversion region and the second photoelectric conversion region. (10) The photodetector according to (9), wherein the plurality of pixels are eight pixels surrounding a pixel having the first photoelectric conversion region and the second photoelectric conversion region. (11) The photodetector device described in any one of (1) to (10), wherein the defocus amount detection circuit includes: an information processing unit that uses two event signals output from the paired first and second photoelectric conversion regions as input parameters to construct an information processing model by machine learning, which outputs a defocus state notification event corresponding to the input parameters; and inputs new input parameters into the constructed, trained information processing model, and outputs a defocus state notification event corresponding to the new input parameters from the information processing model.(12) The photodetector according to (11), wherein the information processing model has an input layer to which the two event signals are input and an output layer that outputs the defocus state notification event, and the output layer outputs a plurality of event signals representing the defocus amount. The photodetector according to (12), wherein the defocus state notification event includes a front-focus detection event signal, a positive-focus detection event signal, or a back-focus detection event signal. (14) The photodetector according to any one of (11) to (13), wherein the information processing model is an SNN (Spiking Neural Network). (15) The photodetector according to any one of (1) to (14), comprising: a first substrate disposed on a light incident surface side; and a second substrate disposed on an opposite side to the light incident surface and stacked on the first substrate, wherein at least a portion of the plurality of pixels is disposed on the first substrate, and at least a portion of the defocus amount detection circuit is disposed on the second substrate. (16) The photodetector according to (15), wherein each of the plurality of pixels has an event detection circuit that generates the event signal, and at least a part of the event detection circuit is disposed on the second substrate. (17) The photodetector according to any one of (1) to (16), wherein the plurality of pixels output an ON event signal when a change in luminance in a direction in which the luminance of incident light increases exceeds the threshold, and output an OFF event signal when a change in luminance in a direction in which the luminance of incident light decreases exceeds the threshold, and the defocus amount detection circuit detects the defocus amount based on the ON event signal and the OFF event signal. (18) The photodetector according to any one of (1) to (16), wherein the plurality of pixels include: a plurality of light-receiving pixels that output a signal indicating whether or not an incident photon has been detected, a counter for each of the plurality of light-receiving pixels that measures the number of times photons have been detected, and a comparator that outputs the event signal when the number of times measured by the counter exceeds the threshold.(19) The photodetector according to any one of (1) to (18), further comprising: a vibration suppression unit that suppresses vibration of a pixel array unit having the plurality of pixels, wherein the defocus amount detection circuit detects the defocus amount based on the event signal output from the pixel array unit after the vibration suppression unit has suppressed vibration of the pixel array unit. (20) An electronic device comprising: the photodetector according to any one of (1) to (19), the optical system, and an optical system driver that moves the optical system along an optical axis to perform focus adjustment based on the defocus amount.

[0181] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0182] DESCRIPTION OF SYMBOLS 1 Photodetector, 2 Event acquisition unit, 3 LR phase correction unit, 4 Defocus amount calculation unit, 5 Event signal processing unit, 6 Sensor interface unit (sensor I / F), 7 Lens control unit, 8 Lens module, 9 Lens, 9x Optical axis, 10 Lens driving unit, 11L Dashed line range, 11R Dot-dash line range, 12 First counter, 13 Second counter, 14 Threshold control unit, 15 Information processing unit, 17 Input layer, 18 Middle layer, 19 Output layer, 20 Pixel, 21 Light receiving chip, 22 Detection chip, 23 Via arrangement unit, 24 Light receiving unit, 25 Pixel array unit, 26 Photoelectric conversion element, 26a Light receiving element, 27 EVS pixel (normal EVS pixel), 27L L-phase EVS pixel, 27R R-phase EVS pixel, 27S SPAD pixel, 27SL L-phase SPAD pixel, 27SR R-phase SPAD pixel, 28 light-shielding member, 29 pixel circuit (readout circuit), 30 electronic device, 31 imaging lens, 31a first photodiode, 31a photoelectric conversion element, 31b second photodiode, 32 image processing unit, 33 recording unit, 34 control unit, 35a first switch, 35b second switch, 36 event detection unit, 37 row drive circuit, 38 column drive circuit, 39 signal processing circuit, 41 event detection circuit, 42 current-voltage conversion unit, 43 buffer, 44 differentiation circuit, 45 quantizer, 50n node (neuron), 51 light receiving unit, 52 up / down counter, 53 comparator, 54 inverter, 55 detection circuit, 56 egomotion cancellation unit, 60 pixel unit circuit

Claims

1. A photodetection device comprising: a plurality of pixels that output an event signal when a change in luminance of incident light incident through an optical system exceeds a predetermined threshold; and a defocus amount detection circuit that detects a defocus amount of the incident light based on the event signal, wherein at least some of the plurality of pixels have at least one of a first photoelectric conversion region and a second photoelectric conversion region that detects a change in luminance within a divided pixel region and outputs the event signal, and the defocus amount detection circuit detects the defocus amount based on the two event signals output from the pair of the first photoelectric conversion region and the second photoelectric conversion region.

2. The photodetection device of claim 1, wherein the plurality of pixels include a plurality of pairs of the first photoelectric conversion region and the second photoelectric conversion region, and the defocus amount detection circuit detects the amount of defocus based on the amount and direction of deviation between the pixel position of the first photoelectric conversion region that outputs the event signal and the pixel position of the second photoelectric conversion region that outputs the event signal, among the plurality of pairs of the first photoelectric conversion region and the second photoelectric conversion region.

3. The optical detection device described in claim 2, wherein the defocus amount detection circuit detects whether the focal position of the optical system is located closer to the subject than the light receiving surface of the plurality of pixels, whether the focal position is located on the light receiving surface, or whether the focal position is located farther from the subject than the light receiving surface based on the pixel position of the first photoelectric conversion area that output the event signal and the pixel position of the second photoelectric conversion area that output the event signal.

4. The photodetection device according to claim 1, wherein at least some of the plurality of pixels have a pair of the first photoelectric conversion region and the second photoelectric conversion region.

5. The photodetection device according to claim 4, wherein some of the plurality of pixels have a pair of the first photoelectric conversion region and the second photoelectric conversion region, and pixels other than the some of the plurality of pixels do not have the first photoelectric conversion region or the second photoelectric conversion region and output the event signal.

6. The photodetection device of claim 1, wherein the pair of first and second photoelectric conversion regions are arranged in different pixels, and a pixel having the first or second photoelectric conversion region has a light-shielding member that covers an area other than the first or second photoelectric conversion region.

7. The photodetection device according to claim 1, wherein at least some of the plurality of pixels have a correction section that outputs a new event signal obtained by correcting and integrating two event signals output from a pair of the first photoelectric conversion region and the second photoelectric conversion region.

8. The optical detection device described in claim 7, wherein the correction unit controls the threshold so that the total number of two event signals output from a pair of the first and second photoelectric conversion regions is equal to the total number of event signals output from pixels that do not have the first and second photoelectric conversion regions and are located around a pixel having a pair of the first and second photoelectric conversion regions.

9. The optical detection device described in claim 7, wherein the correction unit corrects the two event signals output from the first photoelectric conversion region and the second photoelectric conversion region based on the occurrence rate of a plurality of the event signals output from a plurality of pixels that do not have the first photoelectric conversion region and the second photoelectric conversion region and are located around a pixel having a pair of the first photoelectric conversion region and the second photoelectric conversion region.

10. The photodetection device according to claim 9, wherein the plurality of pixels are eight pixels surrounding a pixel having the first photoelectric conversion region and the second photoelectric conversion region.

11. The optical detection device of claim 1, wherein the defocus amount detection circuit has an information processing unit that uses two event signals output from the pair of the first photoelectric conversion region and the second photoelectric conversion region as input parameters to construct an information processing model by machine learning that outputs a defocus situation notification event corresponding to the input parameters, and inputs new input parameters into the constructed, trained information processing model and outputs a defocus situation notification event corresponding to the new input parameters from the information processing model.

12. The optical detection device according to claim 11, wherein the information processing model has an input layer to which the two event signals are input and an output layer that outputs the defocus state notification event, and the output layer outputs a plurality of event signals that represent the defocus amount.

13. The optical detection device according to claim 12, wherein the defocus state notification event includes a front focus detection event signal, a positive focus detection event signal, or a rear focus detection event signal.

14. The optical detection device according to claim 11, wherein the information processing model is a Spiking Neural Network (SNN).

15. The photodetection device according to claim 1, comprising: a first substrate arranged on the light incident surface side; and a second substrate arranged on the opposite side to the light incident surface and stacked on the first substrate, wherein at least a portion of the plurality of pixels are arranged on the first substrate, and at least a portion of the defocus amount detection circuit is arranged on the second substrate.

16. The photodetection device according to claim 15, wherein each of the plurality of pixels has an event detection circuit that generates the event signal, and at least a portion of the event detection circuit is disposed on the second substrate.

17. The photodetection device of claim 1, wherein the plurality of pixels output an on-event signal when a change in luminance in a direction in which the luminance of the incident light increases exceeds the threshold, and output an off-event signal when a change in luminance in a direction in which the luminance of the incident light decreases exceeds the threshold, and the defocus amount detection circuit detects the defocus amount based on the on-event signal and the off-event signal.

18. The photodetection device of claim 1, wherein the plurality of pixels include: a plurality of light-receiving pixels that output a signal indicating whether or not an incident photon has been detected; a counter for each of the plurality of light-receiving pixels that measures the number of times a photon has been detected; and a comparator that outputs the event signal when the number of times measured by the counter exceeds the threshold value.

19. The photodetection device according to claim 1, further comprising a vibration suppression section that suppresses vibration of a pixel array section having the plurality of pixels, and wherein the defocus amount detection circuit detects the defocus amount based on the event signal output from the pixel array section after the vibration suppression section has suppressed the vibration of the pixel array section.

20. An electronic device comprising: the photodetector according to claim 1; the optical system; and an optical system driver that moves the optical system along an optical axis based on the amount of defocus to adjust focus.

Images