Photoelectric conversion device and photodetection system

Abstract

A photoelectric conversion device is provided that can acquire a dynamic range image and detect an event while suppressing power consumption. [Solution] The photoelectric conversion device has a photoelectric conversion unit that outputs a pulse signal in response to the incidence of photons, a first counter that counts the pulse signal, an exposure control circuit that compares the count value of the first counter with a predetermined threshold value multiple times during the exposure period of one frame and controls the count period of the pulse signal by the first counter in accordance with the result of the comparison, a second counter that counts each time the count value of the first counter is determined to be less than or equal to the threshold value, and a comparison circuit that outputs event information in accordance with the result of comparing the count value of the second counter in the first frame with the count value of the second counter in a second frame prior to the first frame.

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device and a photodetection system. [Background technology]

[0002] The Single Photon Avalanche Diode (SPAD) is known as a detector capable of detecting weak light at the single photon level. SPADs use the avalanche multiplication phenomenon, which occurs due to a strong electric field induced in the PN junction of a semiconductor, to amplify the signal charge excited by a photon by several to several million times. By converting the current generated by the avalanche multiplication phenomenon into a pulse signal and counting the number of pulse signals, it is possible to directly measure the number of incident photons.

[0003] Patent Document 1 describes a photoelectric conversion device that combines a recharge method that periodically recharges the SPAD with a method that performs counting operations for each pixel with an accumulation time that corresponds to the brightness of the subject. The photoelectric conversion device described in Patent Document 1 can achieve an expanded dynamic range and low power consumption. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-039400 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, event detection sensors that detect changes in brightness and output them as events have been attracting attention, and studies are being conducted to incorporate an event detection function into SPAD sensors. However, if an event detection function were to be incorporated into the photoelectric conversion device described in Patent Document 1, a multi-bit information retention memory and a large-scale comparison circuit would be required, raising concerns about increased power consumption as the pixel circuit scale increases.

[0006] An object of the present invention is to provide a photoelectric conversion device and a photodetection system that can acquire a dynamic range image and detect an event while suppressing power consumption. [Means for solving the problem]

[0007] According to one disclosure of the present specification, there is provided a photoelectric conversion device having a photoelectric conversion unit that outputs a pulse signal in response to the incidence of a photon, a first counter that counts the pulse signal, an exposure control circuit that compares the count value of the first counter with a predetermined threshold value multiple times during the exposure period of one frame and controls the count period of the pulse signal by the first counter in accordance with the result of the comparison, a second counter that counts each time the count value of the first counter is determined to be equal to or less than the threshold value, and a comparison circuit that outputs event information in accordance with the result of comparing the count value of the second counter in a first frame with the count value of the second counter in a second frame prior to the first frame.

[0008] According to another disclosure of the present specification, there is provided a photoelectric conversion device having a photoelectric conversion unit that outputs a pulse signal in response to incidence of a photon, a first counter that counts the pulse signal, a second counter that counts the time until the first counter reaches a predetermined threshold, and a comparison circuit that outputs event information in response to a result of comparing the count value of the second counter in a first frame with the count value of the second counter in a second frame prior to the first frame. [Effects of the Invention]

[0009] According to the present invention, it is possible to realize a photoelectric conversion device and a photodetection system that are capable of acquiring a dynamic range image and detecting an event while suppressing power consumption. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram (part 1) showing a schematic configuration of a photoelectric conversion device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram (part 2) showing a schematic configuration of the photoelectric conversion device according to the first embodiment. [Figure 3] 1 is a block diagram showing an example of the configuration of a pixel of a photoelectric conversion device according to a first embodiment. [Figure 4] 1 is a perspective view showing an example of the configuration of a photoelectric conversion device according to a first embodiment. [Figure 5] FIG. 2 is a diagram illustrating a basic operation of a pixel in the photoelectric conversion device according to the first embodiment. [Figure 6] 1 is a block diagram showing an example of the configuration of a pixel of a photoelectric conversion device according to a first embodiment. [Figure 7] 4 is a flowchart showing a method for driving pixels of the photoelectric conversion device according to the first embodiment. [Figure 8] FIG. 3 is a timing chart (part 1) showing an example of operation of a pixel of the photoelectric conversion device according to the first embodiment. [Figure 9] FIG. 2 is a timing chart (part 2) showing an example of operation of a pixel of the photoelectric conversion device according to the first embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of a pixel of a photoelectric conversion device according to a second embodiment. [Figure 11] FIG. 10 is a block diagram showing an example of the configuration of a pixel of a photoelectric conversion device according to a third embodiment. [Figure 12] FIG. 10 is a block diagram showing a schematic configuration of a light detection system according to a fifth embodiment. [Figure 13] FIG. 13 is a block diagram showing a schematic configuration of a range image sensor according to a sixth embodiment. [Figure 14] FIG. 13 is a schematic diagram showing an example of the configuration of an endoscopic surgery system according to a seventh embodiment. [Figure 15] FIG. 13 is a schematic diagram showing an example of the configuration of a moving body according to an eighth embodiment. [Figure 16] FIG. 13 is a block diagram showing a schematic configuration of a light detection system according to an eighth embodiment. [Figure 17] FIG. 13 is a flowchart showing the operation of the light detection system according to the eighth embodiment. [Figure 18] FIG. 13 is a schematic diagram showing a schematic configuration of a light detection system according to a ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each of the following embodiments, an apparatus for image capture will be mainly described as an example of a photoelectric conversion device. However, each embodiment is not limited to this apparatus for image capture, and can also be applied to other examples of photoelectric conversion devices. For example, there are distance measurement devices (devices for measuring distance using focus detection or TOF (Time Of Flight)), photometry devices (devices for measuring the amount of incident light, etc.), etc.

[0012] Note that the conductivity types of the transistors described in the following embodiments are merely examples and are not limited to the conductivity types described in the embodiments. The conductivity types described in the embodiments can be changed as appropriate, and the potentials of the gate, source, and drain of the transistor can be changed as appropriate in accordance with this change. For example, in the case of a transistor that operates as a switch, the low and high levels of the potential supplied to the gate can be reversed relative to the description in the embodiments in accordance with the change in conductivity type.

[0013] In the following embodiments, the connection between elements of a circuit may be described. In this case, even if another element is interposed between the elements of interest, the elements of interest are treated as being connected to each other unless otherwise specified. For example, assume that element A is connected to one node of a capacitive element C having multiple nodes, and element B is connected to the other node. Even in such a case, element A and element B are treated as being connected to each other unless otherwise specified.

[0014] The embodiments shown below are intended to embody the technical concept of the present invention and are not intended to limit the present invention. The size and positional relationship of components shown in each drawing may be exaggerated for clarity. In the following description, the same components may be designated by the same reference numerals and their description may be omitted.

[0015] [First embodiment] The schematic configuration of a photoelectric conversion device according to a first embodiment of the present invention will be described with reference to Figs. 1 to 4. Figs. 1 and 2 are block diagrams showing the schematic configuration of a photoelectric conversion device according to this embodiment. Fig. 3 is a block diagram showing an example configuration of a pixel of a photoelectric conversion device according to this embodiment. Fig. 4 is a perspective view showing an example configuration of a photoelectric conversion device according to this embodiment.

[0016] As shown in FIG. 1, the photoelectric conversion device 100 according to this embodiment includes a pixel section 10, a vertical scanning circuit section 40, a readout circuit section 50, a horizontal scanning circuit section 60, an output circuit section 70, and a control pulse generation section 80.

[0017] The pixel unit 10 includes a plurality of pixels 12 arranged in an array of a plurality of rows and a plurality of columns. As described below, each pixel 12 may be composed of a photoelectric conversion unit including a photoelectric conversion element and a signal processing circuit unit that processes a signal output from the photoelectric conversion unit. The number of pixels 12 constituting the pixel unit 10 is not particularly limited. For example, the pixel unit 10 may be composed of a plurality of pixels 12 arranged in an array of several thousand rows and several thousand columns, as in a general digital camera. Alternatively, the pixel unit 10 may be composed of a plurality of pixels 12 arranged in a single row or a single column. Alternatively, the pixel unit 10 may be composed of a single pixel 12.

[0018] A control line 14 is arranged in each row of the pixel array of the pixel unit 10, extending in a first direction (the horizontal direction in FIG. 1 ). The control line 14 is connected to each of the pixels 12 aligned in the first direction and serves as a signal line common to these pixels 12. The first direction in which the control lines 14 extend is sometimes referred to as the row direction or horizontal direction. Each of the control lines 14 may include multiple signal lines for supplying multiple types of control signals to the pixels 12. The control lines 14 in each row are connected to a vertical scanning circuit unit 40.

[0019] In addition, output lines 16 are arranged in each column of the pixel array of the pixel unit 10, extending in a second direction (vertical direction in FIG. 1 ) intersecting the first direction. The output lines 16 are connected to the pixels 12 aligned in the second direction, and serve as signal lines common to these pixels 12. The second direction in which the output lines 16 extend is sometimes referred to as the column direction or vertical direction. Each of the output lines 16 may include multiple signal lines, such as signal lines for transferring multi-bit digital signals output from the pixels 12 bit by bit.

[0020] The control lines 14 of each row are connected to a vertical scanning circuit unit 40. The vertical scanning circuit unit 40 is a control unit that receives control signals output from the control pulse generation unit 80, generates control signals for driving the pixels 12, and supplies the control signals to the pixels 12 via the control lines 14. The vertical scanning circuit unit 40 may include logic circuits such as a shift register and an address decoder. The vertical scanning circuit unit 40 scans the pixels 12 in the pixel unit 10 row by row, and outputs pixel signals of each pixel 12 to a readout circuit unit 50 via an output line 16.

[0021] The output line 16 of each column is connected to the readout circuit unit 50. The readout circuit unit 50 includes a plurality of holding units (not shown) provided corresponding to each column of the pixel array of the pixel unit 10, and has a function of holding, in the holding unit of the corresponding column, pixel signals of the pixels 12 of each column that are output row by row from the pixel unit 10 via the output line 16.

[0022] The horizontal scanning circuit unit 60 is a control unit that receives a control signal output from the control pulse generation unit 80, generates a control signal for reading out pixel signals from the holding units of each column of the readout circuit unit 50, and supplies the control signal to the readout circuit unit 50. The horizontal scanning circuit unit 60 may use logic circuits such as a shift register and an address decoder. The horizontal scanning circuit unit 60 scans the holding units of the readout circuit unit 50 on a column-by-column basis, and sequentially outputs the pixel signals held in the holding units of each column to the output circuit unit 70.

[0023] The output circuit unit 70 has an external interface circuit and is a circuit unit for outputting the pixel signals output from the readout circuit unit 50 to the outside of the photoelectric conversion device 100. The external interface circuit included in the output circuit unit 70 is not particularly limited. For example, a SerDes (SERializer / DESerializer) transmission circuit can be applied to the external interface circuit. Examples of the SerDes transmission circuit include an LVDS (Low Voltage Differential Signaling) circuit and an SLVS (Scalable Low Voltage Signaling) circuit.

[0024] The control pulse generation unit 80 is a control circuit that generates control signals for controlling the operation and timing of the vertical scanning circuit unit 40, the readout circuit unit 50, and the horizontal scanning circuit unit 60, and supplies these to each functional block. Note that at least some of the control signals for controlling the operation and timing of the vertical scanning circuit unit 40, the readout circuit unit 50, and the horizontal scanning circuit unit 60 may be supplied from outside the photoelectric conversion device 100.

[0025] The connection of the functional blocks of the photoelectric conversion device 100 is not limited to the example configuration shown in FIG. 1, and may be configured as shown in FIG. 2, for example.

[0026] In the configuration example of FIG. 2, an output line 16 extending in a first direction is arranged in each row of the pixel array of the pixel unit 10. The output line 16 is connected to each of the pixels 12 aligned in the first direction and serves as a signal line common to these pixels 12. Furthermore, a control line 18 extending in a second direction is arranged in each column of the pixel array of the pixel unit 10. The control line 18 is connected to each of the pixels 12 aligned in the second direction and serves as a signal line common to these pixels 12.

[0027] The control line 18 of each column is connected to a horizontal scanning circuit unit 60. The horizontal scanning circuit unit 60 receives a control signal output from the control pulse generation unit 80, generates a control signal for reading out pixel signals from the pixels 12, and supplies the control signal to the pixels 12 via the control line 18. Specifically, the horizontal scanning circuit unit 60 scans the multiple pixels 12 of the pixel unit 10 on a column-by-column basis, and outputs pixel signals of the pixels 12 in each row belonging to the selected column to an output line 16.

[0028] The output line 16 of each row is connected to the readout circuit unit 50. The readout circuit unit 50 includes a plurality of holding units (not shown) provided corresponding to each row of the pixel array of the pixel unit 10, and has a function of holding, in the holding unit of the corresponding row, pixel signals of the pixels 12 of each row that are output column by column from the pixel unit 10 via the output line 16.

[0029] The readout circuit unit 50 receives a control signal output from the control pulse generation unit 80, and sequentially outputs the pixel signals held in the holding units of each row to the output circuit unit 70. Other configurations in the configuration example of FIG. 2 may be similar to those in the configuration example of FIG.

[0030] As shown in FIG. 3 , each pixel 12 has a photoelectric conversion unit 20 and a signal processing circuit unit 30. The photoelectric conversion unit 20 is a functional block that converts incident light into an electrical signal and is configured to include a photoelectric conversion element. The signal processing circuit unit 30 is a functional block that performs predetermined signal processing on the signal output from the photoelectric conversion unit 20 and may be configured, for example, with a quench circuit 32, a waveform shaping circuit 34, a counter circuit 36, and a selection circuit 38. Note that the signal processing circuit unit 30 is not particularly limited, and may include any one of the quench circuit 32, the waveform shaping circuit 34, the counter circuit 36, and the selection circuit 38.

[0031] The photoelectric conversion unit 20 may be configured to include an avalanche photodiode (hereinafter referred to as "APD") 22. The anode of the APD 22 is connected to a node to which a voltage VL is supplied. The cathode of the APD 22 is connected to one terminal of a quench circuit 32. The connection node between the APD 22 and the quench circuit 32 is the output node of the photoelectric conversion unit 20. The other terminal of the quench circuit 32 is connected to a node to which a voltage VH higher than the voltage VL is supplied. The voltages VL and VH are set so that a reverse bias voltage sufficient for the APD 22 to perform avalanche multiplication operation is applied. For example, a negative high voltage is applied as the voltage VL, and a positive voltage approximately equal to the power supply voltage is applied as the voltage VH. For example, the voltage VL is −30 V, and the voltage VH is 3 V.

[0032] The photoelectric conversion unit 20 may include an APD 22, as described above. When a reverse bias voltage sufficient for avalanche multiplication is supplied to the APD 22, the charge generated by incident light undergoes avalanche multiplication, generating an avalanche current. When a reverse bias voltage is supplied to the APD, the APD can operate in either a Geiger mode or a linear mode. The Geiger mode is an operating mode in which a reverse bias voltage greater than the breakdown voltage of the APD is applied between the anode and cathode. The linear mode is an operating mode in which a reverse bias voltage applied between the anode and cathode is close to or less than the breakdown voltage of the APD. An APD operating in the Geiger mode is called a single photon avalanche diode (SPAD). The APD 22 constituting the photoelectric conversion unit 20 may operate in either the linear mode or the Geiger mode. The operation in the Geiger mode will be described below.

[0033] In this embodiment, the signal is extracted from the cathode side of the APD 22. Therefore, the first conductivity type semiconductor region having majority carriers of charges of the same polarity as the signal charges is an N-type semiconductor region, and the second conductivity type semiconductor region having majority carriers of charges of a polarity opposite to that of the signal charges is a P-type semiconductor region. The first conductivity type carriers are electrons, and the second conductivity type carriers are holes. The present invention also applies when the signal is extracted from the anode side of the APD 22. In this case, the first conductivity type semiconductor region having majority carriers of charges of the same polarity as the signal charges is a P-type semiconductor region, and the second conductivity type semiconductor region having majority carriers of charges of a polarity opposite to that of the signal charges is an N-type semiconductor region. The following description focuses on the case where one node of the APD is at a fixed potential. However, the potentials of both nodes may fluctuate as long as the potential difference between the anode and cathode of the APD 22 satisfies the relationship described below.

[0034] The quench circuit 32 has a function of converting a change in the avalanche current generated in the APD 22 into a voltage signal. The quench circuit 32 also functions as a load circuit during signal multiplication by avalanche multiplication, and has a function of reducing the voltage applied to the APD 22 to suppress avalanche multiplication. The operation of the quench circuit 32 to suppress avalanche multiplication is called a quench operation. The quench circuit 32 also has a function of returning the voltage supplied to the APD 22 to voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation. The operation of the quench circuit 32 to return the voltage supplied to the APD 22 to voltage VH is called a recharge operation. The quench circuit 32 can be configured to include a resistor element, a MOS transistor, etc.

[0035] The waveform shaping circuit 34 has an input node to which the output signal of the photoelectric conversion unit 20 is input, and an output node. The waveform shaping circuit 34 has a function of converting the analog signal output from the photoelectric conversion unit 20 into a pulse signal. The waveform shaping circuit 34 can be configured by a logic circuit including, for example, a NOT circuit (inverter circuit), a NOR circuit, a NAND circuit, etc. Note that while FIG. 3 shows an example in which one inverter circuit is used as the waveform shaping circuit 34, a circuit in which multiple inverter circuits are connected in series may also be used. The output node of the waveform shaping circuit 34 is connected to a counter circuit 36.

[0036] The counter circuit 36 ​​has an input node to which the output signal of the waveform shaping circuit 34 is input, an input node connected to the control line 14, and an output node. The counter circuit 36 ​​counts pulses superimposed on the signal output from the waveform shaping circuit 34 and holds a count value that is the counting result. Signals supplied to the counter circuit 36 ​​from the vertical scanning circuit unit 40 via the control line 14 may include, for example, an enable signal for controlling the pulse count period and a reset signal for resetting the count value held by the counter circuit 36. The output node of the counter circuit 36 ​​is connected to the output line 16 via a selection circuit 38.

[0037] The selection circuit 38 has a function of switching the electrical connection state (connection or non-connection) between the counter circuit 36 ​​and the output line 16. The selection circuit 38 switches the connection state between the counter circuit 36 ​​and the output line 16 in response to a control signal supplied from the vertical scanning circuit unit 40 via the control line 14 (in the configuration example of FIG. 2, a control signal supplied from the horizontal scanning circuit unit 60 via the control line 18). The selection circuit 38 may further include a buffer circuit for outputting a signal.

[0038] It should be noted that it is not necessary to provide one signal processing circuit unit 30 for each pixel 12, and one signal processing circuit unit 30 may be provided for a plurality of pixels 12. In this case, the single signal processing circuit unit 30 can be used to sequentially perform signal processing for a plurality of pixels 12.

[0039] The photoelectric conversion device 100 of this embodiment may be formed on a single substrate, or may be configured as a stacked photoelectric conversion device in which multiple substrates are stacked. In the latter case, for example, as shown in FIG. 4, a stacked photoelectric conversion device can be configured in which a sensor substrate 110 and a circuit substrate 180 are stacked and electrically connected. At least the photoelectric conversion unit 20, one of the components of the pixel 12, can be arranged on the sensor substrate 110. Furthermore, one of the components of the pixel 12, the signal processing circuit unit 30, can be arranged on the circuit substrate 180. The photoelectric conversion unit 20 and the signal processing circuit unit 30 are electrically connected via connection wiring provided for each pixel 12. Furthermore, the circuit substrate 180 can further include a vertical scanning circuit unit 40, a readout circuit unit 50, a horizontal scanning circuit unit 60, an output circuit unit 70, a control pulse generation unit 80, and the like.

[0040] The photoelectric conversion unit 20 and signal processing circuit unit 30 of each pixel 12 may be provided on the sensor substrate 110 and the circuit substrate 180 so as to overlap in a plan view. The vertical scanning circuit unit 40, readout circuit unit 50, horizontal scanning circuit unit 60, output circuit unit 70, and control pulse generation unit 80 may be arranged around the pixel unit 10 made up of a plurality of pixels 12. Note that "plan view" here refers to a view from a direction perpendicular to the surface of the sensor substrate 110.

[0041] By configuring a stacked photoelectric conversion device 100, it is possible to increase the integration density of elements and achieve higher functionality. In particular, by arranging the photoelectric conversion unit 20 and the signal processing circuit unit 30 on separate substrates, it is possible to arrange the photoelectric conversion elements at a high density without sacrificing the light-receiving area of ​​the photoelectric conversion elements that make up the photoelectric conversion unit 20, thereby improving photon detection efficiency.

[0042] The number of substrates constituting the photoelectric conversion device 100 is not limited to two, and the photoelectric conversion device 100 may be constituted by stacking three or more substrates.

[0043] 4, the sensor substrate 110 and the circuit substrate 180 are assumed to be diced chips, but the sensor substrate 110 and the circuit substrate 180 are not limited to chips. For example, the sensor substrate 110 and the circuit substrate 180 may each be a wafer. The sensor substrate 110 and the circuit substrate 180 may be stacked in the wafer state and then diced, or may be formed into chips and then stacked and bonded.

[0044] Next, the basic operations of the APD 22, quench circuit 32, and waveform shaping circuit 34 in the photoelectric conversion device according to this embodiment will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating the basic operation of a pixel in the photoelectric conversion device according to this embodiment. FIG. 5(a) is a circuit diagram showing the APD 22, quench circuit 32, and waveform shaping circuit 34 of the pixel 12. FIG. 5(b) shows the waveform of a signal at the output node (node ​​A) of the photoelectric conversion unit 20, and FIG. 5(c) shows the waveform of a signal at the output node (node ​​B) of the waveform shaping circuit 34. For simplicity of explanation, it is assumed here that the quench circuit 32 is configured with a resistive element, and the waveform shaping circuit 34 is configured with an inverter circuit.

[0045] At time t0, a reverse bias voltage with a potential difference equivalent to (VH - VL) is applied to the APD 22. A reverse bias voltage sufficient to cause avalanche multiplication is applied between the anode and cathode of the APD 22, but in a state where no photons are incident on the APD 22, there are no carriers to serve as seeds for avalanche multiplication. Therefore, avalanche multiplication does not occur in the APD 22, and no current flows through the APD 22.

[0046] At a subsequent time t1, a photon is incident on the APD 22. When the photon is incident on the APD 22, electron-hole pairs are generated by photoelectric conversion, and avalanche multiplication occurs using these carriers as seeds, causing an avalanche current to flow through the APD 22. This avalanche current flows through the quench circuit 32, causing a voltage drop across the quench circuit 32, and the voltage at node A begins to drop. The amount of voltage drop at node A increases, and when the avalanche multiplication stops at time t3, the voltage level at node A no longer drops.

[0047] When the avalanche multiplication in the APD 22 stops, a current that compensates for the voltage drop flows from the node to which the voltage VH is supplied to the node A via the quench circuit 32, and the voltage at the node A gradually increases. After that, at time t5, the node A settles to its original voltage level.

[0048] The waveform shaping circuit 34 binarizes the signal input from node A according to a predetermined decision threshold (the logic threshold voltage of the inverter circuit) and outputs the binarized signal from node B. Specifically, when the voltage level of node A exceeds the decision threshold, the waveform shaping circuit 34 outputs a low-level signal from node B, and when the voltage level of node A is equal to or lower than the decision threshold, the waveform shaping circuit 34 outputs a high-level signal from node B. For example, as shown in FIG. 5(b), assume that the voltage of node A is equal to or lower than the decision threshold during the period from time t2 to time t4. In this case, as shown in FIG. 5(c), the signal level at node B is low during the period from time t0 to time t2 and the period from time t4 to time t5, and is high during the period from time t2 to time t4.

[0049] In this way, the analog signal input from node A is waveform-shaped into a digital signal by the waveform shaping circuit 34. The pulse signal output from the waveform shaping circuit 34 in response to the incidence of a photon on the APD 22 is a photon detection signal.

[0050] Next, the structure of the pixel 12 of the photoelectric conversion device according to this embodiment will be described in more detail with reference to Fig. 6. Fig. 6 is a block diagram showing an example of the configuration of a pixel of the photoelectric conversion device according to this embodiment.

[0051] 6, the pixel 12 of the photoelectric conversion device according to this embodiment includes an APD 22, a P-type MOS transistor Mq and an OR circuit LG1 that constitute a quench circuit 32, and an AND circuit LG2 that constitutes a waveform shaping circuit 34. The pixel 12 also includes a photon detection counter 362 (first counter), an exposure control circuit 364, a time information counter 366 (second counter), a memory 368, and a comparison circuit 370 that constitute a counter circuit 36, as well as a selection circuit 38 (not shown) (see FIG. 3).

[0052] The anode of the APD22 is connected to a node to which a voltage VL is supplied. The cathode of the APD22 is connected to the drain of a P-type MOS transistor Mq, which functions as a quenching element. The source of the P-type MOS transistor Mq is connected to a node to which a voltage VH is supplied. The gate of the P-type MOS transistor Mq is connected to an output node of an OR circuit LG1. A clock signal CLKB (an inverted signal of the clock signal CLK) from the control pulse generating unit 80 and a control signal STOP, which is an output signal of the exposure control circuit 364, are input to two input nodes of the OR circuit LG1. An inverted signal of the cathode voltage VC of the APD22 and the clock signal CLKB from the control pulse generating unit 80 are input to two input nodes of the AND circuit LG2. The output node of the AND circuit LG2 is connected to the photon detection counter 362.

[0053] The photon detection counter 362 is connected to an exposure control circuit 364. The exposure control circuit 364 is connected to a time information counter 366. A determination control signal PDC and a counter threshold CTH from the control pulse generation unit 80 are also input to the exposure control circuit 364. The counter threshold CTH may be stored in advance in the exposure control circuit 364. The time information counter 366 is connected to a memory 368 and a comparison circuit 370. The comparison circuit 370 is connected to the memory 368.

[0054] When light is incident on the APD 22, photoelectric conversion generates electric charges (electron-hole pairs). When a reverse bias voltage equal to or greater than the breakdown voltage is applied to the APD 22, the generated electric charges undergo avalanche multiplication as they pass through a high-electric field region within the device, generating an avalanche current. The cathode of the APD 22 is connected to the voltage VH node via a P-type MOS transistor Mq, and the recharge and quench operations of the APD 22 can be controlled by the P-type MOS transistor Mq.

[0055] The P-type MOS transistor Mq is controlled by the output signal of the OR circuit LG1. That is, when the clock signal CLKB and the control signal STOP are low, the P-type MOS transistor Mq is turned on, and the APD 22 is in a recharge state. As a result, the APD 22 enters a standby state where avalanche multiplication is possible after a predetermined period of time. Furthermore, when at least one of the clock signal CLKB and the control signal STOP is high, the P-type MOS transistor Mq is turned off. When the P-type MOS transistor Mq is turned off in response to a high-level clock signal CLKB, the P-type MOS transistor Mq is turned off after the APD 22 is recharged, and the APD 22 is in a standby state where avalanche multiplication is possible. When the P-type MOS transistor Mq is turned off in response to a high-level control signal STOP, the APD 22 is no longer recharged, and no photon detection signal is output from the waveform shaping circuit 34. That is, the counting operation of the photon detection counter 362 stops.

[0056] An inverted signal of the cathode voltage VC of the APD 22 and the clock signal CLKB are input to the AND circuit LG2 that constitutes the waveform shaping circuit 34. When avalanche multiplication occurs in the APD 22 while the clock signal CLKB is at a high level and the cathode voltage VC falls below the logical threshold voltage of the AND circuit LG2, the AND circuit LG2 outputs a high-level signal (photon detection signal) that indicates the incidence of a photon.

[0057] The photon detection counter 362 counts the photon detection signal output from the AND circuit LG2 and stores it as a photon count value. The counting period of the photon detection signal and resetting of the photon count value in the photon detection counter 362 can be controlled by the exposure control circuit 364. Note that the photon detection signal may be counted by detecting the rising edge of the photon detection signal or by detecting the falling edge of the photon detection signal.

[0058] The exposure control circuit 364 performs a comparison operation to compare the photon count value of the photon detection counter 362 with a counter threshold value CTH (predetermined threshold value) in response to a determination control signal PDC supplied at a predetermined timing from the control pulse generation unit 80 during the count period. Furthermore, the exposure control circuit 364 controls the P-type MOS transistor Mq and the time information counter 366 in response to the result of the comparison operation. The determination control signal PDC is input to the exposure control circuit 364 multiple times (N times) at a predetermined timing during the exposure period of one frame.

[0059] The time information counter 366 performs a predetermined operation in accordance with the result of the comparison operation in the exposure control circuit 364. Specifically, when the photon count value of the photon detection counter 362 is equal to or less than the counter threshold CTH, the time information counter 366 increments the time count value by 1 in accordance with a control signal from the exposure control circuit 364. Furthermore, when the photon count value of the photon detection counter 362 exceeds the counter threshold CTH, the time information counter 366 latches the time count value at that time in accordance with a control signal from the exposure control circuit 364. Resetting of the time count value in the time information counter 366 can be controlled by the exposure control circuit 364.

[0060] The comparison circuit 370 compares the time count value of the time information counter 366 with the time count value of the previous frame stored in the memory 368, and outputs information according to the comparison result. Specifically, the comparison circuit 370 outputs a signal (event information) indicating that an event has been detected when the time count value of the time information counter 366 differs from the count value held in the memory 368. Note that the comparison circuit 370 may output event information indicating that an event has been detected when the difference between the time count value of the time information counter 366 and the count value held in the memory 368 exceeds a predetermined value.

[0061] Next, a method for driving the pixel 12 of the photoelectric conversion device according to this embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the method for driving the pixel of the photoelectric conversion device according to this embodiment. The flowchart in FIG. 7 assumes that the exposure control circuit 364 compares the photon count value with the counter threshold CTH N times (N is an integer equal to or greater than 1) during the exposure period of one frame. In the following description, it is assumed that the determination control signal PDC is input three times (N=3) at times T1, T2, and T3 during the exposure period of one frame. Note that in this specification, the exposure period refers to a period during which the photon detection counter 362 can be set to a count period during which the photon detection signal can be counted, and is assumed to have a common length (exposure time T) for each frame. The number of times the photon count value is compared with the counter threshold CTH during the exposure period does not necessarily have to be three, as long as it is at least once.

[0062] The counter threshold CTH is set according to the input timing of the determination control signal PDC so that the count value of the photon detection counter 362 does not exceed the count saturation value Nsat when the exposure time T is reached. For example, when the comparison operation between the photon count value and the counter threshold CTH is performed N times, the elapsed time from the start of exposure at times T1, T2, ..., TN is, for example, T / m N ,T / m N-1 ,…,T / m 1In this case, the counter threshold CTH can be set to Nsat / m, where Nsat is the count saturation value of the photon detection counter 362. Here, m can be an integer equal to or greater than 2, but is preferably a power of 2. By setting m to a power of 2, when comparing the photon count value with the counter threshold CTH, the function can be realized with a small circuit that simply outputs the m-th bit of the photon count value to the exposure control circuit 364.

[0063] For example, if the photon count value is compared with the counter threshold CTH three times, m is set to 8 (=2 3 ), the elapsed time from the start of exposure at times T1, T2, and T3 is T / 8, respectively. 3 ,T / 8 2 ,T / 8. The counter threshold CTH can be set to Nsat / 8. For example, if the photon detection counter 362 is an 11-bit counter and the count saturation value (Nsat) is 2048 LSB, the counter threshold CTH is 256 LSB (=Nsat / 8).

[0064] When a frame period starts, first, in step S101, the exposure control circuit 364 resets the photon count value of the photon detection counter 362 and the time count value of the time information counter 366 to their initial values ​​(0).

[0065] Next, in step S102, the exposure control circuit 364 controls the control signal STOP to low level, thereby starting the exposure period. When the clock signal CLKB is low level, the P-type MOS transistor Mq is turned on, and the APD 22 is in a recharge state. As a result, the APD 22 enters a standby state in which avalanche multiplication is possible after a predetermined period. When the clock signal CLKB transitions to high level, the P-type MOS transistor Mq is turned off, and the APD 22 is released from the recharge state. If a photon is incident on the APD 22 at this time, avalanche multiplication occurs, causing the cathode voltage VC to drop. Then, the output of the AND circuit LG2 transitions from low level to high level, and the count value of the photon detection counter 362 increases by one. When the clock signal CLKB transitions to low level, the P-type MOS transistor Mq is turned off, and the APD 22 returns to the recharge state. During the exposure period, the above-described operations are repeated in response to the periodic input of the clock signal CLKB. That is, the photon detection counter 362 counts the pulse signals outputted in correspondence with the period during which avalanche multiplication occurs, within the period during which the APD 22 is in the standby state.

[0066] Next, in step S103, the exposure control circuit 364 determines whether or not the exposure time T has elapsed. If the result of the determination is that the exposure time T has elapsed, the process proceeds to step S114 ("YES" in step S103), and if the exposure time T has not elapsed, the process proceeds to step S104 ("NO" in step S103). Here, it is assumed that the time that has elapsed since the start of the exposure period is before time T1, and the process proceeds to step S104.

[0067] Next, in step S104, the exposure control circuit 364 determines whether or not it has received the determination control signal PDC. If the result of the determination is that it has received the determination control signal PDC, the process proceeds to step S105 ("YES" in step S104), and if it has not received the determination control signal PDC, the process returns to step S103 ("NO" in step S104). Here, it is assumed that the determination control signal PDC is received when time T1 has elapsed, and the process proceeds to step S105.

[0068] Next, in step S105, the exposure control circuit 364 determines whether the photon count value of the photon detection counter 362 exceeds the counter threshold value CTH. If the result of the determination is that the photon count value is equal to or less than the counter threshold value CTH, the process proceeds to step S106 ("YES" in step S105), and if the photon count value exceeds the counter threshold value CTH, the process proceeds to step S108 ("NO" in step S105).

[0069] If it is determined in step S105 that the photon count value is equal to or less than the counter threshold value CTH, the exposure control circuit 364 counts up (increases by 1) the time count value of the time information counter 366 in step S106. For example, if the time information counter 366 has a 2-bit configuration, the time count value changes from "00" to "01." After step S106, the process proceeds to step S107.

[0070] In step S107, it is determined whether the time count value has reached the maximum number (N) of times that the determination control signal PDC is input during the exposure period of one frame. If the result of the determination is that the time count value has reached N, the process proceeds to step S112 ("YES" in step S107), and if the time count value is less than N, the process returns to step S103 (NO in step S107).

[0071] 7, a time count value is used to identify the last determination control signal PDC input during the exposure period of one frame (exposure time T), but the method of identifying the last determination control signal PDC is not limited to this. For example, it is also possible to provide a counter that counts the time elapsed from the start of the exposure period, and identify the determination control signal PDC input when the count value of the counter reaches a predetermined value or after it exceeds the predetermined value as the last determination control signal PDC.

[0072] If the photon count value exceeds the counter threshold CTH in step S105, the exposure control circuit 364 changes the control signal STOP from low to high in step S108. As a result, the P-type MOS transistor Mq is fixed to the off state regardless of the clock signal CLKB, and the recharge operation of the APD 22 is stopped, i.e., the exposure is stopped. After step S108, the process proceeds to step S109.

[0073] In step S109, the exposure control circuit 364 latches the time count value at that time in the time information counter 366. Note that if the photon count value exceeds the counter threshold CTH, the time count value is not counted up, and the time count value at that time ("00") is held in the time information counter 366. After step S109, the process proceeds to step S110.

[0074] In step S110, the comparator circuit 370 compares the count value of the previous frame stored in the memory 368 with the time count value (“00”) stored in the time information counter 366. If the comparison result shows that the time count value stored in the time information counter 366 is different from the count value stored in the memory 368 (“YES” in step S110), it is determined that an event has been detected, and the process proceeds to step S111. In step S111, the comparator circuit 370 outputs information indicating that an event has been detected (event information) to the output line 16 via the selection circuit 38. In this case, the event information can be output after time T1, which is before the exposure time T has elapsed. After step S111, the process proceeds to step S114. If the comparison result shows that the time count value stored in the time information counter 366 is the same as the count value stored in the memory 368 (“NO” in step S110), it is determined that an event has not been detected, and the process proceeds to step S114.

[0075] If it is determined in step S107 that the time count value is less than N, the process returns to step S103 and the same processing as described above is repeated. In steps S103 and S104, the photon detection counting operation continues until the second determination control signal PDC is received at time T2.

[0076] When the second determination control signal PDC is received after time T2, the process proceeds to step S105. The exposure control circuit 364 determines whether the photon count value of the photon detection counter 107 exceeds the counter threshold value CTH. If the determination result shows that the photon count value is equal to or less than the counter threshold value CTH, the process proceeds to step S106 ("YES" in step S105), and if the photon count value exceeds the counter threshold value CTH, the process proceeds to step S108 ("NO" in step S105).

[0077] If it is determined in step S105 that the photon count value is equal to or less than the counter threshold value CTH, the exposure control circuit 364 increments the time count value of the time information counter 366 by 1 in step S106. For example, if the time information counter 366 has a 2-bit configuration, the time count value changes from "01" to "10." After step S106, the process proceeds to step S107.

[0078] In step S107, it is determined whether the time count value has reached the maximum number (N) of determination control signals PDC input during the exposure period of one frame. If the result of the determination is that the time count value has reached N, the process proceeds to step S112 ("YES" in step S107), and if the time count value is less than N, the process returns to step S103 (NO in step S107).

[0079] If the photon count value exceeds the counter threshold CTH in step S105, the exposure control circuit 364 changes the control signal STOP from low to high in step S108. As a result, the P-type MOS transistor Mq is fixed to the off state regardless of the clock signal CLKB, and the recharge operation of the APD 22 is stopped, i.e., the exposure is stopped. After step S108, the process proceeds to step S109.

[0080] In step S109, the exposure control circuit 364 latches the time information counter 366 with the time count value at that time. Note that if the photon count value exceeds the counter threshold CTH, the time count value is not counted up, and the time count value ("01") at that time is held in the time information counter 366. After step S109, as described above, an event detection determination process is performed in step S110. If an event is detected in step S110, event information is output in step S111. In this case, the event information can be output after time T2, which is before the exposure time T has elapsed. After step S110 or step S111, the process proceeds to step S114.

[0081] If it is determined in step S107 that the time count value is less than N, the process returns to step S103 and the same processing as described above is repeated. In steps S103 and S104, the photon detection counting operation continues until the third determination control signal PDC is received at time T3.

[0082] After time T3 has elapsed and the third determination control signal PDC has been received, the process proceeds to step S105. The exposure control circuit 364 determines whether the photon count value of the photon detection counter 107 exceeds the counter threshold value CTH. If the determination result shows that the photon count value is equal to or less than the counter threshold value CTH, the process proceeds to step S106 ("YES" in step S105), and if the photon count value exceeds the counter threshold value CTH, the process proceeds to step S108 ("NO" in step S105).

[0083] If it is determined in step S105 that the photon count value is equal to or less than the counter threshold value CTH, the exposure control circuit 364 increments the time count value of the time information counter 366 by 1 in step S106. For example, if the time information counter 366 has a 2-bit configuration, the time count value changes from "10" to "11." After step S106, the process proceeds to step S107.

[0084] In step S107, it is determined whether the time count value has reached the maximum number (N) of times the determination control signal PDC is input during the exposure period of one frame. If the result of the determination is that the time count value has reached N, the process proceeds to step S112 ("YES" in step S107), and if the time count value is less than N, the process returns to step S103 (NO in step S107). Here, the number of times the determination control signal PDC is input during the exposure period of one frame is set to three (N=3), so the process proceeds to step S112.

[0085] In step S112, the same event detection determination process as in step S110 is performed. If an event is detected in step S112, the same event information output process as in step S111 is performed in step S113. In this case, the event information can be output after time T3, before the exposure time T has elapsed. After step S113, the process returns to step S103.

[0086] After step S112 or step S113, the processes of steps S103 and S104 are repeated to continue the photon detection and counting operation until the exposure time T has elapsed. After that, when the exposure time T has elapsed, the process proceeds to step S114.

[0087] In this way, if the photon count value exceeds the counter threshold CTH at time T1, the time count value "00" is held in the time information counter, and the process proceeds to step S114. If the photon count value exceeds the counter threshold CTH at time T2, the time count value "01" is held in the time information counter, and the process proceeds to step S114. If the photon count value exceeds the counter threshold CTH at time T3, the time count value "10" is held in the time information counter, and the process proceeds to step S114. If the exposure time T has elapsed without the photon count value exceeding the counter threshold CTH at time T3, the time count value "11" is held in the time information counter, and the process proceeds to step S114.

[0088] Next, in step S114, the vertical scanning circuit unit 40 drives the selection circuit 38 and outputs the photon count value held in the photon detection counter 362 and the time count value held in the time information counter 366 to the output line 16 via the selection circuit 38.

[0089] Next, in step S115, the exposure control circuit 364 stores the time count value held in the time information counter 366 in the memory 368. The time count value stored in the memory 368 is used in the event detection determination process in step S110 or step S112 of the next frame.

[0090] Next, in step S116, the control pulse generator 80 determines whether or not to continue image acquisition. If the result of the determination is that image acquisition should be continued ("YES" in step S116), the process returns to step S101 and executes processing for the next frame. If the result of the determination is that image acquisition should not be continued ("NO" in step S116), the imaging process ends.

[0091] In this embodiment, the photon count value is compared with the predetermined counter threshold CTH multiple times during the exposure period of one frame. This allows each pixel 12 to select an appropriate count period from multiple count periods of different lengths according to the photon count value, and image information can be acquired without increasing the number of bits in the photon detection counter 362.

[0092] Furthermore, the signal output from each pixel 12 includes count information (photon count value) and exposure time information (time count value). Using this information, the photon count value can be corrected according to the exposure time, thereby obtaining an HDR image. For example, if the time count value is "00," this indicates that the counting operation of the photon detection counter 362 was performed up to time T1. In this case, the photon count value obtained from the photon detection counter 362 is multiplied by (T / T1) to obtain the photon count value corresponding to the exposure time T.

[0093] In this embodiment, the exposure time information (time count value) for each pixel 12 is compared with the exposure time information for the immediately preceding frame, and if the exposure time information differs, it is output as event information. This makes it possible to output, for each pixel 12, event information when a sudden change in luminance occurs.

[0094] Next, an example of the operation of the pixel 12 of the photoelectric conversion device according to this embodiment will be described with reference to FIGS. 8 and 9. FIGS. 8 and 9 are timing diagrams showing an example of the operation of the pixel of the photoelectric conversion device according to this embodiment. FIGS. 8 and 9 show the operations of the time information counter 366, memory 368, and comparison circuit 370 in two consecutive frames (first and second frames). Each frame starts in response to a frame control signal, and an HDR image is acquired at the end of each frame. The photon count value is compared with the counter threshold CTH at times T1, T2, and T3 after the start of exposure in response to a determination control signal PDC.

[0095] FIG. 8 shows an example of operation in which the subject has low luminance in the first and second frames and there is no sudden change in luminance between the first and second frames. When the first frame starts in response to the frame control signal, the photon count value and the time count value are reset, and the exposure period for the first frame begins. When the subject has low luminance, the photon count value is equal to or less than the counter threshold CTH at each of times T1, T2, and T3, and the time count value becomes "11" after time T3. As a result, the count value "11" is stored in memory 368. When the second frame starts in response to the frame control signal, the photon count value and the time count value are reset, and the exposure period for the second frame begins. When the subject has low luminance, the photon count value is equal to or less than the counter threshold CTH at each of times T1, T2, and T3, and the time count value becomes "11" after time T3. Event detection is determined by comparing the information stored in memory 368 with the time count value immediately after time T3, when the time information for the second frame is determined. In this operation example, since there is no difference between the information stored in the memory 368 and the time count value, both being "11", it is determined that no event has been detected, and no event information is output.

[0096] FIG. 9 shows an example of operation when the subject has low brightness in the first frame and high brightness in the second frame, resulting in a rapid change in brightness between the first and second frames. When the first frame starts in response to the frame control signal, the photon count value and the time count value are reset, and the exposure period for the first frame begins. When the subject has low brightness, the photon count value is equal to or less than the counter threshold CTH at times T1, T2, and T3, and the time count value becomes "11" after time T3. This causes the count value "11" to be stored in memory 368. When the second frame starts in response to the frame control signal, the photon count value and the time count value are reset, and the exposure period for the second frame begins. When the subject has high brightness, for example, the photon count value exceeds the counter threshold CTH at time T1, and the control signal STOP is output, causing the photon detection counter 362 to stop counting at time T1. The time count value remains at "00" without being counted up. Event determination is performed by comparing the information stored in memory 368 with the time count value immediately after time T1, which is the time when the time information in the second frame is determined. In this operation example, since the information stored in memory 368 is "11" while the time count value is "00", it is determined that an event has been detected and the event information is output.

[0097] As described above, in this embodiment, the count value of the photon detection counter 362 is compared with the counter threshold CTH multiple times during the exposure period of one frame. Then, a time information counter 366 is provided that counts up each time the count value of the photon detection counter 362 is determined to be equal to or less than the counter threshold CTH. Then, event information is output according to the result of the comparison between the count value of the time information counter 366 in the first frame and the count value of the time information counter 366 in the second frame preceding the first frame.

[0098] Therefore, it is possible to select an appropriate count period according to the photon count value from among multiple count periods of different lengths, and image information can be acquired without increasing the number of bits in the photon detection counter 362. Furthermore, the time count value is compared with the time count value in the previous frame, and if these count values ​​differ, event information is output. Therefore, if a sudden change in brightness occurs, this can be output as event information. Furthermore, by comparing time count values, the number of bits constituting the memory 368 can be reduced compared to when photon count values ​​are compared, and the circuit configuration of the comparison circuit 370 can be simplified. This makes it possible to reduce the circuit area and power consumption.

[0099] As described above, according to this embodiment, it is possible to realize a photoelectric conversion device that can acquire a dynamic range image and detect an event while suppressing power consumption.

[0100] [Second embodiment] A photoelectric conversion device and a driving method thereof according to a second embodiment of the present invention will be described with reference to Fig. 10. Components similar to those in the photoelectric conversion device according to the first embodiment are given the same reference numerals, and descriptions thereof will be omitted or simplified. Fig. 10 is a block diagram showing an example of the configuration of a pixel in the photoelectric conversion device according to this embodiment.

[0101] The photoelectric conversion device according to this embodiment is similar to the photoelectric conversion device according to the first embodiment except for the configuration of the pixels 12. In this embodiment, the differences between the pixels 12 of the photoelectric conversion device according to this embodiment and the pixels 12 of the photoelectric conversion device according to the first embodiment will be mainly described, and descriptions of the same parts as in the first embodiment will be omitted as appropriate.

[0102] 10, in the pixel 12 of the photoelectric conversion device according to this embodiment, the photon detection counter 362 is connected to the exposure control circuit 364 as well as to a memory 368 and a comparison circuit 370. The memory 368 and the comparison circuit 370 are supplied with upper bit information of the photon count value from the photon detection counter 362. Other points are the same as those of the pixel 12 of the first embodiment.

[0103] When the exposure period for one frame ends, in step S115, the photon detection counter 362 stores the most significant bit information of the photon count value and the time count value in memory 368. In the event detection determination in step S110 or step S112 of the next frame, the most significant bit information of the time count value and photon count value of the time information counter 366 for that frame is compared with the count value and information stored in memory 368.

[0104] The comparator circuit 370 first compares the time count value held by the time information counter 366 with the count value stored in the memory 368. If the comparison results in a difference between these values, it determines that an event has been detected, and proceeds to step S111 if an event has been detected in step S110, or proceeds to step S113 if an event has been detected in step S112.

[0105] If the comparison result shows that the time count value held by the time information counter 366 is the same as the count value stored in the memory 368, the comparison circuit 370 compares the upper bit information of the photon count value with the information stored in the memory.

[0106] If the result of comparing the upper bit information of the photon count value with the information stored in memory is that the information is the same, it is determined that no event has been detected. If an event has been detected in step S110, the process proceeds to step S114, and if an event has been detected in step S112, the process proceeds to step S103.

[0107] As a result of comparing the upper bit information of the photon count value with the information stored in memory, if the upper bit information of the photon count value differs from the information stored in memory, it is determined that an event has been detected. If an event has been detected in step S110, the process proceeds to step S111, and if an event has been detected in step S112, the process proceeds to step S113.

[0108] In this way, by using the most significant bit information of the photon count value in addition to the time count value for event detection judgment, the sensitivity of event detection can be increased, and event detection judgment can be performed with higher accuracy. Furthermore, by comparing the most significant bit information of the photon count value, the number of bits of the memory 368 can be reduced and the circuit configuration of the comparison circuit 370 can be simplified compared to when all bits of the photon count value are compared. This makes it possible to reduce the circuit area and power consumption.

[0109] The most significant bit information of the photon detection counter 362 used for comparison is not particularly limited. For example, if the photon detection counter 362 is configured as an 11-bit counter, it can be configured to compare the most significant three bits of information from the 8th bit to the 10th bit. The number of significant bits used for comparison can be set appropriately depending on the accuracy required for event detection, etc.

[0110] As described above, according to this embodiment, it is possible to realize a photoelectric conversion device that can acquire a dynamic range image and detect an event while suppressing power consumption.

[0111] [Third embodiment] A photoelectric conversion device and a driving method thereof according to a third embodiment of the present invention will be described with reference to Fig. 11. Components similar to those of the photoelectric conversion device according to the first or second embodiment are given the same reference numerals, and descriptions thereof will be omitted or simplified. Fig. 11 is a block diagram showing an example of the configuration of a pixel of the photoelectric conversion device according to this embodiment.

[0112] The photoelectric conversion device according to this embodiment is similar to the photoelectric conversion device according to the first embodiment except for the configuration of the pixels 12. In this embodiment, the differences between the pixels 12 of the photoelectric conversion device according to this embodiment and the pixels 12 of the photoelectric conversion device according to the first embodiment will be mainly described, and descriptions of the same parts as in the first embodiment will be omitted as appropriate.

[0113] 11, the pixel 12 of the photoelectric conversion device according to this embodiment does not include an exposure control circuit 364, and is configured so that a signal output from a photon detection counter 362 is supplied to an OR circuit LG1 and a time information counter 366. Also, a clock signal CLK is supplied to the time information counter 366 from a control pulse generation unit 80 or an external device. In other respects, it is the same as the pixel 12 of the first embodiment. Note that the period of the clock signal CLK supplied to the time information counter 366 and the period of the clock signal CLKB supplied to the OR circuit LG1 may be the same or different.

[0114] In this embodiment, event detection is performed by comparing the time information at which the photon detection counter 362 reaches its upper count limit (=Nsat-1) between frames. When the photon detection counter 362 reaches its upper count limit during the exposure period, it transmits a signal indicating that the upper count limit has been reached (upper limit reach notification signal) to the time information counter 366. The upper limit reach notification signal also serves as the control signal STOP described above, and is a signal that transitions from low level to high level in response to the photon count value reaching the upper count limit. The time information counter 366 starts counting the clock signal CLK in synchronization with the start of the exposure period and stops counting the clock signal CLK in response to receiving the upper limit reach notification signal. The count value when counting is stopped in response to the upper limit reach notification signal becomes the time count value in this embodiment.

[0115] When the exposure period for a certain frame ends, similar to step S115 in Fig. 7, the time count value of the time information counter 366 is stored in the memory 368. Then, in the next frame, similar to step S110 or step S112 in Fig. 7, the time count value when the time information counter 366 receives the upper limit value reaching notification signal is compared with the count value stored in the memory 368 to determine whether an event has been detected.

[0116] The event detection determination can be based on whether the time count value when the upper limit reach notification signal is received is the same as the count value stored in memory 368. Alternatively, the event detection determination can be based on whether the difference between the time count value when the upper limit reach notification signal is received and the count value stored in memory 368 exceeds a predetermined event determination threshold. The criteria for event detection determination can be set appropriately depending on the accuracy required for event detection, etc.

[0117] As described above, according to this embodiment, it is possible to realize a photoelectric conversion device that can acquire a dynamic range image and detect an event while suppressing power consumption.

[0118] [Fourth embodiment] A photoelectric conversion device and a driving method thereof according to a fourth embodiment of the present invention will be described. The same components as those in the photoelectric conversion devices according to the first to third embodiments are given the same reference numerals, and their description will be omitted or simplified.

[0119] In the first to third embodiments, the explanations have been given assuming that each of the multiple pixels 12 constituting the pixel unit 10 has an event detection function, but it is not necessarily necessary to perform event detection in all of the pixels 12 constituting the pixel unit 10.

[0120] For example, the pixels 12 constituting the pixel unit 10 may be divided into a plurality of pixel blocks each including at least one pixel 12, and event detection may be performed only in some of these pixel blocks. In this case, the pixels 12 in the pixel blocks in which event detection is not performed may be configured to output only image information by, for example, turning off the power to the comparator circuit 370. Note that the method of outputting only image information from the pixels 12 is not limited to turning off the power to the comparator circuit 370, and any method may be applied.

[0121] The pixel blocks can be set arbitrarily. For example, the pixel unit 10 may be divided into a plurality of pixel blocks in a grid pattern, or the pixel unit 10 may be divided into a plurality of pixel blocks in rows or columns. The number of pixels that perform the event detection operation can also be set arbitrarily. The division manner of the pixel unit 10 and the pixel blocks that perform event detection may be configured to be changeable.

[0122] Furthermore, only some of the pixels 12 among the multiple pixels 12 constituting the pixel unit 10 may have an event detection function. In this case, the memory 368 and the comparison circuit 370 can be omitted for the pixels 12 that do not require the event detection function. Furthermore, the pixels that have the event detection function do not necessarily need to output image information, and may be configured to output only event information.

[0123] As described above, according to this embodiment, it is possible to realize a photoelectric conversion device that can acquire a dynamic range image and detect an event while suppressing power consumption.

[0124] [Fifth embodiment] A light detection system according to a fifth embodiment of the present invention will be described with reference to Fig. 12. Fig. 12 is a block diagram showing a schematic configuration of the light detection system according to this embodiment. In this embodiment, a light detection sensor to which the photoelectric conversion device 100 described in any of the first to fourth embodiments is applied will be described.

[0125] The photoelectric conversion device 100 described in the first to fourth embodiments can be applied to various photodetection systems. Examples of applicable photodetection systems include imaging systems such as digital still cameras, digital camcorders, surveillance cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, and observation satellites. Camera modules equipped with an optical system such as a lens and an imaging device are also included in the photodetection system. Fig. 12 illustrates a block diagram of a digital still camera as an example of such systems.

[0126] 12 includes a photoelectric conversion device 201, a lens 202 that forms an optical image of a subject on the photoelectric conversion device 201, an aperture 204 that adjusts the amount of light passing through the lens 202, and a barrier 206 that protects the lens 202. The lens 202 and the aperture 204 form an optical system that focuses light on the photoelectric conversion device 201. The photoelectric conversion device 201 is the photoelectric conversion device 100 described in the first embodiment, and converts the optical image formed by the lens 202 into image data.

[0127] The photodetection system 200 also includes a signal processing unit 208 that processes an output signal output from the photoelectric conversion device 201. The signal processing unit 208 generates image data from the digital signal output by the photoelectric conversion device 201. The signal processing unit 208 also performs various corrections and compressions as necessary to output the image data. The photoelectric conversion device 201 may include an AD conversion unit that generates a digital signal to be processed by the signal processing unit 208. The AD conversion unit may be formed in a semiconductor layer (semiconductor substrate) on which the photon detection elements of the photoelectric conversion device 201 are formed, or may be formed in a semiconductor layer different from the semiconductor layer on which the photon detection elements of the photoelectric conversion device 201 are formed. The signal processing unit 208 may also be formed in the same semiconductor layer as the photoelectric conversion device 201.

[0128] The light detection system 200 further includes a buffer memory unit 210 for temporarily storing image data, and an external interface unit (external I / F unit) 212 for communicating with an external computer or the like. The light detection system 200 also includes a recording medium 214 such as a semiconductor memory for recording or reading out imaging data, and a recording medium control interface unit (recording medium control I / F unit) 216 for recording or reading out imaging data from the recording medium 214. The recording medium 214 may be built into the light detection system 200 or may be removable. Communication between the recording medium control I / F unit 216 and the recording medium 214 and communication from the external I / F unit 212 may be performed wirelessly.

[0129] The photodetection system 200 further includes an overall control and calculation unit 218 that performs various calculations and controls the entire digital still camera, and a timing generation unit 220 that outputs various timing signals to the photoelectric conversion device 201 and the signal processing unit 208. Here, the timing signals and the like may be input from outside, and the photodetection system 200 only needs to include at least the photoelectric conversion device 201 and the signal processing unit 208 that processes the output signal output from the photoelectric conversion device 201. The timing generation unit 220 may be mounted on the photoelectric conversion device 201. The overall control and calculation unit 218 and the timing generation unit 220 may be configured to perform some or all of the control functions of the photoelectric conversion device 201.

[0130] The photoelectric conversion device 201 outputs an imaging signal to the signal processing unit 208. The signal processing unit 208 performs predetermined signal processing on the imaging signal output from the photoelectric conversion device 201 and outputs image data. The signal processing unit 208 generates an image using the imaging signal. The signal processing unit 208 may be configured to perform distance measurement calculations on the signal output from the photoelectric conversion device 201.

[0131] As described above, according to this embodiment, by configuring a light detection system using the photoelectric conversion devices of the first to fourth embodiments, it is possible to realize a light detection system capable of acquiring higher quality images.

[0132] [Sixth embodiment] A range image sensor according to a sixth embodiment of the present invention will be described with reference to Fig. 13. Fig. 13 is a block diagram showing a schematic configuration of the range image sensor according to this embodiment. In this embodiment, the range image sensor will be described as an example of a light detection system to which the photoelectric conversion device 100 described in any of the first to fourth embodiments is applied.

[0133] 13, the range image sensor 300 according to this embodiment may include an optical system 302, a photoelectric conversion device 304, an image processing circuit 306, a monitor 308, and a memory 310. This range image sensor 300 receives light (modulated light or pulsed light) that is irradiated from a light source device 320 toward a subject 330 and reflected by the surface of the subject 330, and obtains a range image according to the distance to the subject 330.

[0134] The optical system 302 is composed of one or more lenses, and serves to focus image light (incident light) from the subject 330 onto the light receiving surface (sensor section) of the photoelectric conversion device 304.

[0135] The photoelectric conversion device 304 is the photoelectric conversion device 100 described in any of the first to fourth embodiments, and has the function of generating a distance signal indicating the distance to the subject 330 based on image light from the subject 330, and supplying the generated distance signal to the image processing circuit 306.

[0136] The image processing circuit 306 has a function of performing image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 304 .

[0137] The monitor 308 has a function of displaying the distance image (image data) obtained by the image processing in the image processing circuit 306. The memory 310 has a function of storing (recording) the distance image (image data) obtained by the image processing in the image processing circuit 306.

[0138] Thus, according to this embodiment, by constructing a distance image sensor using the photoelectric conversion devices of the first to fourth embodiments, it is possible to realize a distance image sensor that can acquire distance images containing more accurate distance information, in combination with improved characteristics of pixel 12.

[0139] [Seventh embodiment] An endoscopic surgery system according to a seventh embodiment of the present invention will be described with reference to Fig. 14. Fig. 14 is a schematic diagram showing an example of the configuration of the endoscopic surgery system according to this embodiment. In this embodiment, the endoscopic surgery system will be described as an example of a light detection system to which the photoelectric conversion device 100 described in any of the first to fourth embodiments is applied.

[0140] FIG. 14 shows a state in which an operator (doctor) 460 is performing surgery on a patient 472 on a patient bed 470 using an endoscopic surgery system 400.

[0141] 14, an endoscopic surgery system 400 of this embodiment may include an endoscope 410, a surgical tool 420, and a cart 430 on which various devices for endoscopic surgery are mounted. The cart 430 may be mounted with a CCU (camera control unit) 432, a light source device 434, an input device 436, a treatment tool control device 438, a display device 440, and the like.

[0142] The endoscope 410 includes a lens barrel 412, a region of which a predetermined length from the tip is inserted into a body cavity of a patient 472, and a camera head 414 connected to the base end of the lens barrel 412. Although Fig. 14 illustrates the endoscope 410 configured as a so-called rigid lens barrel having a rigid lens barrel 412, the endoscope 410 may also be configured as a so-called flexible lens barrel having a flexible lens barrel. The endoscope 410 is held in a movable state by an arm 416.

[0143] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 412. A light source device 434 is connected to the endoscope 410, and light generated by the light source device 434 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 412, and is irradiated via the objective lens towards an observation target inside the body cavity of the patient 472. The endoscope 410 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0144] An optical system and a photoelectric conversion device (not shown) are provided inside the camera head 414, and light reflected from the observation object (observation light) is collected by the optical system onto the photoelectric conversion device. The photoelectric conversion device photoelectrically converts the observation light to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observation image. The photoelectric conversion device can be the photoelectric conversion device 100 described in any of the first to fourth embodiments. The image signal is transmitted to the CCU 432 as RAW data.

[0145] The CCU 432 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and comprehensively controls the operations of the endoscope 410 and the display device 440. Furthermore, the CCU 432 receives an image signal from the camera head 414, and performs various image processing on the image signal, such as development processing (demosaic processing), to display an image based on the image signal.

[0146] The display device 440 , under the control of the CCU 432 , displays an image based on the image signal that has been subjected to image processing by the CCU 432 .

[0147] The light source device 434 is configured from a light source such as an LED (Light Emitting Diode), and supplies the endoscope 410 with irradiation light when photographing an operation site or the like.

[0148] The input device 436 is an input interface for the endoscopic surgery system 400. A user can input various information and instructions to the endoscopic surgery system 400 via the input device 436.

[0149] The treatment tool control device 438 controls the driving of an energy treatment tool 450 for cauterizing tissue, incising, sealing blood vessels, or the like.

[0150] The light source device 434, which supplies illumination light to the endoscope 410 when photographing the surgical site, can be configured from a white light source configured from, for example, an LED, a laser light source, or a combination of these. When the white light source is configured from a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, making it possible to adjust the white balance of the captured image in the light source device 434. In this case, it is also possible to capture images corresponding to each RGB color in a time-division manner by irradiating the object of observation with laser light from each RGB laser light source in a time-division manner and controlling the drive of the image sensor of the camera head 414 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0151] Furthermore, the light source device 434 may be controlled to change the intensity of light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 414 in synchronization with the timing of the change in the light intensity to acquire images in a time-division manner and combining the images, it is possible to generate an image with a high dynamic range that is free of so-called blocked-up shadows and blown-out highlights.

[0152] The light source device 434 may also be configured to supply light in a predetermined wavelength band corresponding to special light observation. Special light observation utilizes, for example, the wavelength dependency of light absorption in body tissue. Specifically, specific tissue, such as blood vessels on the surface of the mucosa, can be photographed with high contrast by irradiating light with a narrower band than the light (i.e., white light) used in normal observation. Alternatively, special light observation may involve fluorescence observation, in which an image is obtained using fluorescence generated by irradiating excitation light. Fluorescence observation can involve irradiating excitation light onto body tissue and observing the fluorescence from the tissue, or locally injecting a reagent such as indocyanine green (ICG) into the body tissue and irradiating the tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 434 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0153] Thus, according to this embodiment, by configuring an endoscopic surgery system using the photoelectric conversion devices of the first to fourth embodiments, it is possible to realize an endoscopic surgery system that can acquire higher quality images.

[0154] [Eighth embodiment] An optical detection system and a moving body according to an eighth embodiment of the present invention will be described with reference to Figs. 15 to 17. Fig. 15 is a schematic diagram showing an example of the configuration of a moving body according to this embodiment. Fig. 16 is a block diagram showing a schematic configuration of an optical detection system according to this embodiment. Fig. 17 is a flow diagram showing the operation of the optical detection system according to this embodiment. In this embodiment, an example of application of an optical detection system to an in-vehicle camera using the photoelectric conversion device 100 described in any of the first to fourth embodiments is shown.

[0155] FIG. 15 is a schematic diagram showing an example of the configuration of a moving body (vehicle system) according to this embodiment. FIG. 15 shows the configuration of a vehicle 500 (automobile) as an example of a vehicle system incorporating a light detection system to which a photoelectric conversion device according to any one of the first to fourth embodiments is applied. FIG. 15(a) is a schematic front view of the vehicle 500, FIG. 15(b) is a schematic plan view of the vehicle 500, and FIG. 15(c) is a schematic rear view of the vehicle 500. The vehicle 500 is provided with a pair of photoelectric conversion devices 502 on the front side. Here, the photoelectric conversion devices 502 are the photoelectric conversion devices 100 described in the first embodiment. The vehicle 500 also includes an integrated circuit 503, an alarm device 512, and a main control unit 513.

[0156] FIG. 16 is a block diagram showing an example configuration of a photodetection system 501 mounted on a vehicle 500. The photodetection system 501 includes a photoelectric conversion device 502, an image preprocessing unit 515, an integrated circuit 503, and an optical system 514. The photoelectric conversion device 502 is the photoelectric conversion device 100 described in any one of the first to fourth embodiments. The optical system 514 forms an optical image of a subject on the photoelectric conversion device 502. The photoelectric conversion device 502 converts the optical image of the subject formed by the optical system 514 into an electrical signal. The image preprocessing unit 515 performs predetermined signal processing on the signal output from the photoelectric conversion device 502. The function of the image preprocessing unit 515 may be incorporated into the photoelectric conversion device 502. The photodetection system 501 includes at least two sets of the optical system 514, the photoelectric conversion device 502, and the image preprocessing unit 515, and the output from each set of the image preprocessing unit 515 is input to the integrated circuit 503.

[0157] The integrated circuit 503 is an integrated circuit for use in an imaging system, and includes an image processing unit 504, an optical distance measuring unit 506, a parallax calculation unit 507, an object recognition unit 508, and an abnormality detection unit 509. The image processing unit 504 processes an image signal output from an image pre-processing unit 515. For example, the image processing unit 504 performs image processing such as development processing and defect correction on the output signal of the image pre-processing unit 515. The image processing unit 504 includes a memory 505 that temporarily stores the image signal. For example, the positions of known defective pixels in the photoelectric conversion device 502 can be stored in the memory 505.

[0158] The optical distance measurement unit 506 performs focusing and distance measurement of the subject. The parallax calculation unit 507 calculates distance information (distance information) from multiple image data (parallax images) acquired by the multiple photoelectric conversion devices 502. Each of the photoelectric conversion devices 502 may be configured to be able to acquire various information such as distance information. The object recognition unit 508 recognizes subjects such as cars, roads, signs, and people. When the abnormality detection unit 509 detects an abnormality in the photoelectric conversion device 502, it notifies the main control unit 513 of the abnormality.

[0159] The integrated circuit 503 may be realized by dedicated hardware, a software module, or a combination thereof. It may also be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like, or a combination thereof.

[0160] The main control unit 513 supervises and controls the operations of the light detection system 501, the vehicle sensor 510, the control unit 520, etc. Note that the vehicle 500 does not necessarily have to include the main control unit 513. In this case, the photoelectric conversion device 502, the vehicle sensor 510, and the control unit 520 transmit and receive control signals via a communication network. For example, the CAN standard may be applied to the transmission and reception of these control signals.

[0161] The integrated circuit 503 has a function of receiving a control signal from the main control unit 513 or transmitting a control signal or a set value to the photoelectric conversion device 502 by its own control unit.

[0162] The optical detection system 501 is connected to a vehicle sensor 510 and can detect the vehicle's driving conditions, such as vehicle speed, yaw rate, and steering angle, as well as the conditions of the environment outside the vehicle and other vehicles and obstacles. The vehicle sensor 510 also serves as a distance information acquisition means for acquiring distance information to an object. The optical detection system 501 is also connected to a driving assistance control unit 511 that performs various driving assistance functions, such as automatic steering, automatic cruising, and collision prevention functions. In particular, the collision determination function determines whether or not a collision with another vehicle or obstacle has occurred based on the detection results of the optical detection system 501 and the vehicle sensor 510. This allows for avoidance control when a collision is estimated, and activation of safety devices in the event of a collision.

[0163] The optical detection system 501 is also connected to an alarm device 512 that issues an alarm to the driver based on the determination result of the collision determination unit. For example, if the collision determination unit determines that there is a high possibility of a collision, the main control unit 513 performs vehicle control to avoid the collision and mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 512 warns the user by sounding an alarm or the like, displaying alarm information on a display screen of a car navigation system or meter panel, vibrating the seat belt or steering wheel, etc.

[0164] In this embodiment, the surroundings of the vehicle, for example, the front or rear, are photographed by the light detection system 501. Fig. 15(b) shows an example of the arrangement of the light detection system 501 when the light detection system 501 photographs the area in front of the vehicle.

[0165] As described above, the photoelectric conversion device 502 is disposed in front of the vehicle 500. Specifically, if the center line of the vehicle 500's heading or outer shape (for example, vehicle width) is regarded as an axis of symmetry, and the two photoelectric conversion devices 502 are disposed symmetrically about the axis of symmetry, this is preferable for obtaining distance information between the vehicle 500 and an object to be photographed and determining the possibility of a collision. Furthermore, the photoelectric conversion device 502 is preferably disposed so as not to obstruct the driver's field of vision when the driver visually checks the situation outside the vehicle 500 from the driver's seat. The warning device 512 is preferably disposed so as to be easily within the driver's field of vision.

[0166] Next, a fault detection operation of the photoelectric conversion device 502 in the light detection system 501 will be described with reference to Fig. 17. The fault detection operation of the photoelectric conversion device 502 can be performed according to steps S510 to S580 shown in Fig. 17.

[0167] Step S510 is a step for performing startup settings for the photoelectric conversion device 502. That is, settings for the operation of the photoelectric conversion device 502 are transmitted from outside the photodetection system 501 (for example, from the main control unit 513) or from inside the photodetection system 501, and the image capturing operation and fault detection operation of the photoelectric conversion device 502 are started.

[0168] Next, in step S520, pixel signals are acquired from the valid pixels. Furthermore, in step S530, output values ​​are acquired from the failure detection pixels provided for failure detection. These failure detection pixels, like the valid pixels, have photoelectric conversion elements. A predetermined voltage is written to these photoelectric conversion elements. The failure detection pixels output signals corresponding to the voltage written to these photoelectric conversion elements. Note that steps S520 and S530 may be reversed.

[0169] Next, in step S540, a determination is made as to whether the expected output value of the fault detection pixel matches the actual output value from the fault detection pixel. If the result of the determination in step S540 indicates that the expected output value and the actual output value match, the process proceeds to step S550, where it is determined that the imaging operation is normal, and the process proceeds to step S560. In step S560, the pixel signals of the scanning row are sent to memory 505 and temporarily stored. Thereafter, the process returns to step S520, where the fault detection operation continues. On the other hand, if the result of the determination in step S540 indicates that the expected output value and the actual output value do not match, the process proceeds to step S570. In step S570, it is determined that an abnormality exists in the imaging operation, and an alarm is issued to the main control unit 513 or the alarm device 512. The alarm device 512 displays the detection of the abnormality on the display unit. Thereafter, in step S580, the photoelectric conversion device 502 is stopped, and the operation of the light detection system 501 is terminated.

[0170] In this embodiment, the flowchart is looped for each line, but the flowchart may be looped for each set of lines, or the fault detection operation may be performed for each frame. The issuance of the alarm in step S570 may be notified to the outside of the vehicle via a wireless network.

[0171] Furthermore, although the present embodiment has been described as a control for preventing collisions with other vehicles, the present invention is also applicable to control for automatic driving by following other vehicles, control for automatic driving so as not to deviate from a lane, and the like. Furthermore, the light detection system 501 is not limited to vehicles such as the subject vehicle, but can be applied to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the present invention is not limited to moving bodies, but can be applied to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0172] [Ninth embodiment] A light detection system according to a ninth embodiment of the present invention will be described with reference to Fig. 18. Fig. 18 is a schematic diagram showing a configuration example of the light detection system according to this embodiment. In this embodiment, an example of application of the light detection system to eyeglasses (smart glasses) using the photoelectric conversion device 100 described in any of the first to fourth embodiments will be described.

[0173] 18(a) shows glasses 600 (smart glasses) according to one application example. The glasses 600 include lenses 601, a photoelectric conversion device 602, and a control device 603.

[0174] The photoelectric conversion device 602 is the photoelectric conversion device 100 described in any one of the first to fourth embodiments, and is provided on the lens 601. There may be one or more photoelectric conversion devices 602. When multiple photoelectric conversion devices 602 are used, multiple types of photoelectric conversion devices 602 may be combined. The arrangement position of the photoelectric conversion device 602 is not limited to that shown in FIG. 18( a). A display device (not shown) including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 601.

[0175] The control device 603 functions as a power source that supplies power to the photoelectric conversion device 602 and the display device. The control device 603 also has a function of controlling the operations of the photoelectric conversion device 602 and the display device. The lens 601 is provided with an optical system for condensing light onto the photoelectric conversion device 602.

[0176] 18(b) shows glasses 610 (smart glasses) according to another application example. The glasses 610 include lenses 611 and a control device 612. The control device 612 may be equipped with a photoelectric conversion device (not shown) corresponding to the photoelectric conversion device 602 and a display device.

[0177] The lens 611 is provided with a photoelectric conversion device in the control device 612 and an optical system for projecting light from the display device, and an image is projected. The control device 612 functions as a power source that supplies power to the photoelectric conversion device and the display device, and also has a function of controlling the operations of the photoelectric conversion device and the display device.

[0178] The control device 612 may further include a gaze detection unit that detects the gaze of the wearer. In this case, an infrared light emitting unit may be provided in the control device 612, and the infrared light emitted from the infrared light emitting unit may be used to detect the gaze. Specifically, the infrared light emitting unit emits infrared light toward the eyeball of the user gazing at the displayed image. An imaging unit having a light receiving element detects the reflected light of the emitted infrared light from the eyeball, thereby obtaining an image of the eyeball. By providing a reduction unit that reduces the light from the infrared light emitting unit to the display unit in a planar view, it is possible to reduce degradation of image quality.

[0179] The user's line of sight with respect to the displayed image can be detected from an image of the eyeball obtained by capturing infrared light. Any known method can be applied to gaze detection using an image of the eyeball. As an example, a gaze detection method based on a Purkinje image formed by reflection of irradiated light on the cornea can be used. More specifically, a gaze detection process based on the pupil-corneal reflex method is performed. Using the pupil-corneal reflex method, a gaze vector representing the direction (rotation angle) of the eyeball is calculated based on the image of the pupil and the Purkinje image included in the image of the eyeball, thereby detecting the user's gaze.

[0180] The display device of this embodiment may include a photoelectric conversion device having a light receiving element, and may be configured to control a display image based on user line-of-sight information from the photoelectric conversion device. Specifically, the display device determines a first field of view area where the user gazes and a second field of view area other than the first field of view area based on the line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device or by an external control device. If determined by an external control device, they are communicated to the display device via communication. In the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than the resolution of the first field of view area.

[0181] The display area may also be configured to have a first display area and a second display area different from the first display area, and to determine a high-priority area from the first display area and the second display area based on line-of-sight information. The first display area and the second display area may be determined by a control device of the display device or by an external control device. If determined by an external control device, the determination is communicated to the display device via communication. The resolution of the high-priority area may be controlled to be higher than the resolution of areas other than the high-priority area. In other words, the resolution of an area with a relatively low priority may be lowered.

[0182] Note that AI may be used to determine the first field of view area and areas with high priority. The AI ​​may be a model configured to estimate the angle of gaze and the distance to an object in the line of sight from an image of the eyeball, using as training data an image of the eyeball and the actual direction in which the eyeball in the image was looking. The AI ​​program may be included in the display device, the photoelectric conversion device, or an external device. If included in an external device, it is transmitted to the display device via communication.

[0183] When display control is performed based on visual recognition detection, the present invention is preferably applied to smart glasses that further include a photoelectric conversion device for capturing images of the outside world. The smart glasses can display captured external information in real time.

[0184] [Modified embodiment] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, an example in which part of the configuration of any one of the embodiments is added to another embodiment, or an example in which part of the configuration of another embodiment is substituted therefor, is also an embodiment of the present invention.

[0185] For example, in the above embodiment, the event detection determination is performed by comparing the time count value of the time information counter 366 with the time count value of the previous frame stored in the memory 368, but the comparison does not necessarily have to be limited to the time count value of the previous frame. For example, the comparison may be performed with the time count value of two or more frames before. Alternatively, the time count values ​​for several frames may be stored in the memory 368, and the event detection determination may be performed according to the magnitude of the fluctuation in the time count value over several frames.

[0186] In the above embodiment, a recharge method for periodically recharging the APD 22 is applied as the quench circuit 32, but the quench circuit 32 does not necessarily have to be a recharge method. For example, the P-type MOS transistor Mq driven by a periodic signal may be replaced with a resistive element or an active quench circuit.

[0187] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0188] It should be noted that the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from its technical concept or main features.

[0189] The disclosure of the above embodiment includes the following configurations and methods. (Configuration 1) a photoelectric conversion unit that outputs a pulse signal in response to the incidence of a photon; a first counter that counts the pulse signals; an exposure control circuit that compares a count value of the first counter with a predetermined threshold value a plurality of times during an exposure period of one frame, and controls a count period of the pulse signal by the first counter in accordance with a result of the comparison; a second counter that counts each time the count value of the first counter is determined to be equal to or less than the threshold value; a comparison circuit that outputs event information according to a result of a comparison between a count value of the second counter in a first frame and a count value of the second counter in a second frame preceding the first frame; A photoelectric conversion device comprising: (Configuration 2) The exposure control circuit ends the counting period when the count value of the first counter exceeds the threshold value. 2. The photoelectric conversion device according to configuration 1, (Configuration 3) The exposure control circuit ends the count period when the exposure period has elapsed if the count value of the first counter does not exceed the threshold value as a result of the multiple comparisons. 2. The photoelectric conversion device according to configuration 1, (Configuration 4) The second counter counts up each time the count value of the first counter is determined to be equal to or less than the threshold value. 4. The photoelectric conversion device according to any one of configurations 1 to 3. (Configuration 5) The comparison circuit outputs the event information indicating that an event has been detected when the count value of the second counter in the first frame and the count value of the second counter in the second frame differ. 5. The photoelectric conversion device according to any one of configurations 1 to 4. (Configuration 6) The timer further includes a memory for storing the count value of the second counter when the count period ends. 6. The photoelectric conversion device according to any one of configurations 1 to 5. (Configuration 7) After the counting period is completed, the count value of the second counter is stored in the memory before the next frame starts. 7. The photoelectric conversion device according to configuration 6, (Configuration 8) The comparator circuit further compares the information of the most significant bits of the first counter when the count period ends in the first frame with the information of the most significant bits of the first counter when the count period ends in the second frame. 8. The photoelectric conversion device according to any one of configurations 1 to 7. (Configuration 9) The comparison circuit outputs the event information indicating that an event has been detected when the count value of the second counter in the first frame and the count value of the second counter in the second frame are the same and when information of the most significant bits of the first counter in the first frame and information of the most significant bits of the first counter in the second frame are different. 9. The photoelectric conversion device according to configuration 8. (Configuration 10) The method further includes a memory for storing information on the most significant bits of the first counter when the counting period ends. 10. The photoelectric conversion device according to configuration 8 or 9, (Configuration 11) The comparison circuit outputs the event information before the exposure period has elapsed. 11. The photoelectric conversion device according to any one of configurations 1 to 10. (Configuration 12) After the counting period has ended, the count value of the first counter and the count value of the second counter are output as image information. 12. The photoelectric conversion device according to any one of configurations 1 to 11. (Configuration 13) The number of bits of the second counter is less than the number of bits of the first counter. 13. The photoelectric conversion device according to any one of configurations 1 to 12. (Configuration 14) The length of the exposure period is T, the number of times the count value of the first counter is compared with the threshold value during the exposure period is N, the times at which the count value of the first counter is compared with the threshold value are T1, T2, ..., TN, and m is an arbitrary integer equal to or greater than 2, and the times T1, T2, ..., TN are expressed as T / m N ,T / m N-1 ,…,T / m 1 is expressed as 14. The photoelectric conversion device according to any one of configurations 1 to 13. (Configuration 15) The threshold is expressed as Nsat / m, where Nsat is the saturation value of the first counter. 15. The photoelectric conversion device according to configuration 14. (Configuration 16) m is a power of 2 16. The photoelectric conversion device according to configuration 14 or 15. (Configuration 17) a photoelectric conversion unit that outputs a pulse signal in response to the incidence of a photon; a first counter that counts the pulse signals; a second counter that counts the time until the first counter reaches a predetermined threshold; a comparison circuit that outputs event information according to a result of a comparison between a count value of the second counter in a first frame and a count value of the second counter in a second frame preceding the first frame; A photoelectric conversion device comprising: (Configuration 18) The comparison circuit outputs the event information indicating that an event has been detected when the count value of the second counter in the first frame and the count value of the second counter in the second frame differ. 18. The photoelectric conversion device according to configuration 17. (Configuration 19) The comparison circuit outputs the event information indicating that an event has been detected when a difference between the count value of the second counter in the first frame and the count value of the second counter in the second frame exceeds a predetermined value. 18. The photoelectric conversion device according to configuration 17. (Configuration 20) The threshold value is the upper count value of the first counter. 20. The photoelectric conversion device according to any one of configurations 17 to 19, (Configuration 21) The timer further includes a memory for storing the count value of the second counter when the count period ends. 21. The photoelectric conversion device according to any one of configurations 17 to 20. (Configuration 22) the photoelectric conversion unit includes an avalanche photodiode and a control circuit that periodically controls the avalanche photodiode in response to a periodic signal between a standby state in which avalanche multiplication is possible and a recharge state in which avalanche multiplication is resumed after avalanche multiplication has occurred, and The first counter counts the pulse signal outputted in response to a period during which the avalanche multiplication occurs within a period during which the avalanche photodiode is in the standby state. 22. The photoelectric conversion device according to any one of configurations 1 to 21, (Configuration 23) The image sensor has a plurality of pixels each having at least the photoelectric conversion unit. 23. The photoelectric conversion device according to any one of configurations 1 to 22. (Configuration 24) The photoelectric conversion device according to any one of structures 1 to 23; a signal processing device that processes a signal output from the photoelectric conversion device; An optical detection system comprising: (Configuration 25) The signal processing device generates a distance image representing distance information to an object based on the signal. 25. The optical detection system of claim 24. (Configuration 26) A mobile object, The photoelectric conversion device according to any one of structures 1 to 23; a distance information acquisition means for acquiring distance information to an object from a parallax image based on a signal output from the photoelectric conversion device; a control means for controlling the moving object based on the distance information; A moving object characterized by having: [Explanation of symbols]

[0190] 10...Pixel section 12...pixels 20...Photoelectric conversion unit 30...Signal processing circuit section 32...Quench circuit 34…Waveform shaping circuit 36...Counter circuit 38...Selection circuit 362...Photon detection counter 364...Exposure control device 366...Time information counter 368...Memory 370…Comparison circuit

Claims

1. a photoelectric conversion unit that outputs a pulse signal in response to the incidence of a photon; a first counter that counts the pulse signals; an exposure control circuit that compares a count value of the first counter with a predetermined threshold value a plurality of times during an exposure period of one frame, and controls a count period of the pulse signal by the first counter in accordance with a result of the comparison; a second counter that counts each time the count value of the first counter is determined to be equal to or less than the threshold value; a comparison circuit that outputs event information according to a result of a comparison between a count value of the second counter in a first frame and a count value of the second counter in a second frame preceding the first frame; A photoelectric conversion device comprising:

2. The exposure control circuit ends the counting period when the count value of the first counter exceeds the threshold value.

2. The photoelectric conversion device according to claim 1.

3. If the count value of the first counter does not exceed the threshold value as a result of the multiple comparisons, the exposure control circuit ends the count period when the exposure period has elapsed.

2. The photoelectric conversion device according to claim 1.

4. The second counter counts up each time the count value of the first counter is determined to be equal to or less than the threshold value.

2. The photoelectric conversion device according to claim 1.

5. The comparison circuit outputs the event information indicating that an event has been detected when the count value of the second counter in the first frame and the count value of the second counter in the second frame are different.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

6. The timer further includes a memory for storing the count value of the second counter when the count period ends.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

7. After the counting period is completed, the count value of the second counter is stored in the memory before the next frame starts.

7. The photoelectric conversion device according to claim 6.

8. The comparator circuit further compares the information of the most significant bits of the first counter when the count period ends in the first frame with the information of the most significant bits of the first counter when the count period ends in the second frame.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

9. The comparison circuit outputs the event information indicating that an event has been detected when the count value of the second counter in the first frame and the count value of the second counter in the second frame are the same and when information of the most significant bits of the first counter in the first frame and information of the most significant bits of the first counter in the second frame are different.

9. The photoelectric conversion device according to claim 8.

10. The first counter further includes a memory for storing information on the most significant bits of the first counter when the counting period ends.

9. The photoelectric conversion device according to claim 8.

11. The comparison circuit outputs the event information before the exposure period has elapsed.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

12. After the counting period has ended, the count value of the first counter and the count value of the second counter are output as image information.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

13. The number of bits of the second counter is less than the number of bits of the first counter.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

14. The length of the exposure period is T, the number of times the count value of the first counter is compared with the threshold value during the exposure period is N, the times at which the count value of the first counter is compared with the threshold value are T1, T2, ..., TN, and m is an arbitrary integer equal to or greater than 2, and the times T1, T2, ..., TN are expressed as T / m Ｎ , T / m Ｎ－１ , ..., T / m １ is expressed as 5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

15. The threshold value is expressed as Nsat / m, where Nsat is the saturation value of the first counter.

15. The photoelectric conversion device according to claim 14.

16. m is a power of 2 15. The photoelectric conversion device according to claim 14.

17. a photoelectric conversion unit that outputs a pulse signal in response to the incidence of a photon; a first counter that counts the pulse signals; a second counter that counts the time until the first counter reaches a predetermined threshold; a comparison circuit that outputs event information according to a result of a comparison between a count value of the second counter in a first frame and a count value of the second counter in a second frame preceding the first frame; A photoelectric conversion device comprising:

18. The comparison circuit outputs the event information indicating that an event has been detected when the count value of the second counter in the first frame and the count value of the second counter in the second frame differ.

18. The photoelectric conversion device according to claim 17.

19. The comparison circuit outputs the event information indicating that an event has been detected when a difference between the count value of the second counter in the first frame and the count value of the second counter in the second frame exceeds a predetermined value.

18. The photoelectric conversion device according to claim 17.

20. The threshold value is the upper count value of the first counter.

20. The photoelectric conversion device according to claim 17, wherein the first and second electrodes are electrically connected to each other.

21. The timer further includes a memory for storing the count value of the second counter when the count period ends.

20. The photoelectric conversion device according to claim 17, wherein the first and second electrodes are electrically connected to each other.

22. the photoelectric conversion unit includes an avalanche photodiode and a control circuit that periodically controls the avalanche photodiode in response to a periodic signal between a standby state in which avalanche multiplication is possible and a recharge state in which avalanche multiplication is resumed after avalanche multiplication has occurred, and The first counter counts the pulse signal outputted in response to a period during which the avalanche multiplication occurs within a period during which the avalanche photodiode is in the standby state.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

23. The image sensor has a plurality of pixels each having at least the photoelectric conversion unit.

5. The photoelectric conversion device according to claim 1, wherein the first and second electrodes are electrically connected to each other.

24. The photoelectric conversion device according to any one of claims 1 to 4, a signal processing device that processes a signal output from the photoelectric conversion device; An optical detection system comprising:

25. The signal processing device generates a distance image representing distance information to an object based on the signal.

25. The optical detection system of claim 24.

26. A mobile object, The photoelectric conversion device according to any one of claims 1 to 4, a distance information acquisition means for acquiring distance information to an object from a parallax image based on a signal output from the photoelectric conversion device; a control means for controlling the moving object based on the distance information; A moving object characterized by having:

Images