Photoelectric conversion device and photoelectric conversion system having the photoelectric conversion device

Abstract

To reduce an image quality deterioration by adjusting a sensitivity of a photoelectric conversion device, and appropriately counting an incident photon.SOLUTION: A photoelectric conversion device includes: a photoelectric conversion unit that generates a photon detection signal by an avalanche multiplication; a counter circuit that counts the photon detection signal output from the photoelectric conversion unit; a first period contained in a one frame; and a second period that is contained in the one frame, and is not overlapped with the first period after the first period. A value counted in the counter circuit on the basis of the one photon detection signal in the first period is larger than a value counted in the counter circuit on the basis of the one photon detection signal in the second period.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device and a photoelectric conversion system having the photoelectric conversion device. [Background technology]

[0002] 2. Description of the Related Art Photoelectric conversion devices capable of detecting weak light at the single photon level by utilizing avalanche (electron avalanche) multiplication are known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-039400 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the invention described in Patent Document 1, the amount of incident light is calculated from the count value and exposure time information, and the exposure time is controlled for each pixel. In such a configuration, there is a concern that image quality may deteriorate due to the occurrence of LED flicker. [Means for solving the problem]

[0005] One aspect of the present invention is characterized in that it comprises a photoelectric conversion unit that generates a photon detection signal by avalanche multiplication, a counter circuit that counts the photon detection signals output from the photoelectric conversion unit, a first period included in one frame, and a second period included in the one frame that is after the first period and does not overlap with the first period, wherein a value counted by the counter circuit based on one of the photon detection signals in the first period is greater than a value counted by the counter circuit based on one of the photon detection signals in the second period. Effect of the Invention

[0006] According to the present invention, deterioration in image quality can be reduced by adjusting the sensitivity of a photoelectric conversion device and appropriately counting the number of incident photons. [Brief description of the drawings]

[0007] [Figure 1] 1 is a schematic diagram of a photoelectric conversion device according to an embodiment. [Diagram 2] 2 is a schematic diagram of a PD substrate of a photoelectric conversion device according to an embodiment. FIG. [Diagram 3] 2 is a schematic diagram of a circuit board of a photoelectric conversion device according to an embodiment. FIG. [Figure 4] 2 is a configuration example of a pixel circuit of a photoelectric conversion device according to an embodiment. [Diagram 5] 3 is a schematic diagram showing driving of a pixel circuit of a photoelectric conversion device according to an embodiment. FIG. [Figure 6] FIG. 1 is a diagram illustrating the principle of the present invention. [Figure 7] 1 is a block diagram of a pixel of a photoelectric conversion device according to a first embodiment. [Figure 8] 1 is a diagram illustrating an example of the configuration of a pixel circuit of a photoelectric conversion device according to a first embodiment. [Figure 9] 3 is a timing chart of the photoelectric conversion device according to the first embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of the configuration of a pixel circuit of a photoelectric conversion device according to a second embodiment. [Figure 11] 10 is a timing chart of a photoelectric conversion device according to a second embodiment. [Figure 12] 10 is a timing chart of a photoelectric conversion device according to a second embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of the configuration of a pixel circuit of a photoelectric conversion device according to a third embodiment. [Figure 14] 10 is a timing chart of a photoelectric conversion device according to a third embodiment. [Figure 15] FIG. 13 is a diagram illustrating an example of the configuration of a pixel circuit of a photoelectric conversion device according to a modified example of the third embodiment. [Figure 16]13 is a timing chart of a photoelectric conversion device according to a modified example of the third embodiment. [Figure 17] FIG. 13 is a diagram illustrating an example of the configuration of a pixel circuit of a photoelectric conversion device according to a modified example of the fourth embodiment. [Figure 18] 13 is a timing chart of a photoelectric conversion device according to a modified example of the fourth embodiment. [Figure 19] FIG. 13 is a diagram illustrating an example of the configuration of a pixel circuit of a photoelectric conversion device according to a modified example of the fifth embodiment. [Figure 20] 13 is a timing chart of a photoelectric conversion device according to a modified example of the fifth embodiment. [Figure 21] FIG. 13 is a functional block diagram of a photoelectric conversion system according to a sixth embodiment. [Figure 22] FIG. 13 is a schematic diagram according to a seventh embodiment. [Diagram 23] FIG. 13 is a functional block diagram of a photoelectric conversion system according to an eighth embodiment. [Figure 24] FIG. 13 is a functional block diagram of a photoelectric conversion system according to a ninth embodiment. [Diagram 25] FIG. 23 is a functional block diagram of a photoelectric conversion system according to a tenth embodiment. [Figure 26] FIG. 23 is a functional block diagram of a photoelectric conversion system according to an eleventh embodiment. [Figure 27] FIG. 23 is a functional block diagram of a photoelectric conversion system according to a twelfth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The following embodiments are intended to embody the technical ideas of the present invention, and are not intended to limit the present invention. The sizes and positional relationships of the components shown in the drawings may be exaggerated to clarify the description. In the following description, the same components may be designated by the same reference numerals, and the description may be omitted.

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the following description, terms indicating specific directions or positions (for example, "upper", "lower", "right", "left" and other terms including these terms) will be used as necessary. The use of these terms is for the purpose of facilitating understanding of the embodiment with reference to the drawings, and the technical scope of the present invention is not limited by the meaning of these terms.

[0010] In this specification, the plan view refers to a view from a direction perpendicular to the light incident surface of the semiconductor layer. Also, the cross-sectional view refers to a surface in a direction perpendicular to the light incident surface of the semiconductor layer. Note that, when the light incident surface of the semiconductor layer is a rough surface when viewed microscopically, the plan view is defined based on the light incident surface of the semiconductor layer when viewed macroscopically.

[0011] In the following description, the anode of the avalanche photodiode (APD) is set to a fixed potential, and a signal is taken from the cathode side. Therefore, the first conductive type semiconductor region having charges of the same polarity as the signal charge as the majority carrier is an N-type semiconductor region, and the second conductive type semiconductor region having charges of a different polarity from the signal charge as the majority carrier is a P-type semiconductor region. The present invention is also valid when the cathode of the APD is set to a fixed potential, and a signal is taken from the anode side. In this case, the first conductive type semiconductor region having charges of the same polarity as the signal charge as the majority carrier is a P-type semiconductor region, and the second conductive type semiconductor region having charges of a different polarity from the signal charge as the majority carrier is an N-type semiconductor region. In the following, a case where one node of the APD is set to a fixed potential will be described, but the potentials of both nodes may fluctuate.

[0012] In this specification, when the term "impurity concentration" is used simply, it means the net impurity concentration minus the amount compensated by the impurity of the opposite conductivity type. In other words, "impurity concentration" refers to the NET doping concentration. A region where the P-type doped impurity concentration is higher than the N-type doped impurity concentration is a P-type semiconductor region. Conversely, a region where the N-type doped impurity concentration is higher than the P-type doped impurity concentration is an N-type semiconductor region.

[0013] A configuration common to each embodiment of a photoelectric conversion device and a driving method thereof according to the present invention will be described with reference to FIGS. 1 to 5. FIG.

[0014] 1 is a diagram showing the configuration of a stacked photoelectric conversion device according to an embodiment of the present invention. The photoelectric conversion device 100 is formed by stacking and electrically connecting two chips, a sensor chip 11 and a circuit chip 21. A pixel region 12 is arranged on the sensor chip 11, and a circuit region 22 that processes signals detected in the pixel region 12 is arranged on the circuit chip 21.

[0015] 2 is a layout diagram of the sensor chip 11. Pixels 101 each having a photoelectric conversion unit 102 including an APD are arranged in an array of multiple rows and multiple columns to form a pixel region 12. The pixels 101 are typically pixels for forming an image, but when used for TOF (Time of Flight), they do not necessarily need to form an image. In other words, the pixels 101 may be pixels for measuring the time when light arrives and the amount of light.

[0016] 3 is a configuration diagram of the circuit chip 21. It has a signal processing unit 103 that processes the charges photoelectrically converted by the photoelectric conversion unit 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit unit 111, a signal line 113, and a vertical scanning circuit unit 110.

[0017] The photoelectric conversion unit 102 in FIG. 2 and the signal processing unit 103 in FIG. 3 are electrically connected via connection wiring provided for each pixel.

[0018] The vertical scanning circuit section 110 receives a control pulse supplied from a control pulse generating section 115 and supplies the control pulse to each pixel. The vertical scanning circuit section 110 uses logic circuits such as a shift register and an address decoder.

[0019] The signal output from the photoelectric conversion unit 102 of each pixel is processed by a signal processing unit 103. The signal processing unit 103 is provided with a counter, a memory, etc., and digital values ​​are held in the memory.

[0020] The horizontal scanning circuit unit 111 inputs a control pulse for sequentially selecting each column to the signal processing unit 103 in order to read out the signal from the memory of each pixel in which the digital signal is held.

[0021] A signal is output to the signal line 113 from the signal processing unit 103 of the pixel selected by the vertical scanning circuit unit 110 for the selected column.

[0022] The signal output to the signal line 113 is output via an output circuit 114 to a recording unit or a signal processing unit outside the photoelectric conversion device 100 .

[0023] 2, the pixels 100 may be arranged one-dimensionally in the pixel region 12. The function of the signal processing unit 103 does not necessarily need to be provided for each pixel 101, and for example, one signal processing unit 103 may be shared by a plurality of pixels 101, and signal processing may be performed sequentially.

[0024] Fig. 4 is an example of a block diagram including the equivalent circuits of Fig. 2 and Fig. 3. In Fig. 2, a photoelectric conversion unit 102 having an APD 201 is provided on a sensor chip 11, and other members are provided on a circuit chip 21.

[0025] The APD 201 generates charge pairs according to the incident light by photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD 201. A voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD 201. A reverse bias voltage that causes the APD 201 to perform avalanche multiplication is supplied to the anode and cathode. With such a voltage supplied, the charges generated by the incident light undergo avalanche multiplication, generating an avalanche current.

[0026] When a reverse bias voltage is supplied, there are two modes: Geiger mode, which operates with the potential difference between the anode and cathode greater than the breakdown voltage, and linear mode, which operates with the potential difference between the anode and cathode close to or less than the breakdown voltage. APDs that operate in Geiger mode are called SPADs. For example, the voltage VL (first voltage) is -30V, and the voltage VH (second voltage) is 1V.

[0027] The quench element 202 is connected to a power supply that supplies a voltage VH and the APD 201. The quench element 202 has a function of converting a change in avalanche current occurring in the APD 201 into a voltage signal. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).

[0028] The signal processing section 103 has a waveform shaping section 210, a counter circuit 211, and a selection circuit 212. In this specification, it is sufficient that the signal processing section 103 has any one of the waveform shaping section 210, the counter circuit 211, and the selection circuit 212.

[0029] The waveform shaping unit 210 shapes the potential change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0030] The counter circuit 211 counts and holds the count value of the pulse signal output from the waveform shaping unit 210. When a control pulse pRES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset.

[0031] A control pulse pSEL is supplied to the selection circuit 212 from the vertical scanning circuit unit 110 in Fig. 1 via a drive line 214 (not shown in Fig. 3) in Fig. 4, and switches between electrical connection and non-connection between the counter circuit 211 and the signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal.

[0032] The electrical connection may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing unit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0033] In this embodiment, the configuration using the counter circuit 211 has been shown. However, instead of the counter circuit 211, the photoelectric conversion device 100 may be configured to acquire the pulse detection timing using a time to digital converter (hereinafter, TDC) and a memory. In this case, the generation timing of the pulse signal output from the waveform shaping unit 210 is converted into a digital signal by the TDC. To measure the timing of the pulse signal, the TDC is supplied with a control pulse pREF (reference signal) from the vertical scanning circuit unit 110 in FIG. 1 via a drive line. The TDC acquires, as a digital signal, a signal obtained by converting the input timing of the signal output from each pixel via the waveform shaping unit 210 into a relative time based on the control pulse pREF.

[0034] Fig. 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD and the output signal. Fig. 5(a) is a diagram showing the APD 201, the quench element 202, and the waveform shaping unit 210 of Fig. 2. Here, the input side of the waveform shaping unit 210 is called nodeA, and the output side is called nodeB. Fig. 5(b) shows the waveform change of nodeA in Fig. 5(a), and Fig. 5(c) shows the waveform change of nodeB in Fig. 5(a).

[0035] Between times t0 and t1, a potential difference of VH-VL is applied to the APD 201 in FIG. 5(a).

[0036] When a photon is incident at time t1, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops. When the voltage drop increases further and the potential difference applied to the APD 201 decreases, avalanche multiplication in the APD 201 stops, and the voltage level at nodeA does not drop below a certain value. After that, a current that compensates for the voltage drop from voltage VL flows through nodeA, and at time t3, nodeA is stabilized to its original potential level.

[0037] At this time, the portion of the output waveform at nodeA that exceeds a certain threshold value is waveform-shaped by the waveform shaping section 210 and is output as a signal at nodeB.

[0038] The principle of the present invention will be explained with reference to Fig. 6. Here, Fig. 6(i) is a diagram showing the probability of detecting the first photon with respect to time under each light amount condition, and an example of the weighting coefficient in each period. Fig. 6(ii) is a graph showing the probability of detecting a photon with respect to the light amount. Fig. 6(iii) is a graph showing the number of counts of the photon detection signal with respect to the light amount. Here, the photon detected first in each detection period is called the first photon.

[0039] As shown in FIG. 6(i), under high light conditions, the probability of detecting the first photon is higher at earlier times, and the detection probability decreases as time passes. When the first half of the detection period is divided into the first period and the second half into the second period, there is a high probability that detection of the first photon is completed during the first period. Under medium light conditions, the detection probability of the photon also decreases as time passes, but compared to high light conditions, the time dependency of the detection probability of the first photon is low, and the first photon may be detected even during the second period. Under low light conditions, the detection probability of the first photon is almost independent of time.

[0040] In Fig. 6(ii), dashed line A shows the probability of detecting the first photon in the first period for each light intensity, and dashed line B shows the probability of detecting the first photon in the second period for each light intensity. The line A+B, which is the sum of dashed lines A and B, shows the probability of detecting the first photon at each light intensity regardless of the period.

[0041] As shown by the dashed line A, in the first period, the probability of detecting a photon increases as the amount of light increases, and eventually saturates under high light conditions. In particular, under low to medium light conditions, the probability increases exponentially with respect to the increase in the amount of light. On the other hand, as shown by the dashed line B, in the second period, the probability of detecting a photon increases as the amount of light increases under low to medium light conditions, and the probability of detecting a photon decreases under medium to high light conditions. This is because in high light conditions, the first photon is often detected during the first period, and photons detected in the second period are those that enter after the first photon. The probability of detecting the first photon at each light level increases as the amount of light increases under low to medium light conditions, as shown by the straight line A+B, which is the sum of the dashed lines A and B, and converges to a constant value under medium to high light conditions.

[0042] Figure 6(iii) shows a graph of the count value at each light intensity. Similar to the detection probability of the first photon shown in Figure 6(ii), the second and subsequent photons are incident on the APD. That is, under low to medium light conditions, the more the light intensity increases, the more the photon count number increases, and under medium to high light conditions, the count value saturates as the light intensity increases. Under high light conditions, the increase in the count value and the increase in the number of incident photons no longer correspond one-to-one, and it can be said that the weight of one count in the count value increases.

[0043] In other words, under low to medium light conditions, the expected value of the number of detected photons depends on the amount of light, but under medium to high light conditions, the weight of the count value depends on the amount of light.

[0044] (First embodiment) FIG. 7 shows a block diagram of one pixel of the photoelectric conversion device according to the first embodiment.

[0045] A voltage VL (first voltage) is supplied to the anode of the APD 301, and a voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD 301. A reverse bias voltage that causes an avalanche multiplication operation is applied to the APD 301.

[0046] The output signal Vc of the APD 301 is input to a waveform shaping circuit 303 at the subsequent stage. A count control circuit 304, a counter circuit 305, and a readout circuit 306 are further provided at the subsequent stage of the waveform shaping circuit 303, and the output signal of the readout circuit is output to an output line. Here, the count control circuit 304 is controlled by a control signal CTRL, and an enable signal EN and a reset signal RES are input to the counter circuit 305. A selection signal SEL is input to the readout circuit 306. The photoelectric conversion device according to this embodiment includes an APD that operates by a so-called passive recharge method, in which a fixed potential is input to the quench element 302.

[0047] FIG. 8 shows an example of the circuit configuration of the photoelectric conversion device according to the first embodiment.

[0048] As an example of the waveform shaping circuit 303, an inverter circuit is provided.

[0049] A control circuit 311 and an XOR circuit 321 are provided as the count control circuit 304, and T-flip-flop circuits 312, 313, 314, and 315 for configuring an asynchronous binary counter are provided as the counter circuit 305. The read circuit 306 is a tri-state buffer. In the circuit configuration example shown in Fig. 8, a signal corresponding to photon detection is selectively input to the least significant bit or the middle bit of the counter circuit 305 using the control circuit 311, thereby adding a different integer value to the counter circuit 305 according to the photon detection timing.

[0050] Fig. 9 shows a timing chart of the operation per exposure period corresponding to one frame of the photoelectric conversion device having the circuit shown in Fig. 8. Hereinafter, Low and High of the pulse signal will be represented as L and H, respectively.

[0051] If the period from time t0 to time t1 is defined as a first period, the control signal CTRL is controlled to H during the first period, and the weighting coefficient of the count value during this period is set to 4. The period from time t1 to time t3, which is after the first period and does not overlap with the first period in time, is defined as a second period, and the control signal CTRL is controlled to L. The weighting coefficient of the count value during this period is set to 1. After time t3, the signal VR and the control signal CTRL are driven repeatedly from time t0 to time t3 a predetermined number of times.

[0052] At time t0, the enable signal EN rises, starting exposure of the kth frame. At the same time, the signal VR is set to L to initialize the control circuit 311. After the initialization, IN1 and IN2, which correspond to the outputs of the control circuit 311, are both L.

[0053] When a photon is incident on the APD 301 at time t2, the output signal Vc drops. The output signal Vc is input to the control circuit 311 as a pulse signal shaped by the shaping circuit 303, and IN1 is latched to H. IN1 is input to the T-flip-flop 312, which is the least significant bit of the counter. That is, the count value is incremented by +1.

[0054] At time t3, the signal VR becomes L. The control circuit 311 is initialized again, and both IN1 and IN2 become L. The control signal CTRL is H from time t3 to time T5.

[0055] When a photon is incident on the APD 301 at time t4, the output signal Vc drops and a pulse signal is input to the shaping circuit 303. At this time, IN2 is latched to H and input to the XOR circuit 321. The signal output from the T-flip-flop 313 is input to the other input terminal of the XOR circuit 321. That is, IN2 is counted by interrupting the middle bit of the counter, and the counter value is incremented by +4 for each IN2 signal.

[0056] Although photons are incident on the APD 301 at times t6 and t7, the photons are not counted.

[0057] At time t10, the enable signal EN falls, and the exposure of the kth frame ends.

[0058] From time t11 to time t12, the selection signal SEL goes to H, and the signal held in the counter circuit 305 is read out to the outside of the pixel.

[0059] At time t13, a photon is incident on the APD 301, but since the enable signal EN is at L, the photon is not counted.

[0060] From time t14 to t15, the reset signal RES goes to H, and the count value held by the counter circuit 305 is reset.

[0061] At time t16, the enable signal EN rises again, and exposure of the (k+1)th frame begins.

[0062] A configuration has been considered in which one frame is divided into multiple subframes, and exposure is stopped according to a comparison between the count value in each subframe and a threshold value. However, with this configuration, there is a possibility that LED flicker will occur when performing short-second exposure, and that the output will become an abnormal value or signal information will be lost when the amount of light changes within the frame.

[0063] According to the present invention, one frame is divided into multiple subframes, and each subframe is divided unequally, and photons are counted with different weighting depending on the period in which they are detected. This configuration does not cause periodic exposure stop periods, so it is possible to reduce the effects of LED flicker, and it is also possible to continue appropriate photon counting even if the amount of incident light changes within a frame.

[0064] The application of the present invention is not limited to the case where weighting coefficients are set uniformly for all pixels arranged in the pixel region 12 (pixel array). For example, when a first pixel group and a second pixel group are set in the pixel array, the length of the first period and the length of the second period, or the combination of weighting coefficients in each period, may be different between the first pixel group and the second pixel group. The first pixel group may be located in the periphery of the pixel region, and the second pixel group may be located in the center of the pixel region.

[0065] Second embodiment 10 shows an example of the circuit configuration of a photoelectric conversion device according to the second embodiment. The circuit configuration is the same as that of the photoelectric conversion device according to the first embodiment, but the photoelectric conversion device according to this embodiment is characterized in that a clock signal VR is input to the quench element 302 to perform a so-called clock recharge operation. This type of driving has the effect of suppressing current consumption in a high light environment.

[0066] FIG. 11 shows a timing chart of the operation per frame of the photoelectric conversion device according to this embodiment.

[0067] At time t0, the enable signal EN rises, and exposure of the kth frame begins.

[0068] At time t0, the clock signal VR is set to L for a short period of time to connect the cathode terminal of the APD 301 to the power supply voltage and put the APD 301 into a standby state. In this embodiment, a maximum of one photon is detected during the time from when the clock signal VR becomes L to when it becomes L again, regardless of the number of photons incident on the APD 301.

[0069] From time t0 to time t1, the control signal CTRL is controlled to H, and during this period the weighting coefficient of the count value is 4. From time t1 to time t3, the control signal CTRL is controlled to L, and during this period the weighting coefficient of the count value is 1. After time t3, the driving of the control signal CTRL from time t0 to time t3 is repeated a predetermined number of times.

[0070] When a photon is incident on the APD 301 at time t2, the output signal Vc drops. The output signal Vc is input to the control circuit 311 as a pulse signal shaped by the shaping circuit 303, and IN1 becomes H. IN1 is input to the T-flip-flop 312, which is the least significant bit of the counter. That is, the count value is incremented by +1.

[0071] At time t3, the clock signal VR goes low, and the cathode terminal of the APD 301 is connected to the power supply voltage and recharged. At the same time, the control circuit 311 is initialized, and IN1 and IN2 go low. When the signal VR returns to high, the APD 301 goes into standby. The control signal CTRL is high from time t3 to time T5.

[0072] When a photon is incident on the APD 301 at time t4, the output signal Vc drops and a pulse signal is input to the shaping circuit 303. At this time, IN2 becomes H and is input to the XOR circuit 321. The signal output from the T-flip-flop 313 is input to the other input terminal of the XOR circuit 321. That is, IN2 is counted by interrupting the middle bit of the counter, and the counter value is incremented by +4 for each signal of IN2.

[0073] Photons are also incident on APD 301 at times t6 and t7. However, Vc has dropped due to the avalanche multiplication caused by the photon incident at time t4, and no avalanche multiplication occurs due to newly incident photons, so the counter value is not incremented.

[0074] At time t10, the enable signal EN falls, and the exposure of the kth frame ends.

[0075] From time t11 to time t12, the selection signal SEL goes to H, and the signal held in the counter circuit 305 is read out to the outside of the pixel.

[0076] At time t13, a photon is incident on the APD 301, but since the enable signal EN is at L, the photon is not counted.

[0077] From time t14 to time t15, the reset signal RES goes to H, and the count value held by the counter circuit 305 is reset.

[0078] At time t16, the enable signal EN rises, and exposure of the (k+1)th frame begins.

[0079] 12 shows a timing chart according to a modification of the second embodiment. In this embodiment, the period from time t0 to time t1 and the period from time t1 to time t3 have the same length. That is, the period with a weighting factor of 4 and the period with a weighting factor of 1 are set to have the same length.

[0080] At time t0, the enable signal EN rises, starting exposure of the kth frame. At the same time, the signal VR is set to L to recharge the APD 301. When the signal VR becomes H, the APD 301 enters a standby state, and can detect up to one photon incident on the APD 301 until the signal VR is set to L again.

[0081] From time t0 to time t1, the control signal CTRL is controlled to H, and during this period the weighting coefficient of the count value is 4. From time t1 to time t3, the control signal CTRL is controlled to L, and during this period the weighting coefficient of the count value is 1. After time t3, VR and the control signal CTRL repeat the drive from time t0 to time t3 a predetermined number of times.

[0082] When a photon is incident on the APD 301 at time t2, the output signal Vc drops. The output signal Vc is input to the control circuit 311 as a pulse signal shaped by the shaping circuit 303, and IN1 becomes H. IN1 is input to the T-flip-flop 312, which is the least significant bit of the counter. That is, the count value is incremented by +1.

[0083] At time t3, the signal VR becomes L. The cathode terminal of the APD 301 is connected to the power supply voltage and recharged. When the signal VR returns to H, the APD 301 enters a standby state. The control signal CTRL is H from time t3 to time T5.

[0084] When a photon is incident on the APD 301 at time t4, the output signal Vc drops and a pulse signal is input to the shaping circuit 303. At this time, IN2 becomes H and is input to the XOR circuit 321. The signal output from the T-flip-flop 313 is input to the other input terminal of the XOR circuit 321. That is, IN2 is counted by interrupting the middle bit of the counter, and the counter value is incremented by +4 for each signal of IN2.

[0085] Photons are also incident on the APD 301 at times t6 and t7. However, the output signal Vc drops due to avalanche multiplication caused by the incident photon at time t4 in the same recharge cycle, and a voltage equal to or greater than the breakdown voltage is not applied to the APD 301. Therefore, avalanche multiplication due to the incident photons at times t6 and t7 does not occur. The counter value is not incremented due to photon detection.

[0086] At time t10, the enable signal EN falls, and the exposure of the kth frame ends.

[0087] From time t11 to time t12, the selection signal SEL goes to H, and the signal held in the counter circuit 305 is read out to the outside of the pixel.

[0088] At time t13, a photon is incident on the APD 301, but since the enable signal EN is at L, the photon is not counted.

[0089] From time t14 to t15, the reset signal RES goes to H, and the count value held by the counter circuit 305 is reset.

[0090] At time t16, the enable signal EN rises again, and exposure of the (k+1)th frame begins.

[0091] In the case of clock recharge driving, photons are counted during a period when the weighting factor is 1 when "no photons are detected from time t0 to time t1, and photons are detected from time t1 to time t3." Therefore, even if the period when the weighting factor is 4 and the period when the weighting factor is 1 are set to be of equal length, the probability that photons are counted during a period when the weighting factor is 4 is higher than the probability that photons are counted during a period when the weighting factor is 1.

[0092] The length of the period during which each weighting factor is set and the weighting factors to be set are merely examples and are not limited to these. The length of the period during which each weighting factor is set can be set appropriately depending on the weighting factor to be set.

[0093] (Third embodiment) In the third embodiment described below, weighted counting is performed by inputting a different number of pulses to a counter circuit according to the timing of photon detection, rather than inserting a pulse into the middle bit of a counter as in the first embodiment. With this configuration, it is possible to reduce the circuit scale of the pixel circuit compared to the first embodiment.

[0094] In the first and second embodiments, the frame period is divided into two, and the count value is weighted by two weighting coefficients corresponding to each of the two periods. In this embodiment, the frame period can be divided into, for example, three periods, and three weighting coefficients can also be provided. By appropriately setting the number of divisions of the frame period and the weighting coefficients, it is possible to improve the SNR and expand the dynamic range at the same time.

[0095] FIG. 13 shows an example of the circuit configuration of a photoelectric conversion device according to the third embodiment.

[0096] Similar to the photoelectric conversion devices according to the first and second embodiments, a quench element 302 is provided between a power supply and the cathode terminal of the APD 301, and a waveform shaping circuit 303, an AND circuit 325, a counter circuit 305, and a readout circuit 306 are provided downstream of the cathode terminal. An output signal of the waveform shaping circuit 303 is input to one terminal of the AND circuit 325, and a clock signal TCLK is input to the other terminal.

[0097] FIG. 14 shows a timing chart according to this embodiment.

[0098] At time t0, the enable signal EN rises, and exposure of the kth frame begins.

[0099] In this embodiment, one recharge period is from time t0 to time t5 in Fig. 14. Each recharge period is further divided into four periods, and a weighting coefficient is set for each period. Here, the weighting coefficient represents "the total count number added to the counter within the relevant recharge period as a result of photons being received by the APD 301 at the timing of interest."

[0100] At time t1, the clock signal TCLK goes to H. The clock signal TCLK defines the timing for switching the weighting coefficients, and if a photon has already been detected within the relevant recharge period and the output signal Vc is L, the count value of the counter circuit 305 is incremented by 1. Since no photon has been detected at time t1, no increment of the counter value occurs.

[0101] At time t2, a photon is incident on the APD 301, and the output signal Vc becomes L.

[0102] At time t3, the clock signal TCLK goes to H, and the count value of the counter circuit 305 is incremented by one.

[0103] At time t4, the clock signal TCLK goes high again, and the count value of the counter circuit 305 is incremented by 1. That is, the count value is incremented by 2 in total from t1 to t4.

[0104] From time t4 to time t5, the enable signal EN is set to L, and the counter circuit 305 is put into a disabled state. During this period, no photons are detected that are incident on the APD 301.

[0105] After time t5, the operations from time t0 to time t5 are repeated.

[0106] In the recharge cycle starting from time t5, at time t6, a photon is incident on the APD 301. At times t7, t9, and t11, the clock signal TCLK goes high, and the counter circuit 305 is incremented by 1 in synchronization with the rising edge of each clock signal TCLK. That is, the count value is incremented by 3 in total from t5 to t11.

[0107] From time t16 to time t17, the shaping signal SEL goes to H, and the signal held in the counter circuit 305 is read out to the outside of the pixel.

[0108] At time t18, a photon is incident on the APD 301, but since the enable signal EN is at L, the signal based on this photon is not counted.

[0109] From time t19 to time t20, the reset signal RES goes to H, resetting the value held in the counter circuit 305.

[0110] At time t21, the enable signal EN rises, starting exposure of the (k+1)th frame.

[0111] When a photon detection signal is output during a first period, a natural number N is added to the counter circuit 305, and when a photon detection signal is output during a second period, a natural number M is added to the counter circuit 305. Furthermore, when a photon detection signal is output during a third period, a natural number L is added to the counter circuit. At this time, the natural number N is larger than the natural number M, and the natural number L is smaller than the natural number M. In this embodiment, a case where three types of weighting coefficients are set has been described, but the number of settings is not limited to three.

[0112] FIG. 15 shows a modified example of the circuit configuration of the photoelectric conversion device according to the third embodiment.

[0113] As a difference from the circuit shown in Fig. 13, this configuration example has a control circuit 322, a D flip-flop circuit 323, and a multiplexer 324. The D flip-flop circuit 323 holds a history of whether or not a photon was detected in the previous recharge cycle. If no photons were detected in the previous recharge cycle, no weighting is applied to the count value of the current recharge cycle, and one count is added. This configuration reduces the possibility of counting noise as a photon detection result. This suppresses statistical variation in the count value, and is expected to improve the SNR, especially in low light environments.

[0114] Here, the clock signal BCLK is a clock signal for generating pulses of the clock signal VR and the clock signal TCLK. The trigger signal TRIG is a pulse for selectively generating the clock signal VR and the clock signal TCLK from the clock signal BCLK. When the trigger signal TRIG is H and a pulse of the clock signal BCLK is input, a pulse of inverted logic is generated in the clock signal VR, and when the trigger signal TRIG is L and a pulse of the clock signal BCLK is input, a pulse of positive logic is generated in the clock signal TCLK.

[0115] Compared to transmitting the clock signals VR and TCLK over separate signal lines, generating them using a common clock signal BCLK is less susceptible to the effects of skew and the like, and image quality degradation is less likely to occur even when the number of pixels is large.

[0116] The control circuit 322 is configured, for example, by a combination of a NAND circuit and an AND circuit. The clock signal BCLK and the trigger signal TRIG are input to the input terminals of the NAND circuit. The clock signal BCLK and the inverted signal of the trigger signal TRIG are input to the input terminals of the AND circuit.

[0117] FIG. 16 shows a timing chart of the configuration example shown in FIG.

[0118] D flip-flop circuit 323 sets signal STAT to H if a photon was detected in the immediately preceding recharge cycle, and sets it to L if a photon is detected. When signal STAT is L, if a photon is detected within the cycle of interest, the count value is incremented by 1 regardless of the timing. When signal STAT is H, if a photon is detected within the cycle of interest, either 3, 2, or 1 is incremented to the counter depending on the timing.

[0119] At time t0, the enable signal EN rises, starting exposure of the kth frame.

[0120] No photons are detected from time t0 to time t4. The signal STAT becomes L during the recharge period from time t4.

[0121] At time t5, a photon is detected. Because the signal STAT is at L, the counter is incremented by 1 at the same time as the photon detection event occurs.

[0122] From time t9 to time t14, the signal STAT is at H level since photons were detected in the immediately preceding recharge period (time t4 to time t9).

[0123] At time t11, a photon is detected. Because the signal STAT is H, the count value is incremented by one at times t12 and t13 when the clock signal TCLK rises, resulting in a total of two being incremented.

[0124] From time t14 to time t18, a photon was detected in the previous recharge cycle (time t9 to time t14), so the signal STAT remains H. Thereafter, if a photon is detected in the cycle immediately prior to each recharge cycle, the signal STAT becomes H, and if no photon is detected, the signal STAT becomes L.

[0125] At time t15, the enable signal EN falls, and incident light thereafter is not counted.

[0126] At time t23, the enable signal EN rises and exposure of the (k+1)th frame begins. The signal STAT may be forced to L in time with the change of frames, or the level of the signal STAT in the first recharge period of the (k+1)th frame may be determined based on the photon detection result in the last recharge period of the kth frame.

[0127] In this embodiment, the method of weighting the counts is changed based on whether or not photons were detected in the immediately preceding recharge cycle, but the weighting method may also be changed based on whether or not photons were detected in a recharge cycle two or more cycles prior, or may be changed based on whether or not photons were detected in multiple past recharge cycles.

[0128] (Fourth embodiment) FIG. 17 shows an example of the circuit configuration of a photoelectric conversion device according to the fourth embodiment.

[0129] The photoelectric conversion device according to this embodiment is characterized in that it controls whether or not to count up the pixel of interest depending on the presence or absence of photon detection in the surrounding pixels arranged around the pixel of interest. A predetermined value is added to the counter when photons are detected in both the pixel of interest A and the surrounding pixel B within the same recharge period. The surrounding pixels referenced to determine the signal addition of the pixel of interest may be one adjacent pixel or multiple pixels. Pixels arranged in an array may form a one-dimensional or two-dimensional network, and the photon detection states of adjacent pixels may be mutually referenced. In the following description, the pixels adjacent to the pixel of interest are referred to as the surrounding pixels, but the relationship between the pixel of interest and the surrounding pixels is not limited to this.

[0130] The upper part of Fig. 17 shows pixel of interest A, and the lower part shows surrounding pixel B. Pixel of interest A and surrounding pixel B are configured in the same way. That is, a quench element 302 is provided between a power supply and the cathode terminal of APD 301, and a waveform shaping circuit 303 is connected to the cathode terminal. Connected to the rear of this element are a control circuit 311, T flip-flop circuits 312-315 and XOR circuit 321 constituting counter circuit 305, and a readout circuit 306. Corresponding elements of pixel of interest A and surrounding pixel B are assigned common reference symbols, and the suffix "A" or "B" at the end of the reference symbol identifies which pixel the element belongs to.

[0131] The photoelectric conversion device according to this embodiment differs from the photoelectric conversion devices according to the first to third embodiments in that wiring is provided to mutually connect the control circuit 311A ​​and the control circuit 311B. The control circuit 311A ​​controls the outputs of IN1A and IN2A based on the output signal VA of the pixel of interest A and the output signal VB of the peripheral pixel B. As an example, either IN1A or IN1B is latched to H depending on the timing when both VA and VB become H.

[0132] FIG. 18 shows a timing chart for pixel A and pixel B when the configuration of FIG. 17 is used.

[0133] At time t0, the enable signal EN rises, starting exposure of the kth frame. The period from time t0 to time t4 corresponds to one recharge period.

[0134] At time t2, a photon is detected by the APD 301 A. At this time, the pixel B surrounding the pixel A of interest does not detect a photon, so the counter circuit 305 does not add a signal thereto.

[0135] At time t3, the APD 301B of the peripheral pixel B also detects a photon. In response to the rising edge of the output VB of the waveform shaping circuit 303B, IN1A becomes H and a signal is added to the counter circuit 305. At time t3, the control signal CTRL is L, so the count value added is 1.

[0136] At time t5, APD 301B of peripheral pixel B detects a photon, and at time t6, APD 301A of target pixel A detects a photon. Since the control signal CTRL is H at time t6, IN2A goes H in response to the rising edge of output VB of 303B, and a signal is added to XOR circuit 321A. That is, the count value is incremented by 4.

[0137] In this way, by reflecting the presence or absence of photon detection from the surrounding pixels in the addition of the count value of the pixel of interest, the effective sensitivity can be appropriately adjusted, thereby ensuring gradation at higher light levels and expanding the dynamic range.

[0138] Fifth embodiment 19 shows an example of the circuit configuration of a photoelectric conversion device according to the fifth embodiment. In the photoelectric conversion device shown in this embodiment, when the count value of the counter circuit 305 exceeds a predetermined threshold value, the weighting coefficient of the count value is changed.

[0139] 19, the photoelectric conversion device has a quench element 302 between a power supply and a cathode terminal of an APD 301, and a waveform shaping circuit 303 is connected to the cathode terminal. A count control circuit 304, a counter circuit 305, and a readout circuit 306 are arranged downstream of the waveform shaping circuit 303. The output of the counter circuit 305 is also input to a latch circuit 307, and an output signal STAT of the latch circuit 307 is input to the count control circuit 304.

[0140] FIG. 20 shows a timing chart in the case where the photoelectric conversion device having the configuration shown in FIG. 19 is used.

[0141] At time t0, the enable signal EN rises, starting exposure of the kth frame. The period from time t0 to time t3 is one recharge period. When the control signal CTRL is H, 4 is added to the count value, and when the control signal CTRL is L, 1 is added to the count value. In this embodiment, the threshold value of the count value is set to m.

[0142] At time t1, the control signal CTRL falls from H to L. At this time, the output signal STAT is L.

[0143] At time t2, a photon is incident on the APD 301, and the output signal Vc falls. Since the control signal CTRL is at L, the count value is incremented by one.

[0144] At time t4, a photon is detected by the APD 301, and the count value becomes m+1. Since the count value exceeds the threshold m, the output signal STAT switches from L to H.

[0145] In the recharge period after time t7, when the control signal CTRL is H, 2 is added to the count value, and when the control signal CTRL is L, 1 is added to the count value.

[0146] With this configuration, it is possible to expand the dynamic range while reducing the circuit scale of the photoelectric conversion device.

[0147] Sixth embodiment A photoelectric conversion system using the photoelectric conversion device according to each of the above-mentioned embodiments will be described with reference to Fig. 21. Fig. 21 is a block diagram showing a schematic configuration of the photoelectric conversion system according to this embodiment.

[0148] The processing device according to the sixth embodiment includes a control unit 401 , a timing adjustment unit 402 , an image acquisition unit 403 , a readout unit 404 , a gain adjustment unit 405 , a non-linear correction unit 406 , a defect correction unit 407 , a data compression unit 408 , and a memory unit 409 .

[0149] The image acquisition unit 403 is, for example, the pixel 101 in the pixel region 12 shown in Fig. 2, and the readout unit 404 is, for example, the signal processing unit 103 in Fig. 3. The timing adjustment unit 402 is, for example, the control pulse generation unit 115 in Fig. 3. The control unit 401 may be a control unit inside the photoelectric conversion device, or may be outside the photoelectric conversion device. The image acquisition unit 403 is controlled by the timing adjustment unit 402 controlled by the control unit 401.

[0150] The image data generated by the image acquisition unit is subjected to correction processing and input to the storage unit 409. Note that the order of the correction processing is not limited to the order shown in FIG.

[0151] The gain adjustment unit 405 is provided between the readout unit 404 and the nonlinear correction unit 406, and applies a digital gain to the image data generated by the image acquisition unit 403. Data for image correction often has a decimal value, but when the image output is an integer, the correction accuracy may decrease due to quantization error. By applying a gain to the image data in advance, the effect of quantization error can be suppressed and the correction accuracy can be improved. Here, if the quantization error can be suppressed to 1 / 4 times or less of the one-photon signal level, the corrected image will be visually natural. Therefore, it is desirable that the digital gain applied to the image data is, for example, 4 times or more.

[0152] The nonlinear correction unit 406 is disposed between the gain adjustment unit 405 and the defect correction unit 407, and corrects the image data under the control of the control unit 401. As described in the above embodiments, when the image acquisition unit 403 is a photon-counting type detector, the optical response often becomes nonlinear due to the effect of dead time. When affected by a nonlinear optical response, correction based on a linear response may result in overcorrection. Therefore, by performing nonlinear correction on the image data before the calculation process in the defect correction unit 407, overcorrection can be prevented, and appropriate nonlinear correction according to the drive timing can be performed. This nonlinear correction is performed using, for example, a LookUpTable.

[0153] The defect correction unit 407 corrects the data of defective pixels contained in the image data. As a specific example, the output value of the defective pixel is extracted, and the position information and output value of the defective pixel are identified. There are a method of replacing the output value with the average value or median value of the output of the pixels around the identified defective pixel, and a method of dividing the estimated defective image data.

[0154] The data compression unit 408 compresses the corrected image data. In the photoelectric conversion device according to the present invention, a huge amount of image data corresponding to a high dynamic range is generated. By providing the data compression unit 408, the data can be compressed before being stored in the storage unit 409 at the downstream.

[0155] The storage unit 409 is a storage unit that holds at least a part of the image data generated in the previous stage. Specifically, the image data is stored in a memory such as an SRAM, a DRAM, or a non-volatile memory.

[0156] In this manner, according to this embodiment, it is possible to realize a photoelectric conversion system to which the photoelectric conversion device shown in any of the above-described embodiments is applied.

[0157] Seventh embodiment The effects of the present invention will be further described with reference to Fig. 22. In the following description, the case where addition is performed according to each of the above-mentioned embodiments is referred to as a weighted counting method.

[0158] In the conventional clock recharge method shown in Fig. 22(a), the maximum estimated number of incident photons is determined by the number of Recharge CLKs (clock signals TCLK). In other words, the more Recharge CLKs there are, the wider the dynamic range becomes. On the other hand, power consumption increases in proportion to the number of Recharge CLKs, so there is a trade-off between dynamic range and power consumption.

[0159] When the exposure period is T and the pulse interval of Recharge CLK is Δtr, the number of Recharge CLKs is expressed by T / Δtr, so it can also be considered that the dynamic range is determined by T / Δtr.

[0160] In contrast, in the weighted counting method as shown in FIG. 22(b), the number of incident photons is estimated from the timing at which the photons are incident. Therefore, the maximum estimated number of incident photons is determined by T / Δtw. Here, Δtw corresponds to the first period in the first embodiment, and is the period during which an integer value is added to the counter circuit. In other words, the maximum estimated number of incident photons is no longer dependent on the Recharge CLK interval Δtr, and there is no longer a trade-off between the dynamic range and the power consumption. In this case, the weighting coefficient is preferably Δtr / Δtw.

[0161] Specific numerical examples are shown below. In the conventional method, when the exposure period T is 1024 and Δtr is 1, the maximum number of avalanches is 1024, and the maximum number of detected incident photons at this time is 1024. On the other hand, in the weighted count method, when the exposure period T is 1024, Δtr is 4, and Δtw is 1, the maximum number of avalanches is 256. If addition is made to the counter circuit so that the number of counts corresponding to photons incident during Δtw becomes Δtr / Δtw, or 4, then the maximum estimated number of incident photons at this time will be 1024. In other words, it is possible to achieve the same dynamic range while keeping the power consumption associated with recharging to one-quarter of that of the conventional method.

[0162] Here, it is preferable to set the weighting coefficient to Δtr / Δtw because this allows the saturation counts to be the same under driving conditions where both methods have the same dynamic range. Setting such conditions makes it possible to reduce false signals during nonlinear correction. However, since power consumption depends only on Δtr and not on the weighting coefficient, it is not necessary to satisfy the above relationship from the viewpoint of reducing power consumption.

[0163] Eighth embodiment The photoelectric conversion system according to this embodiment will be described with reference to Fig. 23. Fig. 23 is a block diagram showing a schematic configuration of the photoelectric conversion system according to this embodiment.

[0164] The photoelectric conversion device described in the above embodiment is applicable to various photoelectric conversion systems. Examples of the applicable photoelectric conversion system include digital still cameras, digital camcorders, security cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, and observation satellites. In addition, a camera module having an optical system such as a lens and an imaging device is also included in the photoelectric conversion system. FIG. 23 illustrates a block diagram of a digital still camera as an example of these.

[0165] 23 includes an image pickup device 1004, which is an example of a photoelectric conversion device, and a lens 1002 that forms an optical image of a subject on the image pickup device 1004. The photoelectric conversion system further includes an aperture 1003 that varies the amount of light passing through the lens 1002, and a barrier 1001 that protects the lens 1002. The lens 1002 and the aperture 1003 form an optical system that focuses light on the image pickup device 1004. The image pickup device 1004 is a photoelectric conversion device according to any one of the above embodiments, and converts the optical image formed by the lens 1002 into an electrical signal.

[0166] The photoelectric conversion system has a signal processing unit 1007 which is an image generating unit that generates an image by processing an output signal output from an imaging device 1004. The signal processing unit 1007 performs various corrections and compression as necessary to output image data. The signal processing unit 1007 may be formed in a semiconductor layer in which the imaging device 1004 is provided, or may be formed in a semiconductor layer different from that of the imaging device 1004. Moreover, the imaging device 1004 and the signal processing unit 1007 may be formed in the same semiconductor layer.

[0167] The photoelectric conversion system further includes a memory unit 1010 for temporarily storing image data, and an external interface unit (external I / F unit) 1013 for communicating with an external computer or the like. The photoelectric conversion system further includes a recording medium 1012 such as a semiconductor memory for recording or reading out imaging data, and a recording medium control interface unit (recording medium control I / F unit) 1011 for recording or reading out data on the recording medium 1012. The recording medium 1012 may be built into the photoelectric conversion system, or may be removable.

[0168] The photoelectric conversion system further includes an overall control / calculation unit 1009 that performs various calculations and controls the entire digital still camera, and a timing generation unit 1008 that outputs various timing signals to the image capture device 1004 and the signal processing unit 1007. Here, the timing signals and the like may be input from outside, and the photoelectric conversion system only needs to include at least the image capture device 1004 and the signal processing unit 1007 that processes the output signal output from the image capture device 1004.

[0169] The imaging device 1004 outputs an imaging signal to a signal processing unit 1007. The signal processing unit 1007 performs predetermined signal processing on the imaging signal output from the imaging device 1004, and outputs image data. The signal processing unit 1007 generates an image using the imaging signal.

[0170] In this way, according to this embodiment, it is possible to realize a photoelectric conversion system to which the photoelectric conversion device (imaging device) of any of the above embodiments is applied.

[0171] Ninth embodiment The photoelectric conversion system and the moving object of this embodiment will be described with reference to Fig. 24. Fig. 24 is a diagram showing the configuration of the photoelectric conversion system and the moving object of this embodiment.

[0172] FIG. 24(a) shows an example of a photoelectric conversion system related to an in-vehicle camera. The photoelectric conversion system 2300 has an imaging device 2310. The imaging device 2310 is the photoelectric conversion device described in any of the above embodiments. The photoelectric conversion system 2300 has an image processing unit 2312 that performs image processing on a plurality of image data acquired by the imaging device 2310. The photoelectric conversion system 2300 also has a parallax acquisition unit 2314 that calculates parallax (phase difference of parallax images) from a plurality of image data acquired by the photoelectric conversion system 2300. The photoelectric conversion system 2300 further has a distance acquisition unit 2316 that calculates a distance to an object based on the calculated parallax, and a collision determination unit 2318 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 2314 and the distance acquisition unit 2316 are examples of distance information acquisition means that acquire distance information to an object. That is, distance information may be acquired using not only the phase difference but also ToF (Time Of Flight) technology. The collision determination unit 2318 may determine the possibility of a collision using any of these pieces of distance information. The distance information acquisition means may be realized by dedicated hardware or a software module. Also, it may be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like, or a combination of these.

[0173] The photoelectric conversion system 2300 is connected to a vehicle information acquisition device 2320, and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 2300 is also connected to a control ECU 2330, which is a control device (control unit) that outputs a control signal to generate a braking force for the vehicle based on the judgment result of the collision judgment unit 2318. The photoelectric conversion system 2300 is also connected to an alarm device 2340 that issues an alarm to the driver based on the judgment result of the collision judgment unit 2318. For example, when the judgment result of the collision judgment unit 2318 indicates that there is a high possibility of a collision, the control ECU 2330 performs vehicle control to avoid a collision and reduce damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 2340 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., or vibrating the seat belt or steering wheel.

[0174] In this embodiment, the surroundings of the vehicle, for example, the front or rear, are imaged by the photoelectric conversion system 2300. Fig. 24(b) shows a photoelectric conversion system for imaging the area in front of the vehicle (imaging range 2350). A vehicle information acquisition device 2320 sends an instruction to the photoelectric conversion system 2300 or the imaging device 2310. This configuration can further improve the accuracy of distance measurement.

[0175] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from lanes, etc. Furthermore, the photoelectric conversion system is not limited to vehicles such as the vehicle itself, but can be applied to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the present invention can be applied not only to moving bodies, but also to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0176] (Tenth embodiment) The photoelectric conversion system of this embodiment will be described with reference to Fig. 25. Fig. 25 is a block diagram showing an example of the configuration of a range image sensor which is the photoelectric conversion system.

[0177] 25, the distance image sensor 1401 is configured to include an optical system 1402, a photoelectric conversion device 1403, an image processing circuit 1404, a monitor 1405, and a memory 1406. The distance image sensor 1401 can obtain a distance image according to the distance to the subject by receiving light (modulated light or pulsed light) that is projected from a light source device 1411 toward the subject and reflected by the surface of the subject.

[0178] The optical system 1402 is configured to have one or more lenses, and guides image light (incident light) from a subject to the photoelectric conversion device 1403 , forming an image on the light receiving surface (sensor portion) of the photoelectric conversion device 1403 .

[0179] The photoelectric conversion device 1403 is any of the photoelectric conversion devices described in the above-mentioned embodiments, and a distance signal indicating a distance determined from a received light signal output from the photoelectric conversion device 1403 is supplied to an image processing circuit 1404.

[0180] The image processing circuit 1404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 1403. The distance image (image data) obtained by this image processing is then supplied to a monitor 1405 for display, or supplied to a memory 1406 for storage (recording).

[0181] In the range image sensor 1401 configured in this way, by applying the above-mentioned photoelectric conversion device, it is possible to obtain, for example, a more accurate range image as the pixel characteristics improve.

[0182] (Eleventh embodiment) The photoelectric conversion system of this embodiment will be described with reference to Fig. 26. Fig. 26 is a diagram showing an example of a schematic configuration of an endoscopic surgery system which is the photoelectric conversion system of this embodiment.

[0183] 26 shows a state in which an operator (doctor) 1131 is performing surgery on a patient 1132 on a patient bed 1133 using an endoscopic surgery system 1103. As shown in the figure, the endoscopic surgery system 1103 is composed of an endoscope 1100, a surgical tool 1110, and a cart 1134 on which various devices for endoscopic surgery are mounted.

[0184] The endoscope 1100 is composed of a lens barrel 1101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 1132, and a camera head 1102 connected to the base end of the lens barrel 1101. In the illustrated example, the endoscope 1100 is configured as a so-called rigid lens barrel having a rigid lens barrel 1101, but the endoscope 1100 may be configured as a so-called flexible lens barrel having a flexible lens barrel.

[0185] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 1101. A light source device 1203 is connected to the endoscope 1100, and light generated by the light source device 1203 is guided to the tip of the lens barrel 1101 by a light guide extending inside the lens barrel 1101, and is irradiated via the objective lens toward an observation target in a body cavity of a patient 1132. The endoscope 1100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0186] An optical system and a photoelectric conversion device are provided inside the camera head 1102, and reflected light (observation light) from an observation target is collected by the optical system onto the photoelectric conversion device. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The photoelectric conversion device may be any of the photoelectric conversion devices described in the above-mentioned embodiments. The image signal is transmitted to a camera control unit (CCU) 1135 as RAW data.

[0187] The CCU 1135 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and performs overall control of the operations of the endoscope 1100 and the display device 1136. Furthermore, the CCU 1135 receives an image signal from the camera head 1102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0188] Under the control of the CCU 1135 , the display device 1136 displays an image based on an image signal that has been subjected to image processing by the CCU 1135 .

[0189] The light source device 1203 is composed of a light source such as an LED (Light Emitting Diode), and supplies the endoscope 1100 with irradiation light when photographing an operation site or the like.

[0190] The input device 1137 is an input interface for the endoscopic surgery system 1103. A user can input various information and instructions to the endoscopic surgery system 1103 via the input device 1137.

[0191] The treatment tool control device 1138 controls the driving of the energy treatment tool 1112 for cauterizing tissue, incising, sealing blood vessels, or the like.

[0192] The light source device 1203 that supplies irradiation light to the endoscope 1100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 1203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation target with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 1102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0193] Furthermore, the light source device 1203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 1102 in synchronization with the timing of the change in the light intensity to acquire images in a time-division manner and synthesizing the images, it is possible to generate an image with a high dynamic range that is free of so-called blocked-up shadows and blown-out highlights.

[0194] The light source device 1203 may be configured to supply light in a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, the wavelength dependency of light absorption in body tissue is utilized. Specifically, a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast by irradiating light in a narrower band than the irradiation light (i.e., white light) in normal observation. Alternatively, the special light observation may be a fluorescent observation in which an image is obtained by fluorescence generated by irradiating excitation light. In the fluorescent observation, it is possible to irradiate excitation light to the body tissue and observe the fluorescence from the body tissue, or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 1203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0195] (Twelfth embodiment) The photoelectric conversion system of this embodiment will be described with reference to FIG. 27. FIG. 27(a) is a diagram showing an example of the configuration of glasses 1600 (smart glasses) which are the photoelectric conversion system. The glasses 1600 have a photoelectric conversion device 1602. The photoelectric conversion device 1602 is the photoelectric conversion device described in the above-mentioned twelfth embodiment. A display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 1601. The photoelectric conversion device 1602 may be one or more. A combination of multiple types of photoelectric conversion devices may also be used. The arrangement position of the photoelectric conversion device 1602 is not limited to that shown in FIG. 27(a).

[0196] The glasses 1600 further include a control device 1603. The control device 1603 functions as a power source that supplies power to the photoelectric conversion device 1602 and the display device. The control device 1603 also controls the operations of the photoelectric conversion device 1602 and the display device. The lens 1601 is formed with an optical system for focusing light on the photoelectric conversion device 1602.

[0197] FIG. 27(b) illustrates glasses 1610 (smart glasses) according to one application example. The glasses 1610 have a control device 1612, and the control device 1612 is equipped with a photoelectric conversion device corresponding to the photoelectric conversion device 1602 and a display device. The lens 1611 is formed with an optical system for projecting light emitted from the photoelectric conversion device in the control device 1612 and the display device, and an image is projected onto the lens 1611. The control device 1612 functions as a power source for supplying power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device may have a line of sight detection unit that detects the line of sight of the wearer. Infrared light may be used for detecting the line of sight. The infrared light emission unit emits infrared light toward the eyeball of a user gazing at a display image. An imaging unit having a light receiving element detects the reflected light of the emitted infrared light from the eyeball, thereby obtaining an image of the eyeball. By having a reduction means for reducing light from the infrared light emission unit to the display unit in a planar view, deterioration of image quality is reduced.

[0198] The gaze of the user with respect to the displayed image is detected from an image of the eyeball obtained by capturing infrared light. Any known method can be applied to gaze detection using the image of the eyeball. As an example, a gaze detection method based on a Purkinje image formed by reflection of irradiated light on the cornea can be used.

[0199] More specifically, a gaze detection process based on the pupil-corneal reflex method is performed. Using the pupil-corneal reflex method, a gaze vector that indicates the direction (rotation angle) of the eyeball is calculated based on the pupil image and the Purkinje image included in the captured image of the eyeball, thereby detecting the user's gaze.

[0200] The display device of this embodiment may have a photoelectric conversion device having a light receiving element, and may control the display image of the display device based on information on the user's line of sight from the photoelectric conversion device.

[0201] Specifically, the display device determines a first field of view area to which the user gazes and a second field of view area other than the first field of view area based on the line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device, or may be received from an external control device. In the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than that of the first field of view area.

[0202] The display area may have a first display area and a second display area different from the first display area, and a high priority area may be determined from the first display area and the second display area based on line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device, or may be received from an external control device. The resolution of the high priority area may be controlled to be higher than the resolution of areas other than the high priority area. In other words, the resolution of an area with a relatively low priority may be lowered.

[0203] In addition, AI may be used to determine the first field of view area and the area with high priority. The AI ​​may be a model configured to estimate the angle of the line of sight and the distance to an object at the end of the line of sight from the image of the eyeball, using the image of the eyeball and the direction in which the eyeball in the image was actually looking as teacher data. The AI ​​program may be included in the display device, the photoelectric conversion device, or an external device. If included in the external device, it is transmitted to the display device via communication.

[0204] When display control is performed based on visual recognition detection, the present invention is preferably applicable to smart glasses further including a photoelectric conversion device for capturing an image of the outside world. The smart glasses can display captured outside information in real time.

[0205] The above-described embodiments can be modified as appropriate without departing from the technical concept. In addition, an example in which a part of the configuration of any of the embodiments is added to another embodiment, or an example in which a part of the configuration of any of the embodiments is replaced with another embodiment, is also included in the embodiments of the present invention.

[0206] The disclosure of this embodiment also includes the following configurations and methods.

[0207] (Configuration 1) A photoelectric conversion device comprising: a photoelectric conversion unit that generates a photon detection signal by avalanche multiplication; and a counter circuit that counts the photon detection signals output from the photoelectric conversion unit. The photoelectric conversion device comprises a first period included in one frame, and a second period included in the one frame, the second period being after the first period and not overlapping with the first period. The photoelectric conversion device is characterized in that a value counted by the counter circuit based on one of the photon detection signals in the first period is greater than a value counted by the counter circuit based on one of the photon detection signals in the second period.

[0208] (Configuration 2) The photoelectric conversion device according to configuration 1, wherein when the photoelectric conversion unit outputs a photon detection signal in the first period, a natural number N is added to the counter circuit. When the photoelectric conversion unit outputs a photon detection signal in the second period, a natural number M is added to the counter circuit, and the natural number N is greater than the natural number M.

[0209] (Configuration 3) 3. The photoelectric conversion device according to configuration 2, wherein the natural number M is 1.

[0210] (Configuration 4) 4. The photoelectric conversion device according to any one of Structures 1 to 3, wherein the first period and the second period are adjacent to each other in time.

[0211] (Configuration 5) 5. The photoelectric conversion device according to any one of configurations 1 to 4, wherein a clock signal is input to recharge the photoelectric conversion unit and put the photoelectric conversion unit into a standby state.

[0212] (Configuration 6) 6. The photoelectric conversion device according to configuration 5, wherein the first period and the second period occur after the input of the clock signal.

[0213] (Configuration 7) 7. The photoelectric conversion device according to any one of configurations 1 to 6, wherein the length of the first period is shorter than the length of the second period.

[0214] (Configuration 8) The photoelectric conversion device according to configuration 2, wherein when the photoelectric conversion unit outputs a photon detection signal during a third period, a natural number L is added to the counter circuit, and the natural number L is smaller than the natural number M.

[0215] (Configuration 9) 3. The photoelectric conversion device according to configuration 2, wherein a value according to a photon detection result of a previous recharge period is added.

[0216] (Configuration 10) 3. The photoelectric conversion device according to configuration 2, wherein a value according to the photon detection results of surrounding pixels is added.

[0217] (Configuration 11) 3. The photoelectric conversion device according to configuration 2, wherein if a photon is detected during the first period, the natural number M is not added to the counter circuit even if a photon is detected during the second period.

[0218] (Configuration 12) 12. The photoelectric conversion device according to any one of configurations 1 to 11, wherein the first period and the second period each detect up to one photon.

[0219] (Configuration 13) 3. The photoelectric conversion device according to configuration 2, wherein the natural number N or the natural number M is set in accordance with a count value of the counter circuit.

[0220] (Configuration 14) 14. The photoelectric conversion device according to any one of configurations 1 to 13, wherein the lengths of the first period and the second period are set in accordance with a count value of the counter circuit.

[0221] (Configuration 15) The photoelectric conversion device according to configuration 2, characterized in that the length of the first period and the length of the second period, or the combination of the natural number N and the natural number M, is made different between the kth frame and the k+1th frame.

[0222] (Configuration 16) The photoelectric conversion unit has a pixel array in which a plurality of pixels are arranged in a plurality of rows and a plurality of columns, and the pixel array has a first pixel group and a second pixel group, and the first pixel group and the second pixel group have different lengths of the first period and the second period, or different combinations of the natural number N and the natural number M.

[0223] (Configuration 17) 17. A photoelectric conversion system comprising: the photoelectric conversion device according to any one of configurations 1 to 16; and a signal processing unit that generates an image using a signal output from the photoelectric conversion device.

[0224] (Configuration 18) 18. The photoelectric conversion system according to configuration 17, wherein nonlinear correction is performed.

[0225] (Configuration 19) A moving object including the photoelectric conversion device according to any one of configurations 1 to 16, characterized in that the moving object has a control unit that controls the movement of the moving object using a signal output by the photoelectric conversion device. [Explanation of symbols]

[0226] 301 Avalanche Photodiode 302 Quench element 303 Waveform shaping circuit 305 Counter Circuit 311 Control circuit

Claims

1. a photoelectric conversion unit that generates a photon detection signal by avalanche multiplication; a counter circuit that counts the photon detection signal output from the photoelectric conversion unit; a first period included in one frame; a second period included in the one frame and following the first period and not overlapping with the first period; A value counted by a counter circuit based on one of the photon detection signals during the first period is a value greater than a value counted by a counter circuit based on one of the photon detection signals during the second period.

2. when the photoelectric conversion unit outputs a photon detection signal during the first period, a natural number N is added to the counter circuit; when the photoelectric conversion unit outputs a photon detection signal during the second period, a natural number M is added to the counter circuit; 2. The photoelectric conversion device according to claim 1, wherein the natural number N is greater than the natural number M.

3. 3. The photoelectric conversion device according to claim 2, wherein the natural number M is 1.

4. 2. The photoelectric conversion device according to claim 1, wherein the first period and the second period are adjacent to each other in time.

5. 2. The photoelectric conversion device according to claim 1, further comprising an input of a clock signal that recharges the photoelectric conversion unit and puts the photoelectric conversion unit into a standby state.

6. 6. The photoelectric conversion device according to claim 5, wherein the first period and the second period occur after the input of the clock signal.

7. 2. The photoelectric conversion device according to claim 1, wherein the length of the first period is shorter than the length of the second period.

8. when the photoelectric conversion unit outputs a photon detection signal during a third period, a natural number L is added to the counter circuit; 3. The photoelectric conversion device according to claim 2, wherein the natural number L is smaller than the natural number M.

9. 3. The photoelectric conversion device according to claim 2, further comprising an additional value corresponding to a photon detection result in a previous recharge period.

10. 3. The photoelectric conversion device according to claim 2, further comprising an additional value corresponding to the photon detection results of surrounding pixels.

11. 3. The photoelectric conversion device according to claim 2, wherein if a photon is detected during the first period, the natural number M is not added to the counter circuit even if a photon is detected during the second period.

12. 2. The photoelectric conversion device according to claim 1, wherein the first period and the second period each detect up to one photon.

13. 3. The photoelectric conversion device according to claim 2, wherein the natural number N or the natural number M is set in accordance with a count value of the counter circuit.

14. 2. The photoelectric conversion device according to claim 1, wherein the lengths of the first period and the second period are set in accordance with a count value of the counter circuit.

15. The photoelectric conversion device according to claim 2, characterized in that the length of the first period and the length of the second period, or the combination of the natural number N and the natural number M, is made different between the kth frame and the k+1th frame.

16. The photoelectric conversion unit has a pixel array in which a plurality of pixels are arranged in a plurality of rows and a plurality of columns, the pixel array having a first group of pixels and a second group of pixels; The photoelectric conversion device according to claim 2 , wherein the first pixel group and the second pixel group have different lengths of the first period and the second period, or different combinations of the natural number N and the natural number M.

17. The photoelectric conversion device according to any one of claims 1 to 16, a signal processing unit that generates an image using a signal output from the photoelectric conversion device.

18. 18. The photoelectric conversion system according to claim 17, wherein nonlinear correction is performed.

19. A moving object including the photoelectric conversion device according to any one of claims 1 to 16, A moving body comprising: a control unit that controls the movement of the moving body using a signal output from the photoelectric conversion device.

Images