Photoelectric conversion device and photoelectric conversion system having photoelectric conversion device

Abstract

To acquire a larger amount of signal information while maintaining expansion of a dynamic range.SOLUTION: A photoelectric conversion device has: an exposure control unit that generates a signal defining a plurality of second exposure periods that are included in a first exposure period corresponding to one frame and are shorter than the first exposure period; a timing generation unit that generates a pulse signal defining time information in the second exposure period; and a measuring unit that, on the basis of the pulse signal generated by the timing generation unit, measures the number of the pulse signals after detection of a first photon in the second exposure period.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device and a photoelectric conversion system having the photoelectric conversion device. [Background technology]

[0002] Patent Document 1 discloses a configuration in which the time until a counter is saturated for each pixel is measured, and the amount of light is estimated by extrapolation from the time and count value. This configuration makes it possible to expand the dynamic range. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 179928 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the configuration of Patent Document 1, the counter stops due to saturation, and signal information is lost after the counter stops. This can cause, for example, artifacts when shooting a moving object or brightness variations when shooting a flickering light source. [Means for solving the problem]

[0005] A photoelectric conversion device comprising: a photoelectric conversion element that receives photons; an exposure control unit that generates a signal that defines a plurality of second exposure periods that are included in a first exposure period corresponding to one frame and are shorter than the first exposure period; a timing generation unit that generates a pulse signal that defines time information within the second exposure period; and a measurement unit that measures the number of the pulse signals after the first detection of a photon during the second exposure period based on the pulse signal generated by the timing generation unit. Effect of the Invention

[0006] According to the present invention, it is possible to obtain more signal information than in Patent Document 1. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 shows an example of the configuration of a pixel circuit. [Diagram 2] A diagram explaining the relationship between detection time and the number of incident photons. [Diagram 3] A diagram explaining the relationship between detection time and the number of incident photons. [Figure 4] Diagram explaining the probability of detecting a photon [Diagram 5] A diagram explaining the expected value and input / output characteristics with respect to the number of incident photons [Figure 6] FIG. 1 is a diagram illustrating a photoelectric conversion device according to a first embodiment. [Figure 7] FIG. 1 is a diagram illustrating a photoelectric conversion device according to a first embodiment. [Figure 8] FIG. 1 is a diagram illustrating a photoelectric conversion device according to a first embodiment. [Figure 9] FIG. 1 is a diagram illustrating a photoelectric conversion device according to a first embodiment. [Figure 10] FIG. 1 is a diagram for explaining a pixel circuit and driving of a photoelectric conversion device according to a first embodiment; [Figure 11] FIG. 1 is a diagram illustrating an operation sequence of a photoelectric conversion device according to a first embodiment. [Figure 12] A diagram explaining the cases where the clock signal that defines the time information is at equal and non-equal intervals. [Figure 13] A diagram explaining the clock signal that specifies time information and its input / output characteristics. [Figure 14] FIG. 10 is a diagram for explaining a pixel circuit of a photoelectric conversion device according to a second embodiment; [Figure 15] FIG. 11 is a diagram for explaining an operation sequence of a photoelectric conversion device according to a second embodiment. [Figure 16] FIG. 13 is a diagram illustrating a pixel circuit of a photoelectric conversion device according to a third embodiment. [Figure 17] FIG. 11 is a diagram for explaining an operation sequence of a photoelectric conversion device according to a third embodiment. [Figure 18] FIG. 13 is a diagram for explaining the effect of the photoelectric conversion device according to the third embodiment. [Figure 19] FIG. 13 is a diagram for explaining a pixel circuit of a photoelectric conversion device according to a fourth embodiment. [Figure 20] FIG. 13 is a diagram for explaining a pixel circuit of a photoelectric conversion device according to a fourth embodiment. [Figure 21] FIG. 13 is a diagram for explaining an operation sequence of a photoelectric conversion device according to a fourth embodiment. [Figure 22] FIG. 13 is a diagram for explaining an operation sequence of a photoelectric conversion device according to a fourth embodiment. [Diagram 23] FIG. 13 is a diagram illustrating a photoelectric conversion device according to a fifth embodiment. [Figure 24] FIG. 13 is a diagram for explaining a pixel circuit of a photoelectric conversion device according to a fifth embodiment. [Diagram 25] FIG. 13 is a diagram for explaining a signal processing block of a photoelectric conversion device according to a sixth embodiment. [Figure 26] FIG. 13 is a schematic diagram according to a seventh embodiment. [Figure 27] FIG. 13 is a functional block diagram of a photoelectric conversion system according to an eighth embodiment. [Figure 28] FIG. 13 is a functional block diagram of a photoelectric conversion system according to a ninth embodiment. [Figure 29] FIG. 23 is a functional block diagram of a photoelectric conversion system according to a tenth embodiment. [Diagram 30] FIG. 23 is a functional block diagram of a photoelectric conversion system according to an eleventh embodiment. [Diagram 31] FIG. 23 is a functional block diagram of a photoelectric conversion system according to a twelfth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The following embodiments are intended to embody the technical ideas of the present invention, and are not intended to limit the present invention. The sizes and positional relationships of the components shown in the drawings may be exaggerated to clarify the description. In the following description, the same components may be designated by the same reference numerals, and the description may be omitted.

[0009] In the following description, terms indicating specific directions or positions (for example, "upper", "lower", "right", "left" and other terms including these terms) are used as necessary. The use of these terms is for the purpose of facilitating understanding of the embodiments with reference to the drawings, and the technical scope of the present invention is not limited by the meaning of these terms.

[0010] In this specification, a plan view means a view perpendicular to a major surface of a semiconductor layer, and is synonymous with a top view, and a cross-sectional view means a view perpendicular to a plane perpendicular to the major surface of the semiconductor layer.

[0011] In the following description, the anode of the photodiode (PD) is set to a fixed potential, and a signal is taken from the cathode side. Therefore, the first conductive type semiconductor region having charges of the same polarity as the signal charge as the majority carrier is an N-type semiconductor region, and the second conductive type semiconductor region having charges of a different polarity from the signal charge as the majority carrier is a P-type semiconductor region. The cathode of the PD may be set to a fixed potential, and the signal may be taken from the anode side. In this case, the first conductive type semiconductor region having charges of the same polarity as the signal charge as the majority carrier is a P-type semiconductor region, and the second conductive type semiconductor region having charges of a different polarity from the signal charge as the majority carrier is an N-type semiconductor region. In the following, a case where one node of the PD is set to a fixed potential will be described, but the potentials of both nodes may be configured to fluctuate.

[0012] (Basic configuration) 1 is a diagram showing a schematic configuration of a photoelectric conversion device according to this embodiment. The photoelectric conversion device 100 includes a photoelectric conversion element 1, an exposure control unit 2, a timing generation unit 3, and a measurement unit 4.

[0013] The photoelectric conversion element 1 detects incident photons and converts them into an electrical signal. The photoelectric conversion element 1 may be a linear mode avalanche photodiode (APD) operated near the breakdown voltage, or a single photon avalanche photodiode (SPAD) operated in Geiger mode.

[0014] The exposure control unit 2 generates a signal that defines an exposure period (first exposure period) corresponding to one frame. The "exposure period corresponding to one frame" is also referred to as "one frame period." The exposure control unit 2 also generates a signal that defines exposure periods (second exposure periods) corresponding to multiple subframes included in the exposure period corresponding to one frame. The "exposure period corresponding to the subframes" is also referred to as "subframe period."

[0015] The timing generating section 3 generates a pulse signal for defining time information within a subframe period.

[0016] The measurement unit 4 receives as input a signal from the photoelectric conversion element 1, a control signal from the exposure control unit 2, and a control signal from the timing generation unit 3. From these signals, the measurement unit 4 measures a value corresponding to the detection time of the first detected photon from the start of the subframe period. The measurement unit may use a general TDC circuit as a circuit for measuring time, or may use other methods for measurement.

[0017] FIG. 2A is a diagram showing the exposure period and the detection time, and FIG. 2B is a diagram showing the relationship between the detection time and the number of incident photons.

[0018] Here, the exposure period is T aac , the time from the start of the exposure period to the detection of a photon is T detect Then, the number of incident photons during the exposure period is N ph is N ph =(T aac / T detect ) Therefore, if the exposure period and detection time are known, it is possible to estimate the number of incident photons. In other words, imaging is possible because the number of incident photons during the exposure period is known without counting the number of incident photons themselves.

[0019] Figures 3(A) and (B) show the case where the exposure shown in Figure 2 is performed multiple times. Here, each measurement period is called a subframe period, and the subframe period is repeated N times. One frame period is the sum of subframes 1 to N. In this case, the average time that photons are detected in the subframes during one frame period is called T ave Then, the number of incident photons in one frame is N ph is N ph =(T aac / T ave ) × N. Figure 3(C) shows the relationship between the average detection time and the number of incident photons.

[0020] 3, the explanation focuses on the average time of photon detection, but the number of incident photons can also be estimated from the integrated time of photon detection. In addition, the averaging and integration processes may be performed within the photoelectric conversion device, or the signal for each subframe may be output to the outside of the photoelectric conversion device and may be performed by an external processing circuit.

[0021] The above-mentioned relational expression between the number of incident photons and the detection time is suitable when the minimum unit of the detection time is sufficiently smaller than the detection time of the photon, but when the minimum unit of the detection time and the detection time of the photon are comparable, an effect may occur. For example, even within the minimum unit of the detection time, the number of incident photons assumed to be detected may change depending on the timing of detection, so an error may occur in the above-mentioned relational expression. A situation in which such an error cannot be ignored is said to be influenced by the discreteness of the detection time.

[0022] In Figure 4, we explain how to estimate the number of photons when the detection time is considered to be discrete. Figure 4 is a conceptual diagram explaining the probability of detecting a photon. When the detection time is 1, 2, ..., the probability of detecting a photon at a certain time is an exponential distribution (f(t, λ) = λe -λt , λ is the number of events per unit time). In this case, λ is the number of incident photons N ph, exposure time, and unit time (the smallest unit of detection time). Specifically, λ is (number of incident photons / exposure time) × (unit time). For example, if the number of incident photons is 1, the exposure time is 1000, and the unit time is 1, then (1 / 1000) × 1, and λ is 0.001. According to this exponential distribution, when the horizontal axis is the detection time, the photon detection probability can be expressed by the graph shown in Figure 4(A). Here, the solid line indicates the case of high illuminance, the dashed line indicates the case of medium illuminance, and the dashed line indicates the case of low illuminance. The cumulative detection probability for an interval is F(t, λ) = 1-e -λt and can be expressed by the graph shown in FIG. 4(B). Furthermore, the interval cumulative detection probability obtained by accumulating the detection probability for each detection interval (0 to 1, 1 to 2, ...) is expressed as F'(n, λ) = F(t n ,λ)-F(t n+1 , λ) and can be represented by the graph shown in Figure 4(C).

[0023] By performing this calculation, the photon detection probability can be calculated for each detection interval. The sum of the products of the cumulative detection probability and the corresponding detection times (expected value E) is

[0024]

number

[0025] This expected value E can be calculated as T ave From the above calculations, even if the detection time becomes discrete, the number of incident photons N ph It is possible to estimate

[0026] Figure 5(A) shows a specific example of the expected value for the number of incident photons. When the exposure period is 1000 and the unit time is 1, the output E (expected value of detection time) and output E' (expected value of exposure time - detection time) obtained when the input (number of incident photons) is changed to 1, 10, 100, and 1000 are shown.

[0027] 5B and 5C show graphs of the input / output characteristics for the output E and the output E'. The output E is the number of incident photons N ph The larger the number of incident photons, the smaller the output E'. ph The curve shows that the larger the output E', the larger the

[0028] In general, an imaging device outputs a signal such that the output increases as the number of incident photons increases. Therefore, in consideration of the signal processing in the subsequent stages, it is preferable to output output E' instead of output E.

[0029] According to this configuration, the number of incident photons during the exposure period can be estimated from the timing of photon detection, so that the dynamic range can be expanded beyond the number of detected photons. In addition, the length of the subframe period can be set by the photon exposure control unit, so that the counter can be adjusted so as not to saturate during one frame period. Therefore, by avoiding the counter stopping during one frame period while securing the dynamic range, loss of signal information can be prevented.

[0030] It should be noted that, by utilizing the above-mentioned technical concept, a configuration may be additionally adopted in which the counter is stopped during one frame period.

[0031] Each embodiment will be described below.

[0032] (First embodiment) FIG. 6 is a diagram showing the configuration of a stacked photoelectric conversion device 100 according to this embodiment. The photoelectric conversion device 100 is configured by stacking and electrically connecting two substrates, a sensor substrate 11 (first substrate) and a circuit substrate 21 (second substrate). The sensor substrate 11 has a plurality of photoelectric conversion elements 1. The circuit substrate 21 has a circuit of a signal processing unit 103. In the following, the sensor substrate 11 and the circuit substrate 21 will be described as diced chips, but are not limited to chips. For example, each substrate may be a wafer. Also, each substrate may be stacked in a wafer state and then diced, or may be chipped and then stacked and bonded.

[0033] A pixel region 12 is disposed on the sensor substrate 11, and a circuit region 22 that processes signals detected in the pixel region 12 is disposed on the circuit substrate 21.

[0034] 7 is a diagram showing an example of the arrangement of the sensor substrate 11. Pixels 101, each having a photoelectric conversion element 1 including an avalanche photodiode (APD), are arranged in a two-dimensional array in a plan view to form a pixel region 12.

[0035] 8 is a configuration diagram of the circuit board 21. It has a signal processing unit 103 that processes charges photoelectrically converted by the photoelectric conversion element 1, a column circuit 112, a control pulse generating unit 115, a horizontal scanning circuit unit 111, a signal line 113, a vertical scanning circuit unit 110, a control line 116, and a control line 117.

[0036] The pixel 101 having the photoelectric conversion element 1 in Fig. 7 and the signal processing unit 103 in Fig. 8 are electrically connected via a connection wiring provided for each pixel. The pixel 101 and the signal processing unit 103 are sometimes referred to as a pixel circuit.

[0037] 8 receives a control pulse supplied from a control pulse generating unit 115 and supplies the control pulse to each pixel. The vertical scanning circuit unit 110 uses logic circuits such as a shift register and an address decoder. The signal output from the photoelectric conversion element 1 is processed by a signal processing unit 103.

[0038] The signal processing unit 103 is provided with a counter, a memory, etc., and the memory holds digital values.

[0039] The horizontal scanning circuit unit 111 inputs a control pulse for sequentially selecting each column to the signal processing unit 103 in order to read out the signal from the memory of each pixel in which the digital signal is held.

[0040] A signal is output to the signal line 113 from the signal processing unit 103 of the pixel selected by the vertical scanning circuit unit 110 for the selected column.

[0041] The signal output to the signal line 113 is output via an output circuit 114 to a recording unit or a signal processing unit outside the photoelectric conversion device 100 .

[0042] In Fig. 7, the photoelectric conversion elements in the pixel region may be arranged one-dimensionally. The effect of this embodiment can be obtained even if there is only one pixel, and the case where there is only one pixel is also included. The function of the signal processing unit does not necessarily need to be provided for each photoelectric conversion element, and for example, one signal processing unit may be shared by multiple photoelectric conversion elements and signal processing may be performed sequentially.

[0043] 7 and 8, a plurality of signal processing units 103 are arranged in a region overlapping the pixel region 12 in a planar view. A vertical scanning circuit unit 110, a horizontal scanning circuit unit 111, a column circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a planar view. In other words, the sensor substrate 11 has the pixel region 12 and a non-pixel region arranged around the pixel region 12, and the vertical scanning circuit unit 110, the horizontal scanning circuit unit 111, the column circuit 112, the output circuit 114, and the control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a planar view.

[0044] Fig. 9 is an example of a block configuration of a pixel array, a timing generation unit, and a measurement unit. The exposure control unit 2 and the timing generation unit 3 can be included in the vertical scanning circuit unit 110 and the control pulse generation unit 115 shown in Fig. 8. The pixel circuit includes a photoelectric conversion element 1 and a measurement unit 4.

[0045] The measurement unit 4 is composed of a signal processing circuit 201, a timing judgment circuit 202, and a counter circuit 203. The signal processing circuit 201 is, for example, a waveform shaping circuit, or a resistor or switch provided between the voltage applied to the photoelectric conversion element and the photoelectric conversion element. The signal processing unit 103 shown in Fig. 8 corresponds to the measurement unit 4 (signal processing circuit 201, timing judgment circuit 202, counter circuit 203) shown in Fig. 9.

[0046] The exposure control unit 2 outputs a pulse signal called P_PCLK, which is input to the signal processing circuit 201. The timing generation unit 3 outputs a pulse signal called P_TCLK, which is input to the timing determination circuit 202.

[0047] FIG. 10(A) shows an example of a pixel circuit configuration. The photoelectric conversion element 1 is an APD, which generates a pair of charges according to incident light by photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the photoelectric conversion element 1. A voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the photoelectric conversion element 1. A reverse bias voltage (a voltage equal to or higher than the breakdown voltage) that causes the photoelectric conversion element 1 to perform avalanche multiplication is supplied to the anode and cathode. By supplying such a voltage, the charges generated by the incident light undergo avalanche multiplication, generating an avalanche current.

[0048] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the potential difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the potential difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. For example, the voltage VL (first voltage) is -30V, and the voltage VH (second voltage) is 1V. An APD may operate in either linear mode or Geiger mode.

[0049] The signal processing circuit 201 includes a quench element 204 and a waveform shaping circuit 205. The quench element 204 is connected to a power supply that supplies a voltage VH and the photoelectric conversion element 1. The quench element 204 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and suppresses the voltage supplied to the photoelectric conversion element 1 to suppress avalanche multiplication (quench operation). The quench element 204 also has a function of returning the voltage supplied to the photoelectric conversion element 1 to the voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation). FIG. 10(A) shows an example in which a switch such as a transistor is disposed between the power supply and the photoelectric conversion element 1 to switch the electrical connection. The resistance of the quench element 204 may be a wiring resistance, and it is also possible to omit the resistance from the equivalent circuit diagram.

[0050] The waveform shaping circuit 205 shapes the potential change of the cathode of the photoelectric conversion element 1 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping circuit 205. Although an example using one inverter as the waveform shaping circuit 205 is shown in Fig. 14, a circuit in which a plurality of inverters are connected in series may be used, or other circuits having a waveform shaping effect may be used.

[0051] The timing determination circuit 202 is connected to a waveform shaping circuit 205 and a counter circuit 203. A signal output from the waveform shaping circuit 205 and P_TCLK supplied from outside the pixel are input, and a signal is output to the counter circuit 203 according to a combination of these signals. An example of this is an AND circuit.

[0052] FIG. 10B shows the relationship between P_PCLK, V_ph (cathode potential of the photoelectric conversion element 1), and P_ph (output from the waveform shaping circuit 205) in clock recharge driving. When P_PCLK transitions from L level to H level at time t1 and the switch is turned on, the cathode terminal of the photoelectric conversion element 1 is electrically connected to the power supply voltage, and a reverse bias is applied to the photoelectric conversion element 1. That is, when P_PCLK becomes H level and the switch is turned on, it is in charge mode. Since the charge mode is repeated multiple times, it is also called recharge mode. When the reverse bias is applied, the cathode potential V_ph rises, and when it exceeds the judgment threshold, the output P_ph transitions from H level to L level. After that, P_PCLK transitions from H level to L level, and the photoelectric conversion element 1 enters a standby mode in which it waits for the incidence of photons. When a photon is incident at time t2, the cathode potential V_ph drops, and when it exceeds the decision threshold at time t3, the output P_ph transitions from L level to H level. After that, at time t4, P_PCLK transitions again from L level to H level, the switch turns on, and the above operation is repeated.

[0053] According to this clock recharge driving, even if many photons are incident on the standby mode, the output signal can be counted at least once, and therefore this driving is effective as a measure against pile-up.

[0054] 10C is a truth table showing the output P_sig for the inputs P_ph and P_TCLK to the timing judgment circuit 202. 0 indicates the L level, and 1 indicates the H level. Since the timing judgment circuit 202 is composed of an AND circuit, the output P_sig becomes the H level only when the inputs P_ph and P_TCLK are both at the H level.

[0055] FIG. 11 is a drive timing diagram explaining the exposure period and the pulse signal. One frame period is made up of N subframes. The start and end of the subframe are defined by P_PCLK. One subframe contains M pulses of P_TCLK, which are input to the timing determination circuit 202 so that pulses of the same period are repeated for each subframe. In FIG. 11, P_TCLK is non-equidistant and is approximately proportional to a logarithm. As will be described later, P_TCLK may be equally spaced. It may also be non-equidistant pulses approximately proportional to the reciprocal.

[0056] FIG. 11 shows a concrete example of the concept of FIG. 3 described above. The P_PCLK pulse is the start of a subframe, and the interval between subframes is T acc When a photon is incident, P_ph transitions from L level to H level, and when a P_PCLK pulse is input, it transitions from H level to L level due to the recharge operation of the photoelectric conversion element 1. P_TCLK is a pulse signal for defining time information within a subframe.

[0057] According to the configuration of the timing judgment circuit 202 described in FIG. 10, the number of P_TCLKs after the timing at which a photon is first detected in a subframe is counted, so the signal input to the counter becomes P_sig and the counter counts 3. This count value corresponds to the incidence of a photon in the time window from M-4 to M-3 of P_TCLK. In this way, when a photon is incident early in a subframe, the counter count value becomes large, and when a photon is incident late in a subframe, the counter count value becomes small. In other words, it is possible to measure a value corresponding to the time from the start of a subframe to when a photon is first detected, based on P_TCLK.

[0058] By repeating this counting N times and adding up the count values, a count value equivalent to the detection time expectation E' is obtained. As a result, as explained above, it is possible to estimate the number of incident photons from the expectation E'.

[0059] According to this configuration, the number of incident photons during the exposure period can be estimated from the timing of photon detection, so that the dynamic range can be expanded beyond the number of detected photons. In addition, the length of the subframe period can be set by the photon exposure control unit, so that the counter can be adjusted so as not to saturate during one frame period. Therefore, by avoiding the counter stopping during one frame period while securing the dynamic range, loss of signal information can be prevented.

[0060] Incidentally, in conventional clock recharge driving without the timing judgment circuit 202, the number of subframe periods is the maximum number that can be counted in one frame period. For example, when the number of subframes is N, the maximum count value is N, and up to N photons can be detected. Also, the dynamic range (defined as the maximum output value) is N. Basically, power consumption is proportional to the number of detections, so if you try to reduce the number of detections to reduce power consumption, the dynamic range will decrease, and if you try to increase the dynamic range, the power consumption will increase. In other words, there is a trade-off between the dynamic range and power consumption.

[0061] On the other hand, in the photoelectric conversion device according to the present embodiment that includes the timing judgment circuit 202, the estimated number of photons corresponds to the dynamic range. Although the power consumption is proportional to the number of detections as in the conventional case, the estimated number of photons does not depend on the number of photon detections, and therefore the trade-off between the dynamic range and the power consumption can be eliminated.

[0062] (Evenly spaced and non-evenly spaced P_TCLK) FIG. 12 is a conceptual diagram comparing the cases where P_TCLK is equally spaced and non-equally spaced. The upper part of FIG. 13(A) shows an example where 16 equally spaced pulses occur within one subframe. It can be said that equally spaced pulses correspond to real time. The lower part shows an example where non-equally spaced pulses are taken as the approximate logarithm (base 2) of real time. The reason why it is said to be "approximately" here is that the logarithm of 1 when the base is 2 is 0, and the logarithm of 2 when the base is 2 is 1, but in FIG. 13(A), 1 is added to make them 1 and 2, respectively, in order to compare with equally spaced pulses. Such non-equally spaced pulse intervals are expressed as being set so that the pulse space is logarithmically compressed in real space.

[0063] Figure 13(B) is a graph showing the relationship between the number of incident photons (input) and the count value (output) for equally spaced pulses. As shown in this figure, when compared at the same number of incident photons (input), the output of non-equally spaced pulses is smaller than the output of equally spaced pulses. In other words, non-equally spaced pulses can determine the same number of incident photons with a smaller count value. For this reason, non-equally spaced pulses can reduce the scale of the counter circuit, and the pixel circuit area can be reduced. On the other hand, with equally spaced pulses, the average interval between pulses can be made smaller, so the S / N is relatively higher than with non-equally spaced pulses.

[0064] Second embodiment In this embodiment, a configuration example in which a pixel circuit different from that in the first embodiment is provided will be described.

[0065] 14(A) and the pixel circuit shown in FIG 10 in that the switch for inputting P_PCLK provided between the power supply voltage and the photoelectric conversion element 1 is eliminated, and a latch circuit 206 is added to the signal processing circuit 201. P_PCLK is configured to be input to the latch circuit 206. Even in the case shown in FIG 14, since it is sufficient to provide an element that functions as a resistor as the quench element 204, not only a metal resistor but also a transistor or the like may be used as the quench element 204.

[0066] Fig. 14(B) shows the change in the cathode potential V_ph of the photoelectric conversion element 1 shown in Fig. 14(A). The difference from Fig. 10(B) is that after the cathode potential drops due to the incidence of photons, the voltage autonomously returns to the initial state. This pixel operation is called passive operation.

[0067] Fig. 15 is a drive timing diagram according to this embodiment. The difference from Fig. 11 is that P_ph repeats High and Low multiple times during one subframe in response to the incidence of photons. However, P_out output from the latch circuit 206 and input to the timing judgment circuit coincides with P_ph in Fig. 11, so that the resulting output P_sig is the same. In this way, the same effect as in the first embodiment can be obtained with a passively driven ADP element as well.

[0068] That is, according to this configuration, the number of incident photons during the exposure period can be estimated from the timing of photon detection, so that the dynamic range can be expanded beyond the number of detected photons. Also, since the length of the subframe period can be set by the photon exposure control unit, it is possible to adjust the counter so that it does not saturate during one frame period. Therefore, by avoiding the counter stopping during one frame period while securing the dynamic range, it is possible to prevent loss of signal information.

[0069] Furthermore, with this configuration, the estimated number of photons corresponding to the dynamic range does not depend on the number of photon detections corresponding to the power consumption, so that the trade-off between the dynamic range and the power consumption can be eliminated. Third embodiment In this embodiment, a configuration example in which a pixel circuit different from that of the above embodiment is provided will be described.

[0070] The difference between the pixel circuit of this embodiment shown in Fig. 16 and the pixel circuit shown in Fig. 10 is that a signal selection circuit 207 and a signal holding circuit 208 are added. By adding these circuits, it becomes possible to switch the signal input to the counter circuit in the M subframe between P_out and P_ph by referring to information on the presence or absence of photons incident in the M-1 subframe. Since the output of the subsequent subframe is changed depending on the result of the previous subframe, this circuit is referred to here as a time correlation filter.

[0071] 17 shows the operation sequence of this embodiment. In the M subframe, since no photons are incident, P_ph is at L level. This L level information is held by the signal holding circuit 208.

[0072] Next, when P_PCLK, which specifies the start of the M+1 subframe, transitions from L level to H level, an L level signal is output from the signal holding circuit 208, and the L level signal is input to the signal selection circuit 207. As a result, the signal selection circuit 207 is configured to output P_ph as P_sig, rather than outputting P_out as P_sig. When a photon is incident in the M+1 subframe, P_ph transitions from L level to H level, and therefore P_sig also transitions from L level to H level. Furthermore, since P_ph transitions from L level to H level, the H level information is held in the signal holding circuit 208.

[0073] Next, when P_PCLK, which specifies the start of the M+2 subframe, transitions from L level to H level, P_ph transitions from H level to L level, and P_sig also transitions from H level to L level. Furthermore, due to the transition of the level of P_PCLK, a H level signal is output from the signal holding circuit 208, and the signal selection circuit 207 is configured to output P_out as P_sig. In other words, the signal selection circuit 207 is configured to be able to output a signal corresponding to a pulse signal that specifies time information within the M+2 subframe period. When a photon is incident in the M+2 subframe, P_ph transitions from L level to H level, and therefore the P_out signal is output as P_sig.

[0074] Figure 18 shows an operation sequence to explain the effect of the time correlation filter. The time correlation filter is effective when dealing with signals with about one count or less per subframe. This is because as the incidence frequency of photons decreases, the probability of a photon incidence becomes constant regardless of time based on the exponential distribution function. In other words, the probability of a photon entering each time window is almost the same.

[0075] In Fig. 18, one photon is incident in two subframes. In this way, when a photon is incident at the timing shown in Fig. 18, P_sig counts 1 when the time correlation filter is present, and P_sig counts 3 when the time correlation filter is not present.

[0076] In this way, when a signal with about 1 count or less per subframe is handled, if a configuration without a time correlation filter is adopted, the number of incident photons will be counted as higher than the actual number of incident photons, and errors in low illuminance areas will become large. In other words, this will be a factor in increasing noise. In contrast, in this embodiment, when a signal with about 1 count or less per subframe is handled, only 1 count is performed unless a signal is detected continuously, so errors can be suppressed and factors that increase noise can be reduced. Therefore, the signal-to-noise ratio in low output areas can be improved. Furthermore, since the dark count rate (DCR) in the dark state is also a signal with about 1 count or less per subframe, it can be reduced by the same logic by adopting this time correlation filter.

[0077] In the above, an example of a circuit that outputs one count unless a signal is detected continuously has been described, but the circuit may also output a predetermined value, or the predetermined value may be a circuit that outputs a value of one count or more.

[0078] (Fourth embodiment) The difference between the pixel circuit of this embodiment shown in Fig. 19 and the pixel circuit shown in Fig. 16 is that an intra-pixel pulse generating circuit 209 is added. The intra-pixel pulse generating circuit 209 is a circuit for generating P_PCLK and P_TCLK from one reference clock signal (P_BCLK in this case).

[0079] When measuring the time from the start of a subframe until a photon arrives, it is necessary to prevent a timing shift between P_PCLK, which determines the start of the subframe, and P_TCLK, which measures a value corresponding to the elapsed time from the start of the subframe. If the timing of these signals shifts, the measurement value error increases and noise performance deteriorates. With the configuration according to this embodiment, it is no longer necessary to consider the influence of the outside of the pixel circuit, so it is possible to suppress the timing shift.

[0080] Fig. 20 is a modification of Fig. 19. For the trigger signal input to the signal holding circuit 208, P_TRG can be used.

[0081] Fig. 21(A) shows a specific example of the intra-pixel pulse generating circuit 209. Fig. 21(B) shows the operation sequence of the intra-pixel pulse generating circuit 209. In Fig. 30(A), P_BCLK is a reference clock signal, and is input to an AND circuit. P_BCLK becomes P_TCLK, which measures a value corresponding to the elapsed time from the start of a subframe. P_TRG, which is a trigger signal, is input to the AND circuit, and when P_TRG is at a high level, P_PCLK is output. With this circuit configuration, P_PCLK and P_TCLK can be generated from one reference clock signal, suppressing timing discrepancies.

[0082] Fig. 22 is a modification of Fig. 21. As shown in Fig. 22(A), the trigger signal P_TRG is inverted and input to an AND circuit. Therefore, P_PCLK is output when P_TRG is at H level, and P_TCLK is output when P_TRG is at L level. As a result, P_PCLK and P_TCLK do not overlap, and an interval period is provided between them.

[0083] When P_PCLK and P_TCLK overlap, depending on the relative positions of the two pulses, it is anticipated that the last P_TCLK may not be counted correctly, which can be an issue. Therefore, this issue can be resolved by defining the start of one subframe period as the P_PCLK pulse, i.e., the (M+1)th pulse of P_BLK, and the end as the last P_TCLK, i.e., the Mth pulse of P_BLK.

[0084] Fifth embodiment A fifth embodiment will be described with reference to Figures 23 and 24. This embodiment differs from the first embodiment in that it has three laminated layers.

[0085] In FIG. 23, a third substrate 31 (third substrate) and a second circuit region 32 are added.

[0086] Fig. 24 shows a conceptual diagram of pixel circuit layout in a three-layer stack. Compared to the pixel circuit in Fig. 10, a second timing judgment circuit 302 and a second counter circuit 303 are added, and these circuits are arranged on a third substrate 31.

[0087] The second timing judgment circuit 302 is configured to receive P_TCLK2 and the output from the waveform shaping circuit 205. In FIG. 24, it is assumed that the vertical scanning circuit section 110 and the control pulse generating section 115 are provided on the second substrate 21, and therefore, a configuration is shown in which P_TCLK2 is input to the third substrate 31 from a circuit provided on the second substrate 21. Since the third substrate 31 does not have the quench element 204 and the waveform shaping circuit 205, more space can be secured than in the second substrate 21. Therefore, it is also possible to provide the vertical scanning circuit section 110 and the control pulse generating section 115 on the third substrate 31. In this case, P_PCLK and P_TCLK1 may be configured to be input from the third substrate 31 to the second substrate 21.

[0088] In this embodiment, since three layers are laminated, multiple circuits can be easily parallelized, and high performance can be achieved. Specifically, since the timing judgment circuits 202 and 302 are provided in parallel, two outputs with different count values ​​can be obtained from one photon detection signal using P_TCLK1, 2 having different signal waveforms. The merits of various characteristics (dynamic range, SN ratio, appropriate exposure, power consumption, etc.) change depending on the number of pulses and pulse interval of P_TCLK. Therefore, if there are two outputs with different characteristics, it is possible to select the optimal output according to the imaging scene, and to generate an image with better quality by combining them.

[0089] For example, when performing measurement with one P_TCLK as in the first embodiment, if a scene in which the amount of light changes rapidly within one frame is captured, a problem is expected in that measurement errors will increase. This is because the difference in the measurement values ​​of the photon detection timing of each subframe will be large compared to when average light is incident within one frame.

[0090] One method to solve this problem is to acquire signals in parallel that can estimate the change in light intensity within one frame, and then make corrections. Specifically, this can be achieved by configuring P_TCLK2 to count only the first pulse, so that only incident photons that are detected early are counted.

[0091] Sixth embodiment A photoelectric conversion system using the photoelectric conversion device according to each of the above-mentioned embodiments will be described with reference to Fig. 25. Fig. 25 is a block diagram showing a schematic configuration of the photoelectric conversion system according to this embodiment.

[0092] The processing device according to this embodiment includes a control unit 401 , a timing adjustment unit 402 , an image acquisition unit 403 , a readout unit 404 , a gain adjustment unit 405 , a non-linear correction unit 406 , a defect correction unit 407 , a data compression unit 408 , and a memory unit 409 .

[0093] The image acquisition unit 403 is, for example, a pixel circuit, and the readout unit 404 is, for example, provided after the counter circuit 203. The control unit 401 may be a control unit inside the photoelectric conversion device, or may be outside the photoelectric conversion device. The image acquisition unit 403 is controlled by a timing adjustment unit 402 controlled by the control unit 401. The image data generated by the image acquisition unit is input to a storage unit 409 after being subjected to a correction process. Note that the order of the correction processes is not limited to the order shown in FIG. 25.

[0094] The gain adjustment unit 405 is provided between the readout unit 404 and the nonlinear correction unit 406, and applies a digital gain to the image data generated by the image acquisition unit 403. Data for image correction often has a decimal value, but when the image output is an integer, the correction accuracy may decrease due to quantization error. By applying a gain to the image data in advance, the effect of quantization error can be suppressed and the correction accuracy can be improved. Here, if the quantization error can be suppressed to 1 / 4 times or less of the one-photon signal level, the corrected image will be visually natural. Therefore, it is desirable that the digital gain applied to the image data is, for example, 4 times or more.

[0095] The nonlinear correction unit 406 is disposed between the gain adjustment unit 405 and the defect correction unit 407, and corrects the image data under the control of the control unit 401. When the image acquisition unit 403 is a photon-counting type detector, the optical response often becomes nonlinear due to the effect of dead time. When affected by a nonlinear optical response, correction based on a linear response may result in overcorrection. Therefore, overcorrection can be prevented by performing nonlinear correction on the image data prior to the calculation processing in the defect correction unit 407, and appropriate nonlinear correction according to the drive timing can be performed. This nonlinear correction is performed using, for example, a LookUpTable.

[0096] The defect correction unit 407 corrects the data of defective pixels contained in the image data. As a specific example, the output value of the defective pixel is extracted, and the position information and output value of the defective pixel are identified. There are a method of replacing the output value with the average value or median value of the output of the pixels around the identified defective pixel, and a method of dividing the estimated defective image data.

[0097] The data compression unit 408 compresses the corrected image data. In the photoelectric conversion device according to the present invention, a huge amount of image data corresponding to a high dynamic range is generated. By providing the data compression unit 408, the data can be compressed before being stored in the storage unit 409 at the downstream.

[0098] The storage unit 409 is a storage unit that holds at least a part of the image data generated in the previous stage. Specifically, the image data is stored in a memory such as an SRAM, a DRAM, or a non-volatile memory.

[0099] In this manner, according to this embodiment, it is possible to realize a photoelectric conversion system to which the photoelectric conversion device shown in any of the above-described embodiments is applied.

[0100] Seventh embodiment The effects of the present invention will be further described with reference to Fig. 26. In the following description, the case where addition is performed according to each of the above-mentioned embodiments is referred to as a weighted counting method.

[0101] In the conventional clock recharge method shown in Fig. 26(a), the maximum estimated number of incident photons is determined by the number of Recharge CLKs (P_PCLK). In other words, the more the number of Recharge CLKs, the wider the dynamic range. On the other hand, power consumption increases in proportion to the number of Recharge CLKs, so there is a trade-off between dynamic range and power consumption.

[0102] When the exposure period is T and the pulse interval of Recharge CLK is Δtr, the number of Recharge CLKs is expressed by T / Δtr, so it can also be considered that the dynamic range is determined by T / Δtr.

[0103] In contrast, in the weighted counting method as shown in FIG. 26(b), the number of incident photons is estimated from the timing at which the photons are incident. Therefore, the maximum estimated number of incident photons is determined by T / Δtw. Here, Δtw is the period from the pulse of Recharge CLK (P_PCLK in the embodiment) to the input of the first pulse of P_TCLK. In other words, the maximum estimated number of incident photons is no longer dependent on the Recharge CLK interval Δtr, and there is no longer a trade-off between the dynamic range and power consumption. In this case, the weighting coefficient (the amount of increase in the count value corresponding to the incidence of one photon) is preferably Δtr / Δtw.

[0104] Specific numerical examples are shown below. In the conventional method, when the exposure period T is 1024 and Δtr is 1, the maximum number of avalanches is 1024, and the maximum number of detected incident photons at this time is 1024. On the other hand, when the exposure period T of the weighted count method is 1024, Δtr is 4, and Δtw is 1, the maximum number of avalanches is 256. If the count number corresponding to the photons incident at Δtw is added to the counter circuit to be Δtr / Δtw, which is 4, the maximum estimated number of incident photons at this time is 1024. In other words, the power consumption associated with recharging can be suppressed to 1 / 4 of that of the conventional method, while achieving the same dynamic range. In each of the above-mentioned embodiments, in order to add 4 at the timing of Δtw, it can be realized by inputting 4 pulses of P_TCLK in a time sufficiently shorter than Δtw, for example.

[0105] Here, it is preferable to set the weighting coefficient to Δtr / Δtw because this allows the saturation counts to be the same under driving conditions where both methods have the same dynamic range. Setting such conditions makes it possible to reduce false signals during nonlinear correction. However, since power consumption depends only on Δtr and not on the weighting coefficient, it is not necessary to satisfy the above relationship from the viewpoint of reducing power consumption.

[0106] Eighth embodiment The photoelectric conversion system according to this embodiment will be described with reference to Fig. 27. Fig. 27 is a block diagram showing a schematic configuration of the photoelectric conversion system according to this embodiment.

[0107] The photoelectric conversion device described in the above embodiment is applicable to various photoelectric conversion systems. Examples of the applicable photoelectric conversion system include digital still cameras, digital camcorders, security cameras, copiers, fax machines, mobile phones, car-mounted cameras, and observation satellites. In addition, a camera module having an optical system such as a lens and an imaging device is also included in the photoelectric conversion system. FIG. 27 illustrates a block diagram of a digital still camera as an example of these.

[0108] 27 includes an image pickup device 1004, which is an example of a photoelectric conversion device, and a lens 1002 that forms an optical image of a subject on the image pickup device 1004. The photoelectric conversion system further includes an aperture 1003 that varies the amount of light passing through the lens 1002, and a barrier 1001 that protects the lens 1002. The lens 1002 and the aperture 1003 form an optical system that focuses light on the image pickup device 1004. The image pickup device 1004 is a photoelectric conversion device according to any one of the above embodiments, and converts the optical image formed by the lens 1002 into an electrical signal.

[0109] The photoelectric conversion system has a signal processing unit 1007 which is an image generating unit that generates an image by processing an output signal output from an imaging device 1004. The signal processing unit 1007 performs various corrections and compression as necessary to output image data. The signal processing unit 1007 may be formed in a semiconductor layer in which the imaging device 1004 is provided, or may be formed in a semiconductor layer different from that of the imaging device 1004. Moreover, the imaging device 1004 and the signal processing unit 1007 may be formed in the same semiconductor layer.

[0110] The photoelectric conversion system further includes a memory unit 1010 for temporarily storing image data, and an external interface unit (external I / F unit) 1013 for communicating with an external computer or the like. The photoelectric conversion system further includes a recording medium 1012 such as a semiconductor memory for recording or reading out imaging data, and a recording medium control interface unit (recording medium control I / F unit) 1011 for recording or reading out data on the recording medium 1012. The recording medium 1012 may be built into the photoelectric conversion system, or may be removable.

[0111] The photoelectric conversion system further includes an overall control / calculation unit 1009 that performs various calculations and controls the entire digital still camera, and a timing generation unit 1008 that outputs various timing signals to the image capture device 1004 and the signal processing unit 1007. Here, the timing signals and the like may be input from outside, and the photoelectric conversion system only needs to include at least the image capture device 1004 and the signal processing unit 1007 that processes the output signal output from the image capture device 1004.

[0112] The imaging device 1004 outputs an imaging signal to a signal processing unit 1007. The signal processing unit 1007 performs predetermined signal processing on the imaging signal output from the imaging device 1004, and outputs image data. The signal processing unit 1007 generates an image using the imaging signal.

[0113] In this way, according to this embodiment, it is possible to realize a photoelectric conversion system to which the photoelectric conversion device (imaging device) according to any one of the above embodiments is applied.

[0114] Ninth embodiment The photoelectric conversion system and the moving object of this embodiment will be described with reference to Fig. 28. Fig. 28 is a diagram showing the configuration of the photoelectric conversion system and the moving object of this embodiment.

[0115] FIG. 28(a) shows an example of a photoelectric conversion system related to an in-vehicle camera. The photoelectric conversion system 2300 has an imaging device 2310. The imaging device 2310 is the photoelectric conversion device described in any of the above embodiments. The photoelectric conversion system 2300 has an image processing unit 2312 that performs image processing on a plurality of image data acquired by the imaging device 2310. The photoelectric conversion system 2300 also has a parallax acquisition unit 2314 that calculates parallax (phase difference of parallax images) from a plurality of image data acquired by the photoelectric conversion system 2300. The photoelectric conversion system 2300 also has a distance acquisition unit 2316 that calculates a distance to an object based on the calculated parallax, and a collision determination unit 2318 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 2314 and the distance acquisition unit 2316 are examples of distance information acquisition means that acquire distance information to an object. That is, distance information may be acquired using not only phase difference but also ToF (Time Of Flight) technology. The collision determination unit 2318 may determine the possibility of a collision using any of these pieces of distance information. The distance information acquisition means may be realized by dedicated hardware, or may be realized by a software module. In addition, it may be realized by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like, or may be realized by a combination of these.

[0116] The photoelectric conversion system 2300 is connected to a vehicle information acquisition device 2320, and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 2300 is also connected to a control ECU 2330, which is a control device (control unit) that outputs a control signal to generate a braking force for the vehicle based on the judgment result of the collision judgment unit 2318. The photoelectric conversion system 2300 is also connected to an alarm device 2340 that issues an alarm to the driver based on the judgment result of the collision judgment unit 2318. For example, when the judgment result of the collision judgment unit 2318 indicates that there is a high possibility of a collision, the control ECU 2330 performs vehicle control to avoid a collision and reduce damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 2340 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., or vibrating the seat belt or steering wheel.

[0117] In this embodiment, the surroundings of the vehicle, for example, the front or rear, are imaged by the photoelectric conversion system 2300. Fig. 28(b) shows the photoelectric conversion system when imaging the area in front of the vehicle (imaging range 2350). A vehicle information acquisition device 2320 sends an instruction to the photoelectric conversion system 2300 or the imaging device 2310. This configuration can further improve the accuracy of distance measurement.

[0118] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from lanes, etc. Furthermore, the photoelectric conversion system is not limited to vehicles such as the vehicle itself, but can be applied to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the present invention can be applied not only to moving bodies, but also to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0119] Tenth embodiment The photoelectric conversion system of this embodiment will be described with reference to Fig. 29. Fig. 29 is a block diagram showing an example of the configuration of a range image sensor which is the photoelectric conversion system.

[0120] 29, the distance image sensor 1401 is configured to include an optical system 1402, a photoelectric conversion device 1403, an image processing circuit 1404, a monitor 1405, and a memory 1406. The distance image sensor 1401 can obtain a distance image according to the distance to the subject by receiving light (modulated light or pulsed light) that is projected from a light source device 1411 toward the subject and reflected by the surface of the subject.

[0121] The optical system 1402 is configured to have one or more lenses, and guides image light (incident light) from a subject to the photoelectric conversion device 1403 , forming an image on the light receiving surface (sensor portion) of the photoelectric conversion device 1403 .

[0122] The photoelectric conversion device 1403 is any of the photoelectric conversion devices described in the above-mentioned embodiments, and a distance signal indicating a distance determined from a received light signal output from the photoelectric conversion device 1403 is supplied to an image processing circuit 1404.

[0123] The image processing circuit 1404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 1403. The distance image (image data) obtained by this image processing is then supplied to a monitor 1405 for display, or supplied to a memory 1406 for storage (recording).

[0124] In the range image sensor 1401 configured in this way, by applying the above-mentioned photoelectric conversion device, it is possible to obtain, for example, a more accurate range image as the pixel characteristics improve.

[0125] Eleventh embodiment The photoelectric conversion system of this embodiment will be described with reference to Fig. 30. Fig. 30 is a diagram showing an example of a schematic configuration of an endoscopic surgery system which is the photoelectric conversion system of this embodiment.

[0126] 30 shows a state in which an operator (doctor) 1131 is performing surgery on a patient 1132 on a patient bed 1133 using an endoscopic surgery system 1103. As shown in the figure, the endoscopic surgery system 1103 is composed of an endoscope 1100, a surgical tool 1110, and a cart 1134 on which various devices for endoscopic surgery are mounted.

[0127] The endoscope 1100 is composed of a lens barrel 1101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 1132, and a camera head 1102 connected to the base end of the lens barrel 1101. In the illustrated example, the endoscope 1100 is configured as a so-called rigid lens barrel having a rigid lens barrel 1101, but the endoscope 1100 may be configured as a so-called flexible lens barrel having a flexible lens barrel.

[0128] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 1101. A light source device 1203 is connected to the endoscope 1100, and light generated by the light source device 1203 is guided to the tip of the lens barrel 1101 by a light guide extending inside the lens barrel 1101, and is irradiated via the objective lens toward an observation target in a body cavity of a patient 1132. The endoscope 1100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0129] An optical system and a photoelectric conversion device are provided inside the camera head 1102, and reflected light (observation light) from an observation target is collected by the optical system onto the photoelectric conversion device. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The photoelectric conversion device may be any of the photoelectric conversion devices described in the above-mentioned embodiments. The image signal is transmitted to a camera control unit (CCU) 1135 as RAW data.

[0130] The CCU 1135 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and performs overall control of the operations of the endoscope 1100 and the display device 1136. Furthermore, the CCU 1135 receives an image signal from the camera head 1102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0131] Under the control of the CCU 1135 , the display device 1136 displays an image based on an image signal that has been subjected to image processing by the CCU 1135 .

[0132] The light source device 1203 is composed of a light source such as an LED (Light Emitting Diode), and supplies the endoscope 1100 with irradiation light when photographing an operation site or the like.

[0133] The input device 1137 is an input interface for the endoscopic surgery system 1103. A user can input various information and instructions to the endoscopic surgery system 1103 via the input device 1137.

[0134] The treatment tool control device 1138 controls the driving of the energy treatment tool 1112 for cauterizing tissue, incising, sealing blood vessels, or the like.

[0135] The light source device 1203 that supplies irradiation light to the endoscope 1100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 1203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation target with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 1102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0136] Furthermore, the light source device 1203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 1102 in synchronization with the timing of the change in the light intensity to acquire images in a time-division manner and synthesizing the images, it is possible to generate an image with a high dynamic range that is free of so-called blocked-up shadows and blown-out highlights.

[0137] The light source device 1203 may be configured to supply light in a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, the wavelength dependency of light absorption in body tissue is utilized. Specifically, a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast by irradiating light in a narrower band than the irradiation light (i.e., white light) in normal observation. Alternatively, the special light observation may be a fluorescent observation in which an image is obtained by fluorescence generated by irradiating excitation light. In the fluorescent observation, it is possible to irradiate excitation light to the body tissue and observe the fluorescence from the body tissue, or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 1203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0138] Twelfth embodiment The photoelectric conversion system of this embodiment will be described with reference to FIG. 31. FIG. 31(a) is a diagram showing an example of the configuration of glasses 1600 (smart glasses) which are a photoelectric conversion system. The glasses 1600 have a photoelectric conversion device 1602. The photoelectric conversion device 1602 is the photoelectric conversion device described in the above-mentioned twelfth embodiment. A display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 1601. The photoelectric conversion device 1602 may be one or more. A combination of multiple types of photoelectric conversion devices may also be used. The arrangement position of the photoelectric conversion device 1602 is not limited to that shown in FIG. 31(a).

[0139] The glasses 1600 further include a control device 1603. The control device 1603 functions as a power source that supplies power to the photoelectric conversion device 1602 and the display device. The control device 1603 also controls the operations of the photoelectric conversion device 1602 and the display device. The lens 1601 is formed with an optical system for focusing light on the photoelectric conversion device 1602.

[0140] FIG. 31(b) illustrates glasses 1610 (smart glasses) according to one application example. The glasses 1610 have a control device 1612, and the control device 1612 is equipped with a photoelectric conversion device corresponding to the photoelectric conversion device 1602 and a display device. The lens 1611 is formed with an optical system for projecting light emitted from the photoelectric conversion device in the control device 1612 and the display device, and an image is projected onto the lens 1611. The control device 1612 functions as a power source for supplying power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device may have a line of sight detection unit that detects the line of sight of the wearer. Infrared light may be used for detecting the line of sight. The infrared light emission unit emits infrared light toward the eyeball of a user gazing at a display image. An imaging unit having a photoelectric conversion element detects the reflected light of the emitted infrared light from the eyeball, thereby obtaining an image of the eyeball. By having a reduction means for reducing light from the infrared light emission unit to the display unit in a planar view, deterioration of image quality is reduced.

[0141] The gaze of the user with respect to the displayed image is detected from an image of the eyeball obtained by capturing infrared light. Any known method can be applied to gaze detection using the image of the eyeball. As an example, a gaze detection method based on a Purkinje image formed by reflection of irradiated light on the cornea can be used.

[0142] More specifically, a gaze detection process based on the pupil-corneal reflex method is performed. Using the pupil-corneal reflex method, a gaze vector that indicates the direction (rotation angle) of the eyeball is calculated based on the pupil image and the Purkinje image included in the captured image of the eyeball, thereby detecting the user's gaze.

[0143] The display device of this embodiment may have a photoelectric conversion device having a photoelectric conversion element, and may control a display image on the display device based on information on a user's line of sight from the photoelectric conversion device.

[0144] Specifically, the display device determines a first field of view area to which the user gazes and a second field of view area other than the first field of view area based on the line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device, or may be received from an external control device. In the display area of ​​the display device, the display resolution of the first field of view area may be controlled to be higher than the display resolution of the second field of view area. In other words, the resolution of the second field of view area may be lower than that of the first field of view area.

[0145] The display area may have a first display area and a second display area different from the first display area, and a high priority area may be determined from the first display area and the second display area based on line-of-sight information. The first field of view area and the second field of view area may be determined by a control device of the display device, or may be received from an external control device. The resolution of the high priority area may be controlled to be higher than the resolution of areas other than the high priority area. In other words, the resolution of an area with a relatively low priority may be lowered.

[0146] In addition, AI may be used to determine the first field of view area and the area with high priority. The AI ​​may be a model configured to estimate the angle of the line of sight and the distance to an object at the end of the line of sight from the image of the eyeball, using the image of the eyeball and the direction in which the eyeball in the image was actually looking as teacher data. The AI ​​program may be included in the display device, the photoelectric conversion device, or an external device. If included in the external device, it is transmitted to the display device via communication.

[0147] When display control is performed based on visual recognition detection, the present invention is preferably applicable to smart glasses further including a photoelectric conversion device for capturing an image of the outside world. The smart glasses can display captured outside information in real time.

[0148] The above-described embodiments can be modified as appropriate without departing from the technical concept. In addition, an example in which a part of the configuration of any of the embodiments is added to another embodiment, or an example in which a part of the configuration of any of the embodiments is replaced with another embodiment, is also included in the embodiments of the present invention.

[0149] The disclosure of this embodiment also includes the following configurations and methods.

[0150] (Configuration 1) A photoelectric conversion element that receives photons; an exposure control unit that generates a signal that defines a plurality of second exposure periods that are included in a first exposure period corresponding to one frame and are shorter than the first exposure period; a timing generating unit that generates a pulse signal that defines time information within the second exposure period; a measurement unit that measures the number of the pulse signals generated by the timing generation unit during the second exposure period after a first detection of a photon; A photoelectric conversion device having the above structure.

[0151] (Configuration 2) the timing generation unit generates equally spaced pulse signals; 2. The photoelectric conversion device according to configuration 1, wherein the pulse signal is input to the measurement unit at least during the second exposure period.

[0152] (Configuration 3) the timing generation unit generates pulse signals at non-equidistant intervals; 2. The photoelectric conversion device according to configuration 1, wherein the pulse signal is input to the measurement unit at least during the second exposure period.

[0153] (Configuration 4) 4. The photoelectric conversion device according to configuration 3, wherein the non-equidistant pulse signal is configured so that a period thereof becomes longer according to the elapsed time of the second exposure period.

[0154] (Configuration 5) 5. The photoelectric conversion device according to configuration 3 or 4, wherein the intervals of the non-equidistant pulse signals are set so as to achieve logarithmic compression of real time.

[0155] (Configuration 6) 6. The photoelectric conversion device according to any one of configurations 1 to 5, wherein the measurement section includes a waveform shaping circuit that converts the signal from the photoelectric conversion element into a pulse signal.

[0156] (Configuration 7) the measurement unit includes a timing determination circuit, The photoelectric conversion device according to any one of configurations 1 to 6, wherein the timing judgment circuit outputs a pulse signal generated by the timing generation unit after the timing at which a photon is first detected in the second exposure period.

[0157] (Configuration 8) the measurement unit includes a counter circuit, 8. The photoelectric conversion device according to configuration 7, wherein the counter circuit performs counting for each input of a pulse signal output from the timing determination circuit during the second exposure period and sets the count value as a count value.

[0158] (Configuration 9) 9. The photoelectric conversion device according to configuration 8, wherein the counter circuit adds up the count values ​​of the plurality of second exposure periods and outputs the sum.

[0159] (Configuration 10) 10. The photoelectric conversion device according to claim 1, wherein the photoelectric conversion element is an avalanche photodiode.

[0160] (Configuration 11) 11. The photoelectric conversion device according to configuration 10, further comprising a switch for performing a charging operation disposed between the avalanche photodiode and a power source for applying a reverse bias to the avalanche photodiode.

[0161] (Configuration 12) 12. The photoelectric conversion device according to configuration 11, wherein the second exposure period is a period from when the charging operation is performed to when the next charging operation is performed.

[0162] (Configuration 13) 12. The photoelectric conversion device according to configuration 11, wherein the pulse signal input to the switch that performs the charging operation and the pulse signal generated by the timing generating unit are generated from a reference clock signal.

[0163] (Configuration 14) The measurement unit has a selection circuit, the plurality of second exposure periods include one of the second exposure periods and another of the second exposure periods, A photoelectric conversion device according to any one of configurations 1 to 13, wherein the selection circuit selects whether to output a signal corresponding to the pulse signal that specifies time information within the other second exposure period, depending on whether a photon is not detected during the one second exposure period or whether a photon is detected during the other second exposure period.

[0164] (Configuration 15) The photoelectric conversion device described in configuration 14, characterized in that when no photon is detected during the second exposure period of one of the photoelectric conversion devices, the selection circuit outputs a predetermined value regardless of the time from the start of the second exposure period of the other photoelectric conversion device to the first detection of a photon.

[0165] (Configuration 16) The first substrate and the second substrate are laminated together, the first substrate has the photoelectric conversion element, 16. The photoelectric conversion device according to any one of configurations 1 to 15, wherein the second substrate has the exposure control unit, the timing generation unit, and the measurement unit.

[0166] (Configuration 17) A first substrate, a second substrate, and a third substrate are laminated together, the first substrate has the photoelectric conversion element, the second substrate has the waveform shaping circuit, the timing determination circuit, and the counter circuit; 9. The photoelectric conversion device according to configuration 8, wherein the third substrate has the timing judgment circuit and the counter circuit.

[0167] (Configuration 18) a pulse signal generated by the timing generating unit and input to the timing judgment circuit included in the second board; 18. The photoelectric conversion device according to configuration 17, wherein the pulse signal generated by the timing generating section and input to the timing judgment circuit included in the third substrate is different.

[0168] (Configuration 19) A photoelectric conversion device according to any one of configurations 1 to 18, a signal processing unit that generates an image using a signal output from the photoelectric conversion device.

[0169] (Configuration 20) A moving object including the photoelectric conversion device according to any one of configurations 1 to 18, A moving object having a control unit that controls the movement of the moving object using a signal output by the photoelectric conversion device. [Explanation of symbols]

[0170] 1 Photoelectric conversion device 2 Exposure control section 3 Timing Generation Section 4. Measurement section 201 Signal Processing Circuit 202 Timing Judgment Circuit 203 Counter Circuit 204 Quench element 205 Waveform shaping circuit 206 Latch Circuit 207 Signal Selection Circuit 208 Signal holding circuit 209 In-pixel pulse generation circuit

Claims

1. A photoelectric conversion element that receives photons; an exposure control unit that generates a signal that defines a plurality of second exposure periods that are included in a first exposure period corresponding to one frame and are shorter than the first exposure period; a timing generating unit that generates a pulse signal that defines time information within the second exposure period; a measurement unit that measures the number of the pulse signals generated by the timing generation unit during the second exposure period after a first detection of a photon; A photoelectric conversion device having the above structure.

2. the timing generation unit generates equally spaced pulse signals; The photoelectric conversion device according to claim 1 , wherein the pulse signal is input to the measurement unit at least during the second exposure period.

3. the timing generation unit generates pulse signals at non-equidistant intervals; The photoelectric conversion device according to claim 1 , wherein the pulse signal is input to the measurement unit at least during the second exposure period.

4. 4. The photoelectric conversion device according to claim 3, wherein the non-equidistant pulse signal is configured so that a period of the non-equidistant pulse signal increases according to an elapsed time of the second exposure period.

5. 5. The photoelectric conversion device according to claim 4, wherein the intervals of the non-equidistant pulse signals are set so as to achieve logarithmic compression of real time.

6. 2. The photoelectric conversion device according to claim 1, wherein the measurement section includes a waveform shaping circuit that converts the signal from the photoelectric conversion element into a pulse signal.

7. the measurement unit includes a timing determination circuit, 7. The photoelectric conversion device according to claim 6, wherein the timing determination circuit outputs a pulse signal generated by the timing generation unit after the timing at which a photon is first detected in the second exposure period.

8. the measurement unit includes a counter circuit, 8. The photoelectric conversion device according to claim 7, wherein the counter circuit performs counting for each input of a pulse signal output from the timing determination circuit during the second exposure period, and sets the count value as the count value.

9. 9. The photoelectric conversion device according to claim 8, wherein the counter circuit adds up the count values ​​for the plurality of second exposure periods and outputs the sum.

10. 2. The photoelectric conversion device according to claim 1, wherein the photoelectric conversion element is an avalanche photodiode.

11. 11. The photoelectric conversion device according to claim 10, further comprising a switch for performing a charging operation disposed between the avalanche photodiode and a power source that applies a reverse bias to the avalanche photodiode.

12. The photoelectric conversion device according to claim 11 , wherein the second exposure period is a period from a timing when the charging operation is performed to a timing when the next charging operation is performed.

13. The photoelectric conversion device according to claim 11 , wherein the pulse signal input to the switch that performs the charging operation and the pulse signal generated by the timing generating section are generated from a reference clock signal.

14. The measurement unit has a selection circuit, the plurality of second exposure periods include one of the second exposure periods and another of the second exposure periods, The photoelectric conversion device according to claim 1, wherein the selection circuit selects whether to output a signal corresponding to the pulse signal that specifies time information within the other second exposure period depending on whether a photon is detected during the second exposure period of the one of the photoelectric conversion devices or not.

15. The photoelectric conversion device according to claim 14, characterized in that, when no photon is detected during one of the second exposure periods, the selection circuit outputs a predetermined value regardless of the time from the start of the other second exposure period to the first detection of a photon.

16. The first substrate and the second substrate are laminated together, the first substrate has the photoelectric conversion element, The photoelectric conversion device according to claim 1 , wherein the second substrate comprises the exposure control unit, the timing generation unit, and the measurement unit.

17. A first substrate, a second substrate, and a third substrate are laminated together, the first substrate has the photoelectric conversion element, the second substrate has the waveform shaping circuit, the timing determination circuit, and the counter circuit; The photoelectric conversion device according to claim 8 , wherein the third substrate has the timing determination circuit and the counter circuit.

18. a pulse signal generated by the timing generating unit and input to the timing judgment circuit included in the second board; The photoelectric conversion device according to claim 17 , wherein the pulse signal generated by the timing generating section and input to the timing judgment circuit provided on the third substrate is different from the pulse signal generated by the timing generating section.

19. The photoelectric conversion device according to any one of claims 1 to 18, a signal processing unit that generates an image using a signal output from the photoelectric conversion device.

20. A moving object including the photoelectric conversion device according to any one of claims 1 to 18, A moving object having a control unit that controls the movement of the moving object using a signal output by the photoelectric conversion device.

Images