Photoelectric converter and photoelectric converter system having a photoelectric converter

Abstract

This makes it possible to acquire more signal information while maintaining an expanded dynamic range. [Solution] The system includes: an exposure control unit that generates signals defining a plurality of second exposure periods shorter than the first exposure period and included in a first exposure period corresponding to one frame; a timing generation unit that generates pulse signals defining time information within the second exposure periods; a measurement unit that measures the number of pulse signals since the first detection of a photon in the second exposure period based on the pulse signals generated by the timing generation unit; and a pulse control unit that controls the number of pulse signals in the second exposure period following the second exposure period based on the value of the measurement unit.

Description

[Technical Field]

[0001] The present invention relates to a photoelectric converter and a photoelectric converter system having a photoelectric converter. [Background technology]

[0002] Patent Document 1 discloses a configuration in which the time until a counter saturates is measured for each pixel, and the amount of light is estimated by extrapolation from the time and count value. This configuration allows for an expansion of the dynamic range. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2020 / 179928 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] According to the configuration of Patent Document 1, the counter stops due to saturation, resulting in the loss of signal information after the counter stops. This can lead to artifacts during shooting of moving objects, or variations in brightness during shooting of flicker light sources. [Means for solving the problem]

[0005] A photoelectric conversion device comprising: a photoelectric conversion element that receives photons; an exposure control unit that generates signals defining a plurality of second exposure periods that are included in a first exposure period corresponding to one frame and are shorter than the first exposure period; a timing generation unit that generates pulse signals defining time information within the second exposure periods; a measurement unit that measures the number of pulse signals since the first detection of a photon in the second exposure period based on the pulse signals generated by the timing generation unit; and a pulse control unit that controls the number of pulse signals in the second exposure period following the second exposure period based on the value of the measurement unit. [Effects of the Invention]

[0006] According to the present invention, it is possible to acquire more signal information as compared with Patent Document 1.

Brief Description of Drawings

[0007] [Figure 1] Figure showing a configuration example of a pixel circuit [Figure 2] Figure explaining the relationship between the detection time and the number of incident photons [Figure 3] Figure explaining the relationship between the detection time and the number of incident photons [Figure 4] Figure explaining the detection probability of photons [Figure 5] Figure explaining the expected value with respect to the number of incident photons and the input / output characteristics [Figure 6] Figure explaining the photoelectric conversion device according to the first embodiment [Figure 7] Figure explaining the photoelectric conversion device according to the first embodiment [Figure 8] Figure explaining the photoelectric conversion device according to the first embodiment [Figure 9] Figure explaining the photoelectric conversion device according to the first embodiment [Figure 10] Figure explaining the pixel circuit and driving of the photoelectric conversion device according to the first embodiment [Figure 11] Figure explaining the operation sequence of the photoelectric conversion device according to the first embodiment [Figure 12] Figure explaining the case where the clock signal defining the time information is equally spaced and non-equally spaced [Figure 13] Figure explaining the clock signal defining the time information and the input / output characteristics [Figure 14] Figure explaining the pixel circuit of the photoelectric conversion device according to the second embodiment [Figure 15] Figure explaining the operation sequence of the photoelectric conversion device according to the second embodiment [Figure 16] Figure explaining the pixel circuit of the photoelectric conversion device according to the third embodiment [Figure 17] Figure explaining the operation sequence of the photoelectric conversion device according to the third embodiment [Figure 18] FIG. for explaining the effect of the photoelectric conversion device according to the third embodiment [Figure 19] FIG. for explaining the photoelectric conversion device according to the fourth embodiment [Figure 20] FIG. for explaining the pixel circuit of the photoelectric conversion device according to the fourth embodiment [Figure 21] FIG. for explaining the signal processing block of the photoelectric conversion device according to the fifth embodiment [Figure 22] It is a schematic diagram according to the sixth embodiment. [Figure 23] It is a functional block diagram of the photoelectric conversion system according to the seventh embodiment. [Figure 24] It is a functional block diagram of the photoelectric conversion system according to the eighth embodiment. [Figure 25] It is a functional block diagram of the photoelectric conversion system according to the ninth embodiment. [Figure 26] It is a functional block diagram of the photoelectric conversion system according to the tenth embodiment. [Figure 27] It is a functional block diagram of the photoelectric conversion system according to the eleventh embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0008] The following embodiments are for embodying the technical idea of the present invention and do not limit the present invention. The sizes and positional relationships of the members shown in each drawing may be exaggerated for clarity of explanation. In the following description, the same components may be denoted by the same reference numerals and the description thereof may be omitted.

[0009] In the following description, terms indicating a specific direction or position (for example, "up", "down", "right", "left", and other terms including those terms) are used as necessary. The use of those terms is for facilitating the understanding of the embodiments with reference to the drawings, and the technical scope of the present invention is not limited by the meanings of those terms.

[0010] In this specification, a plan view refers to a view taken from a direction perpendicular to the main surface of the semiconductor layer, and is synonymous with a top view. A cross-sectional view refers to a view taken from a direction perpendicular to the surface perpendicular to the main surface of the semiconductor layer.

[0011] In the following explanation, the anode of the photodiode (PD) is set to a fixed potential, and the signal is taken from the cathode side. Therefore, the semiconductor region of the first conductivity type, where the majority carriers are charges of the same polarity as the signal charge, is the N-type semiconductor region, and the semiconductor region of the second conductivity type, where the majority carriers are charges of a different polarity than the signal charge, is the P-type semiconductor region. Alternatively, the cathode of the PD may be set to a fixed potential, and the signal may be taken from the anode side. In this case, the semiconductor region of the first conductivity type, where the majority carriers are charges of the same polarity as the signal charge, is the P-type semiconductor region, and the semiconductor region of the second conductivity type, where the majority carriers are charges of a different polarity than the signal charge, is the N-type semiconductor region. The following explanation describes the case where one node of the PD is set to a fixed potential, but the potentials of both nodes may be configured to fluctuate.

[0012] (Basic configuration) Figure 1 shows a schematic configuration of the photoelectric conversion device according to this embodiment. The photoelectric conversion device 100 includes a photoelectric conversion element 1, an exposure control unit 2, a timing generation unit 3, a measurement unit 4, and a pulse control unit 5.

[0013] The photoelectric conversion element 1 detects the incident photon and converts it into an electrical signal. The photoelectric conversion element 1 may be a linear-mode avalanche photodiode operating around the breakdown voltage, or a single-photon avalanche photodiode operating in Geiger mode.

[0014] The exposure control unit 2 generates a signal that defines the exposure period corresponding to one frame (the first exposure period). The "exposure period corresponding to one frame" is sometimes also called the "one-frame period." The exposure control unit 2 also generates a signal that defines the exposure periods corresponding to multiple subframes included in the exposure period corresponding to one frame (the second exposure period). The "exposure period corresponding to a subframe" is sometimes also called the "subframe period."

[0015] The timing generation unit 3 generates pulse signals to define time information within the subframe period (within the second exposure period).

[0016] The measurement unit 4 receives signals from the photoelectric conversion element 1, control signals from the exposure control unit 2, and control signals from the timing generation unit 3 via the pulse control unit 5. The measurement unit 4 measures a numerical value from these signals that corresponds to the detection time of the first photon detected from the start of the subframe period. The measurement unit may use a general TDC circuit as the circuit for measuring time, or it may measure by other methods.

[0017] The pulse control unit 5 receives signals from the measurement unit 4 and signals from the timing generation unit 3. Based on the values ​​from the measurement unit 4, the pulse control unit 5 controls the number of control signals input to the measurement unit 4.

[0018] Figure 2(A) shows the relationship between exposure time and detection time, and Figure 2(B) shows the relationship between detection time and the number of incident photons.

[0019] Here, the exposure period is T aac T is the time from the start of the exposure period until a photon is detected. detect In this case, the number of incident photons N that enter the exposure period is... ph is, N ph =(T aac / T detect It is expressed as follows: Therefore, if the exposure period and detection time are known, it is possible to estimate the number of incident photons. In other words, even without counting the number of incident photons themselves, the number of incident photons that occur during the exposure period can be determined, making imaging possible.

[0020] FIG. 3(A) and (B) are diagrams showing the case where the exposure shown in FIG. 2 is performed multiple times. Here, the measurement period for each time is called a sub-frame period, and it represents that the sub-frame period is repeated N times. One frame period is the sum from 1 sub-frame to N sub-frames. At this time, the average of the time during which photons are detected in sub-frames during one frame period is T ave If we set it as such, the number of incident photons N ph in one frame ph = (T aac / T<00s0011>) × N can be estimated. FIG. 3(C) is a diagram showing the correspondence between the average value of the detection time and the number of incident photons.

[0021] In FIG. 3, the explanation was given focusing on the average time of photon detection, but it is also possible to estimate the number of incident photons from the integration time of photon detection. In addition, the averaging and integration processes may be performed inside the photoelectric conversion device, or the signal for each sub-frame may be output outside the photoelectric conversion device and performed by an external processing circuit.

[0022] The above-described relational expression between the number of incident photons and the detection time is suitable when the minimum unit of the detection time is sufficiently smaller than the detection time of photons, but an influence may occur when the minimum unit of the detection time and the detection time of photons are of the same degree. For example, even within the minimum unit of the detection time, the number of incident photons assumed by the timing at which photons are originally detected may vary, so an error may occur in the above relational expression. Such a situation where the error cannot be ignored is expressed as the detection time being affected by discreteness.

[0023] In FIG. 4, the estimation of the number of photons when the detection time is regarded as discrete will be described. FIG. 4 is a conceptual diagram explaining the detection probability of photons. When the detection times are 1, 2,... etc., the probability that a photon is detected at a certain time is known to follow an exponential distribution (f(t,λ) = λe -λt , where λ is the number of events per unit time). In this case, λ is the number of incident photons N phλ is a value determined by the exposure time and the unit time (the smallest unit of detection time). Specifically, λ is (number of incident photons / exposure time) × (unit time). For example, if the number of incident photons is 1, the exposure time is 1000, and the unit time is 1, then (1 / 1000) × 1, and λ is 0.001. Following this exponential distribution, the probability of detecting a photon can be represented by the graph shown in Figure 4(A) when the horizontal axis is the detection time. Here, the solid line represents high illuminance, the dashed line represents medium illuminance, and the dotted line represents low illuminance. Also, the interval cumulative detection probability is F(t,λ)=1-e -λt It is expressed as and can be represented by the graph shown in Figure 4(B). Furthermore, the interval cumulative detection probability, obtained by accumulating the detection probability for each detection interval (0~1, 1~2, ...), is F'(n,λ)=F(t n ,λ)-F(t n+1 It can be expressed as λ, and can be represented by the graph shown in Figure 4(C).

[0024] By performing such calculations, the photon detection probability can be determined for each detection interval. The sum of the products of these interval cumulative detection probabilities and the corresponding detection times (expected value E) is:

[0025]

number

[0026] This can be calculated as follows. This expected value E is shown in Figure 3(B) as T ave This corresponds to the value E. From the above calculations, even if the detection time is discrete, the number of incident photons N can be calculated from the expected value E. ph It is possible to estimate this.

[0027] Figure 5(A) shows a specific example of the expected value with respect to the number of incident photons. It shows the output E (expected value of detection time) and output E' (expected value of exposure time - detection time) obtained when the input (number of incident photons) is changed to 1, 10, 100, and 1000, with an exposure period of 1000 and a unit time of 1.

[0028] Furthermore, Figures 5(B) and 5(C) show graphs of the input-output characteristics for output E and output E'. Output E is equal to the number of incident photons N. ph The curve shows that the output E decreases as N increases. On the other hand, the output E' is determined by the number of incident photons N. ph The curve shows that the output E' increases as the value of increases.

[0029] In imaging devices, the signal output is generally configured such that the output increases as the number of incident photons (the input) increases. Therefore, considering subsequent signal processing, it is preferable to output output E' instead of output E.

[0030] With this configuration, the number of incident photons during the exposure period can be estimated from the photon detection timing, making it possible to expand the dynamic range beyond the number of detected photons. Furthermore, since the length of the subframe period can be set by the photon exposure control unit, it is possible to adjust the counter so that it does not saturate during one frame period. In this way, by ensuring a wide dynamic range while avoiding the counter stopping during one frame period, signal information loss can be prevented.

[0031] Furthermore, by utilizing the technical concept described above, an additional configuration that stops the counter during a single frame period may be adopted.

[0032] The following describes each embodiment.

[0033] (First Embodiment) Figure 6 shows the configuration of a stacked photoelectric converter 100 according to this embodiment. The photoelectric converter 100 is constructed by stacking two substrates, a first substrate 11 (sensor substrate) and a second substrate 21 (circuit board), and electrically connecting them. The first substrate 11 has a plurality of photoelectric conversion elements 1. The circuit board has the circuit of a signal processing unit 103. In the following description, the first substrate 11 and the second substrate 21 will be described as diced chips, but are not limited to chips. For example, each substrate may be a wafer. Also, each substrate may be stacked in a wafer state and then diced, or chips may be formed first and then stacked and bonded together.

[0034] The first substrate 11 is provided with a pixel region 12, and the second substrate 21 is provided with a circuit region 22 for processing signals detected in the pixel region 12.

[0035] Figure 7 shows an example of the arrangement of the first substrate 11. Pixels 101, each having a photoelectric conversion element 1 including an avalanche photodiode (APD), are arranged in a two-dimensional array in a planar view, forming a pixel region 12.

[0036] Figure 8 is a diagram of the configuration of the second substrate 21. It includes a signal processing unit 103 for processing the charge photoelectrically converted by the photoelectric conversion element 1, a column circuit 112, a control pulse generation unit 115, a horizontal scanning circuit unit 111, a signal line 113, a vertical scanning circuit unit 110, a control line 116, and a control line 117.

[0037] The pixel 101, which has the photoelectric conversion element 1 shown in Figure 7, and the signal processing unit 103 shown in Figure 8 are electrically connected via connecting wiring provided for each pixel. The pixel 101 and the signal processing unit 103 are sometimes referred to as a pixel circuit.

[0038] The vertical scanning circuit section 110 in Figure 8 receives control pulses supplied from the control pulse generation section 115 and supplies control pulses to each pixel. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit section 110. The signal output from the photoelectric conversion element 1 is processed by the signal processing section 103.

[0039] The signal processing unit 103 is equipped with a counter and memory, and digital values ​​are stored in the memory.

[0040] The horizontal scanning circuit unit 111 inputs control pulses to the signal processing unit 103 to sequentially select each column in order to read the signal from the memory of each pixel in which the digital signal is held.

[0041] For the selected column, a signal is output from the signal processing unit 103 of the pixel selected by the vertical scanning circuit unit 110 to the signal line 113.

[0042] The signal output to signal line 113 is output via output circuit 114 to an external recording unit or signal processing unit of the photoelectric converter 100.

[0043] In Figure 7, the arrangement of photoelectric conversion elements in the pixel region may be one-dimensional. Furthermore, the effects of this embodiment can be obtained even with just one pixel, and this includes the case with only one pixel. The signal processing unit does not necessarily need to be provided for each photoelectric conversion element; for example, one signal processing unit may be shared by multiple photoelectric conversion elements, and signal processing may be performed sequentially.

[0044] As shown in Figures 7 and 8, multiple signal processing units 103 are arranged in the region that overlaps with the pixel region 12 in a plan view. Then, the vertical scanning circuit unit 110, the horizontal scanning circuit unit 111, the column circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged so as to overlap between the edge of the first substrate 11 and the edge of the pixel region 12 in a plan view. In other words, the first substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12, and the vertical scanning circuit unit 110, the horizontal scanning circuit unit 111, the column circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged in the region that overlaps with the non-pixel region in a plan view.

[0045] Figure 9 shows an example of a block configuration of a pixel array, timing generation unit, and measurement unit. The exposure control unit 2 and timing generation unit 3 may be included in the vertical scanning circuit unit 110 and control pulse generation unit 115 shown in Figure 8. The pixel circuit includes a photoelectric conversion element 1, a measurement unit 4, and a pulse control unit 5.

[0046] The measurement unit 4 consists of a signal processing circuit 201, a timing determination circuit 202, and a counter circuit 203. The signal processing circuit 201 is, for example, a waveform shaping circuit or a resistor or switch placed between the voltage applied to the photoelectric conversion element and the photoelectric conversion element. The signal processing unit 103 shown in Figure 8 corresponds to the measurement unit 4 (signal processing circuit 201, timing determination circuit 202, counter circuit 203) shown in Figure 9.

[0047] The pulse control unit 5 consists of a logic circuit 210 and a selection circuit 211. The logic circuit 210 compares the value output from the counter circuit 203 with its own output and outputs the comparison result to the selection circuit 211. The selection circuit 211 receives multiple pulse signals P_TCLK1, P_TCLK2, and P_TCLK3 from the timing generation unit 3, and selects which pulse signal to input to the timing determination circuit 202 based on the comparison result from the selection circuit 211.

[0048] The exposure control unit 2 outputs a pulse signal called P_PCLK, which is input to the signal processing circuit 201. The timing generation unit 3 outputs pulse signals called P_TCLK1, P_TCLK2, and P_TCLK3, and the pulse signal selected via the selection circuit 211 is input to the timing determination circuit 202.

[0049] Figure 10(A) shows an example of a pixel circuit configuration. The photoelectric conversion element 1 is a single-photon avalanche photodiode (SPAD), which generates charge pairs corresponding to incident light through photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the photoelectric conversion element 1. In addition, a voltage VH (second voltage), which is higher than the voltage VL supplied to the anode, is supplied to the cathode of the photoelectric conversion element 1. A reverse bias voltage (a voltage above the breakdown voltage) is supplied to the anode and cathode such that the photoelectric conversion element 1 performs avalanche multiplication. By supplying such voltages, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated.

[0050] When a reverse bias voltage is supplied, there are two modes of operation: Geiger mode, where the potential difference between the anode and cathode is greater than the breakdown voltage, and linear mode, where the potential difference between the anode and cathode is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD. For example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V. The APD may operate in linear mode or Geiger mode.

[0051] The signal processing circuit 201 includes a quench element 204 and a waveform shaping circuit 205. The quench element 204 is connected to a power supply that provides voltage VH and to the photoelectric conversion element 1. The quench element 204 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the photoelectric conversion element 1 and thereby suppressing avalanche multiplication (quench operation). The quench element 204 also functions to restore the voltage supplied to the photoelectric conversion element 1 to voltage VH by flowing the current that compensates for the voltage drop caused by the quench operation (recharge operation). The quench element 204 can be made of, for example, a PMOS transistor or an NMOS transistor. Figure 10(A) shows an example in which a switch such as a transistor is placed between the power supply and the photoelectric conversion element 1 to switch the electrical connection. The resistance of the quench element 204 may be a wiring resistance, and the resistance can be omitted from the equivalent circuit diagram.

[0052] The waveform shaping circuit 205 shapes the potential change of the cathode of the photoelectric conversion element 1 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping circuit 205. Figure 10 shows an example in which one inverter is used as the waveform shaping circuit 205, but a circuit in which multiple inverters are connected in series may be used, or other circuits that have a waveform shaping effect may be used.

[0053] The timing determination circuit 202 is connected to the waveform shaping circuit 205 and the counter circuit 203. The signal output from the waveform shaping circuit 205 and the pulse signal P_TCLK output from the selection circuit 211 are input, and a signal is output to the counter circuit 203 according to the combination of these signals. One example is an AND gate.

[0054] The logic circuit 210 has, for example, multiple threshold holding units. The thresholds can be set to predetermined values. Each of the multiple threshold holding units holds a different value as a reference threshold. The logic circuit 210 compares the value output from the counter circuit 203 with the thresholds held in the multiple threshold holding units and outputs the comparison result to the selection circuit 211. In Figures 9 and 10, three pulse signals P_TCLK1, P_TCLK2, and P_TCLK3 are input to the selection circuit 211, but at least two pulse signals are required to be input to the selection circuit 211, and there may be four or more.

[0055] Figure 10(B) shows the relationship between P_PCLK, V_ph (cathode potential of photoelectric element 1), and P_ph (output from waveform shaping circuit 205) during clock recharge drive. At time t1, P_PCLK transitions from L level to H level, and when the switch is turned ON, the cathode terminal of photoelectric element 1 is electrically connected to the power supply voltage, and a reverse bias is applied to photoelectric element 1. That is, when P_PCLK is at the H level and the switch is turned ON, it is in charge mode. Since the charge mode is repeated multiple times, it is also called recharge mode. When a reverse bias is applied, the cathode potential V_ph rises, and when it exceeds the judgment threshold, the output P_ph transitions from H level to L level. Subsequently, P_PCLK transitions from H level to L level, and photoelectric element 1 enters standby mode, waiting for photon incidence. When a photon is incident at time t2, the cathode potential V_ph decreases, and when it exceeds the threshold at time t3, the output P_ph transitions from L level to H level. Then, at time t4, P_PCLK transitions again from L level to H level, the switch turns ON, and the above operation is repeated.

[0056] This clock recharge drive is effective as a pile-up countermeasure because, even if many photons are incident on the standby mode, at least one output signal can be counted.

[0057] Figure 10(C) is a truth table showing the output P_sig for the inputs P_ph and P_TCLK to the timing determination circuit 202. 0 indicates a low level, and 1 indicates a high level. Since the timing determination circuit 202 is composed of AND gates, the output P_sig will be high only when both inputs P_ph and P_TCLK are high levels.

[0058] Figure 11 is a drive timing diagram illustrating the exposure period and pulse signals. One frame period consists of N subframes. The start and end of each subframe are defined by P_PCLK. Each subframe contains pulses selected from P_TCLK1 to P_TCLK3. The pulse of P_TCLK1 is included M times in one subframe, the pulse of P_TCLK2 is included N times in one subframe, and the pulse of P_TCLK3 is included O times in one subframe. In this case, the number of pulses satisfies the relationship M > N > O. This is input to the timing determination circuit 202. In Figure 11, P_TCLK is non-equally spaced, with intervals approximately proportional to the logarithm. As will be described later, P_TCLK may also be equally spaced. Alternatively, non-equally spaced pulses proportional approximately to the reciprocal may also be used.

[0059] Figure 11 is a concrete example of the concept in Figure 3 described above. The P_PCLK pulse is the starting point of a subframe, and the interval between subframes is T. acc This is how it works. P_ph transitions from L level to H level when a photon is incident, and transitions from H level to L level when the P_PCLK pulse is input due to the recharge operation of the photoelectric conversion element 1. P_TCLK is a pulse signal that defines the time information within the subframe.

[0060] According to the configuration of the timing determination circuit 202 described in Figure 10, the number of P_TCLK intervals since the first detection of a photon within the subframe is counted. The timing determination circuit 202 then outputs a pulse signal from the timing after the first detection of a photon within the subframe period.

[0061] In Figure 11, in the first subframe (A), the signal input to the counter is as shown in P_sig, and the counter counts to 6. Thus, when a photon is incident towards the beginning of the subframe, the counter's count value is large, and when a photon is incident towards the end of the subframe, the counter's count value is small. In other words, based on P_TCLK, it is possible to measure a value corresponding to the time from the start of the subframe until the first photon is detected.

[0062] By repeating this counting process N times and adding the count values, a count value corresponding to the expected detection time E' can be obtained. This makes it possible to estimate the number of incident photons from the expected value E', as explained above.

[0063] With this configuration, the number of incident photons during the exposure period can be estimated from the photon detection timing, making it possible to expand the dynamic range beyond the number of detected photons. Furthermore, since the length of the subframe period can be set by the photon exposure control unit, it is possible to adjust the counter so that it does not saturate during one frame period. In this way, by ensuring a wide dynamic range while avoiding the counter stopping during one frame period, signal information loss can be prevented.

[0064] Incidentally, in conventional clock recharge drive without the timing judgment circuit 202, the number of subframe periods becomes the maximum number that can be counted in one frame period. For example, if the number of subframes is N, the maximum count value is N, and up to N photons can be detected. Also, the dynamic range (defined as the maximum output value) is N. Basically, since power consumption is proportional to the number of detections, if you try to reduce the number of detections to lower power consumption, the dynamic range decreases, and if you try to increase the dynamic range, power consumption increases. In other words, there is a trade-off relationship between dynamic range and power consumption.

[0065] On the other hand, in the photoelectric converter according to this embodiment, which includes the timing determination circuit 202, the estimated number of photons corresponds to the dynamic range. Power consumption is proportional to the number of detections, as in the conventional method, but the estimated number of photons does not depend on the number of photon detections, thus eliminating the trade-off relationship between dynamic range and power consumption.

[0066] Furthermore, in this embodiment, since the number of P_TCLK pulses is changed according to the count value, it is possible to further expand the dynamic range without increasing the number of bits.

[0067] For example, in Figure 10, the threshold value of threshold holding unit 1 is set to "14", the threshold value of threshold holding unit 2 is set to "126", and the threshold value of threshold holding unit 3 is set to "510". In the first subframe, the selection circuit 211 selects P_TCLK1 and inputs P_TCLK1 as P_TCLK to the timing determination circuit 202.

[0068] Within a single frame, if the pixel value output from the counter circuit 203 of the previous subframe is between 0 and 14, P_TCLK1 is input to the timing determination circuit 202 as P_TCLK in the next subframe as well.

[0069] Within a single frame, if the pixel value output from the counter circuit 203 of the previous subframe is greater than a predetermined value of 14 (15-126), then P_TCLK2 is input to the timing determination circuit 202 as P_TLCK in the next subframe. P_TCLK2 has fewer pulses per subframe than P_TCLK1.

[0070] Within a single frame, if the pixel value output from the counter circuit 203 of the previous subframe is between 127 and 510, P_TCLK3 is input to the timing determination circuit 202 as P_TLCK in the next subframe. P_TCLK3 has fewer pulses per subframe than P_TCLK2.

[0071] Within a single frame, if the pixel value output from the counter circuit 203 of the previous subframe is greater than 511, P_TCLK4 is input to the timing determination circuit 202 as P_TLCK in the next subframe. P_TCLK4 has fewer pulses per subframe than P_TCLK3.

[0072] This configuration makes it possible to expand the dynamic range without increasing the number of bits in the counter circuit 203 included in the measurement unit 4. Next, we will explain the case of imaging a scene in which the brightness is high only for a short period within one frame period, and low for the rest of the time. In subframes with high brightness, the count is smaller than the actual number of photons entering. In subframes with low brightness, the count is assumed to be close to the actual number of photons entering. Since the correction processing in the linear correction unit is performed on the premise that the number of photons entering is uniform throughout one frame period, the effect of missing photons in high-brightness subframes becomes large, and the corrected pixel value becomes lower than the brightness visible to the human eye. To mitigate this problem, by increasing the count value in each subframe during periods when the pixel value within one frame is low, a sufficient count value can be obtained even in scenes in which the brightness is high only for a short period within one frame period.

[0073] In Figure 11, within a single frame, the first subframe inputs P_TCLK1 as P_TLCK to the timing determination circuit 202, but this is not limited to this. For example, within a single frame, the first subframe can also input P_TCLK2 as P_TLCK to the timing determination circuit 202. In this case, if the pixel value output from the counter circuit 203 is less than a predetermined value of 15 (i.e., 14 or less), P_TCLK1 can be input as P_TLCK to the timing determination circuit 202 in the next subframe.

[0074] (P_TCLK with equal and unequal intervals) Figure 12 is a conceptual diagram comparing the cases where P_TCLK is equally spaced and unequally spaced. The upper part of Figure 13(A) shows an example where there are 16 equally spaced pulses in one subframe. Equally spaced pulses can be said to correspond to real time. The lower part shows an example where unequally spaced pulses are approximately logarithmic (base 2) of real time. The reason for using "approximately" is that the logarithm of 1 with base 2 is 0, and the logarithm of 2 with base 2 is 1. In Figure 13(A), 1 is added to make them 1 and 2 respectively in order to compare with equally spaced pulses. Such unequally spaced pulse intervals are described as being set so that the pulse space is logarithmically compressed real space. The intervals of unequally spaced pulse signals can be configured so that the period increases according to the elapsed time in the subframe.

[0075] Figure 13(B) is a graph showing the relationship between the number of incident photons (input) and the count value (output) for equally spaced pulses. As shown in this figure, when comparing with the same number of incident photons (input), the output of non-equally spaced pulses is smaller than that of equally spaced pulses. In other words, non-equally spaced pulses can determine the same number of incident photons with fewer count values. For this reason, the counter circuit can be made smaller with non-equally spaced pulses, and the pixel circuit area can be reduced. On the other hand, with equally spaced pulses, the average interval between pulses can be made smaller, so the signal-to-noise ratio (S / N) is relatively higher compared to non-equally spaced pulses.

[0076] (Second Embodiment) This embodiment describes a configuration example that includes a pixel circuit different from that of the first embodiment. Since the configuration is substantially the same as that of the first embodiment, a description of the other aspects will be omitted.

[0077] The difference between the pixel circuit of this embodiment shown in Figure 14(A) and the pixel circuit shown in Figure 10 is that the switch to which P_PCLK is input, which was provided between the power supply voltage and the photoelectric conversion element 1, is eliminated, and a latch circuit 206 is added to the signal processing circuit 201. P_PCLK is configured to be input to the latch circuit 206. Even in the case shown in Figure 14, any element that functions as a resistor can be provided as the quench element 204, so not only a metal resistor but also a transistor or the like may be used as the quench element 204.

[0078] Figure 14(B) shows the change in the cathode potential V_ph of the photoelectric conversion element 1 shown in Figure 14(A). The difference from Figure 10(B) is that after the cathode potential drops due to photon incidence, the voltage autonomously returns to its initial state. This pixel operation is called so-called passive operation.

[0079] Figure 15 is a drive timing diagram according to this embodiment. In Figure 15, one of P_TCLK1 to 3 is indicated as P_TCLK. The difference from Figure 11 is that P_ph repeats High and Low multiple times in response to photon incidence within one subframe. However, since P_out, which is output from the latch circuit 206 and input to the timing determination circuit, matches P_ph in Figure 11, the resulting output P_sig is the same. In this way, the same effects as in the first embodiment can be obtained even with a passively driven ADP element.

[0080] In other words, with this configuration, the number of incident photons during the exposure period can be estimated from the timing of photon detection, making it possible to expand the dynamic range beyond the number of detected photons. Furthermore, since the length of the subframe period can be set by the photon exposure control unit, it is possible to adjust the counter so that it does not saturate during one frame period. Therefore, by ensuring a wide dynamic range while avoiding the counter stopping during one frame period, signal information loss can be prevented.

[0081] Furthermore, with this configuration, the estimated number of photons corresponding to the dynamic range does not depend on the number of photon detections corresponding to the power consumption, thus eliminating the trade-off relationship between dynamic range and power consumption. (Third embodiment) This embodiment describes a configuration example that includes a pixel circuit different from the embodiment described above.

[0082] The difference between the pixel circuit of this embodiment shown in Figure 16 and the pixel circuit shown in Figure 10 is the addition of a signal selection circuit 207 and a signal holding circuit 208. By adding these circuits, it becomes possible to switch the signal input to the counter circuit in the M subframe between P_out and P_ph by referring to the information on whether or not photons were incident in the M-1 subframe. Since the output of the subsequent subframe is changed according to the result of the previous subframe, this circuit is referred to here as a time correlation filter.

[0083] Figure 17 shows the operation sequence of this embodiment. In the M subframe, no photons are incident, so P_ph is at the L level. This L level information is held by the signal holding circuit 208.

[0084] Next, when P_PCLK, which defines the start of the M+1 subframe, transitions from a low level to a high level, a low-level signal is output from the signal holding circuit 208, and this low-level signal is input to the signal selection circuit 207. As a result, the signal selection circuit 207 does not output P_out as P_sig, but instead outputs P_ph as P_sig. When a photon is incident in the M+1 subframe, P_ph transitions from a low level to a high level, and therefore P_sig also transitions from a low level to a high level. Also, because P_ph transitions from a low level to a high level, the high-level information is held by the signal holding circuit 208.

[0085] Next, when P_PCLK, which defines the start of the M+2 subframe, transitions from L level to H level, P_ph transitions from H level to L level, and P_sig also transitions from H level to L level. Furthermore, due to the level transition of P_PCLK, an H level signal is output from the signal holding circuit 208, and the signal selection circuit 207 is configured to output P_out as P_sig. In other words, the signal selection circuit 207 is configured to output a signal corresponding to the pulse signal that defines the time information within the M+2 subclaim period. When a photon is incident in the M+2 subframe, P_ph transitions from L level to H level, so the signal P_out is output as P_sig.

[0086] Figure 18 shows the operation sequence to explain the effect of the time correlation filter. The time correlation filter is effective when dealing with signals of approximately one count or less per subframe. This is because, as the incidence frequency of photons decreases, the probability of a photon occurring becomes constant regardless of time, based on an exponential distribution function. In other words, the probability of entering each time window becomes approximately the same.

[0087] In Figure 18, one photon is incident on two subframes. When a photon is incident at the timing shown in Figure 18, P_sig counts to 1 with a time correlation filter and to 3 without a time correlation filter.

[0088] When dealing with signals of approximately 1 count or less per subframe, adopting a configuration without a time correlation filter results in counting a higher number of incident photons than the actual number, leading to larger errors in low-light regions. In other words, it becomes a source of increased noise. In contrast, in this embodiment, when dealing with signals of approximately 1 count or less per subframe, only one count is recorded unless a signal is detected continuously, thus suppressing errors and reducing the source of noise increase. Therefore, the signal-to-noise ratio in the low-power region can be improved. Furthermore, since the dark count rate (DCR) in dark conditions is also approximately 1 count or less per subframe, it can be reduced by the same principle by employing this time correlation filter.

[0089] In addition, the above example circuit describes one that outputs 1 count unless a signal is continuously detected, but it may also be a circuit that outputs a predetermined value, and the predetermined value may be a circuit that outputs a value of 1 count or more.

[0090] (Fourth Embodiment) A fourth embodiment will be described using Figures 19 and 20. This embodiment differs in that it has a three-layer laminated structure.

[0091] Figure 19 shows the addition of a third substrate 31 and a second circuit region 32.

[0092] Figure 20 shows a conceptual diagram of the pixel circuit layout in a three-layer stacked structure. Compared to the pixel circuit in Figure 10, a second timing determination circuit 302, a second counter circuit 303, a second logic circuit 310 which is a second pulse control unit, and a second selection circuit 311 have been added, and these circuits are located on the third substrate 31.

[0093] The second timing determination circuit 302 is configured to receive the pulse signal selected by the second selection circuit 311 and the output from the waveform shaping circuit 205. In Figure 20, it is assumed that the vertical scanning circuit section 110 and the control pulse generation section 115 are provided on the second board 21, so the configuration shows that P_TCLK1~4 are input from the circuits provided on the second board 21 to the third board 31. Since the third board 31 does not have a quench element 204 and a waveform shaping circuit 205, it has more space than the second board 21. Therefore, it is also possible to provide the vertical scanning circuit section 110 and the control pulse generation section 115 on the third board 31. In this case, P_PCLK and P_TCLK1~4 may be configured to be input from the third board 31 to the second board 21.

[0094] Furthermore, in this embodiment, the threshold values ​​set by the threshold holding units 1 to 3 of the logic circuit 210 and the threshold values ​​set by the second logic circuit 310 may be different. For example, the threshold values ​​may be set lower in the threshold holding units 1 to 3 of the logic circuit 210, and higher in the threshold holding units 1 to 3 of the second logic circuit 310 than in the threshold holding units of the logic circuit 210. If the two counters have different numbers of bits, setting different threshold values ​​for each makes it possible to effectively use the implemented bit width.

[0095] In this embodiment, because a three-layer stacked structure is used, multiple circuits can be easily parallelized, enabling high functionality. Specifically, since timing determination circuits 202 and 302 are provided in parallel, two outputs with different count values ​​can be obtained from a single photon detection signal using P_TCLK1 and P_TCLK1, which have different signal waveforms. The advantages and disadvantages of various characteristics (dynamic range, signal-to-noise ratio, optimal exposure, power consumption, etc.) vary depending on the number of pulses and pulse interval of P_TCLK. Therefore, having two outputs with different characteristics allows for the selection of the optimal output according to the imaging scene, and it is also possible to combine them to generate an image with better image quality.

[0096] For example, when performing measurements with a single P_TCLK as in the first embodiment, a problem is anticipated where measurement errors become large when capturing scenes in which the amount of light changes rapidly within a single frame. This is because the difference in the measured timing of photon detection in each subframe becomes larger compared to when average light is incident within a single frame.

[0097] One way to solve this problem is to acquire and correct signals in parallel that can estimate the change in light intensity within a single frame. Specifically, this can be achieved by setting P_TCLK2 to count only the first pulse, so that only incident photons with an early detection timing are counted.

[0098] The threshold values ​​of the threshold holding units 1-3 of the logic circuit 210 and the threshold values ​​of the threshold holding units 1-3 of the second logic circuit 310 may be set to the same value.

[0099] (Fifth embodiment) A photoelectric conversion system using the photoelectric conversion device according to each embodiment described above will be explained with reference to Figure 21. Figure 21 is a block diagram showing the schematic configuration of the photoelectric conversion system according to this embodiment.

[0100] The processing apparatus according to this embodiment includes a control unit 401, a timing adjustment unit 402, an image acquisition unit 403, a reading unit 404, a gain adjustment unit 405, a nonlinear correction unit 406, a scratch correction unit 407, a data compression unit 408, and a storage unit 409.

[0101] The image acquisition unit 403 is, for example, a pixel circuit, and the readout unit 404 is, for example, located after the counter circuit 203. The control unit 401 may be an internal control unit of the photoelectric converter or it may be external to the photoelectric converter. The image acquisition unit 403 is controlled by a timing adjustment unit 402 controlled by the control unit 401. The image data generated by the image acquisition unit is input to the storage unit 409 after correction processing. Note that the order of the correction processing is not limited to the order shown in Figure 25.

[0102] The gain adjustment unit 405 is located between the readout unit 404 and the nonlinear correction unit 406, and applies a digital gain to the image data generated by the image acquisition unit 403. While image correction data often contains decimal values, if the image output is an integer, quantization errors can reduce the accuracy of the correction. By applying a gain to the image data beforehand, the effects of quantization errors can be suppressed, improving the accuracy of the correction. Here, if the quantization error can be suppressed to less than 1 / 4 of the single-photon signal level, the corrected image will appear visually natural. Therefore, it is desirable that the digital gain applied to the image data be, for example, 4 times or more.

[0103] The nonlinear correction unit 406 is positioned between the gain adjustment unit 405 and the scratch correction unit 407, and corrects the image data under the control of the control unit 401. When the image acquisition unit 403 is a photon counting type detector, the optical response often becomes nonlinear due to the effect of dead time. When affected by a nonlinear optical response, overcorrection may occur if correction is performed assuming a linear response. Therefore, overcorrection can be prevented by performing nonlinear correction on the image data before the calculation processing in the scratch correction unit 407, and appropriate nonlinear correction can be performed according to the drive timing. This nonlinear correction is performed, for example, using a LookUpTable.

[0104] The nonlinear correction unit 406 has a correction circuit that corrects the signal output from the image acquisition unit 403. The correction circuit changes the correction means according to the number of pulse signals selected by the selection circuit 211 at each pixel. For example, if the pulse signal P_TCLK2 is selected in a subframe, the pixel value is corrected so that the correction amount is larger than when the pulse signal P_TCLK1 is selected. Also, if the pulse signal P_TCLK1 is selected in a subframe, the pixel value is corrected so that the correction amount is smaller than when the pulse signal P_TCLK2 is selected. In subframes, the number of pulses input decreases at high illumination, which may result in the output being lower than the original brightness at high illumination. According to this embodiment, by performing appropriate correction, it is possible to expand the dynamic range at high illumination.

[0105] The scratch correction unit 407 corrects the data of scratched pixels included in the image data. Specifically, it extracts the output value of the scratched pixel and identifies the position information and output value of the scratched pixel. Methods include replacing the identified scratched pixel with the average or median value of the output of pixels surrounding it, or dividing it by the estimated scratched image data.

[0106] The data compression unit 408 compresses the corrected image data. The photoelectric conversion device according to the present invention generates a vast amount of image data that corresponds to a high dynamic range. By providing the data compression unit 408, the data can be compressed before being stored in the subsequent storage unit 409.

[0107] The memory unit 409 is a memory unit that holds at least a portion of the image data generated in the preceding stage. Specifically, the image data is stored in a memory such as SRAM, DRAM, or non-volatile memory.

[0108] Thus, according to this embodiment, a photoelectric conversion system can be realized by applying the photoelectric conversion device shown in any of the embodiments described above.

[0109] (Sixth Embodiment) The effects of the present invention will be further explained using Figure 22. In the following description, the case in which addition is performed according to each of the above embodiments will be referred to as the weighted counting method.

[0110] In the conventional clock recharge method shown in Figure 22(a), the maximum estimated number of incident photons is determined by the number of Recharge CLK (P_PCLK) cycles. That is, the more Recharge CLK cycles there are, the wider the dynamic range. On the other hand, power consumption increases proportionally to the number of Recharge CLK cycles, so there is a trade-off between dynamic range and power consumption.

[0111] If the exposure period is T and the pulse interval of Recharge CLK is Δtr, then the number of Recharge CLK pulses is expressed as T / Δtr, and therefore, the dynamic range can be considered to be determined by T / Δtr.

[0112] In contrast, as shown in Figure 22(b), the weighted counting method estimates the number of incident photons from the timing of the photon incidence. Therefore, the maximum estimated number of incident photons is determined by T / Δtw. Here, Δtw is the period from the pulse of Recharge CLK (P_PCLK in this embodiment) until the pulse of the first P_TCLK is input. In other words, the maximum estimated number of incident photons no longer depends on the Recharge CLK interval Δtr, and the dynamic range and power consumption are no longer in a trade-off relationship. In this case, the weighting coefficient (the increase in the count value corresponding to the incidence of 1 photon) is preferably Δtr / Δtw.

[0113] Specific numerical examples are shown below. In the conventional method, when the exposure period T is 1024 and Δtr is 1, the maximum number of avalanches is 1024, and the maximum number of incident photons detected at this time is 1024. On the other hand, in the weighted counting method, when the exposure period T is 1024, Δtr is 4, and Δtw is 1, the maximum number of avalanches is 256. If the count corresponding to the photons incident at Δtw is added to the counter circuit so that it is Δtr / Δtw = 4, the maximum estimated number of incident photons at this time will be 1024. In other words, it is possible to achieve the same dynamic range while reducing the power consumption associated with recharging to 1 / 4 of that of the conventional method. In each of the embodiments described above, adding 4 at the timing of Δtw can be achieved, for example, by inputting 4 pulses of P_TCLK for a time sufficiently shorter than Δtw.

[0114] The preference for using Δtr / Δtw as the weighting coefficient is that it allows the saturation count to match under driving conditions where both equations have equivalent dynamic ranges. This condition helps reduce false signals during nonlinear correction. However, since power consumption depends only on Δtr and not on the weighting coefficient, it is not always necessary to satisfy the above relationship from the perspective of power consumption reduction.

[0115] (Seventh Embodiment) The photoelectric conversion system according to this embodiment will be described with reference to Figure 23. Figure 23 is a block diagram showing the schematic configuration of the photoelectric conversion system according to this embodiment.

[0116] The photoelectric conversion device described in the above embodiment is applicable to various photoelectric conversion systems. Examples of applicable photoelectric conversion systems include digital still cameras, digital camcorders, surveillance cameras, photocopiers, fax machines, mobile phones, in-vehicle cameras, and observation satellites. Camera modules, which include optical systems such as lenses and imaging devices, are also included in photoelectric conversion systems. Figure 23 shows a block diagram of a digital still camera as an example of these.

[0117] The photoelectric conversion system illustrated in Figure 23 includes an imaging device 1004, which is an example of a photoelectric conversion device, and a lens 1002 that forms an optical image of a subject onto the imaging device 1004. The photoelectric conversion system further includes an aperture 1003 for varying the amount of light passing through the lens 1002, and a barrier 1001 for protecting the lens 1002. The lens 1002 and aperture 1003 are an optical system that focuses light onto the imaging device 1004. The imaging device 1004 is a photoelectric conversion device of any of the above embodiments, which converts the optical image formed by the lens 1002 into an electrical signal.

[0118] The photoelectric conversion system includes a signal processing unit 1007, which is an image generation unit that generates an image by processing the output signal output from the imaging device 1004. The signal processing unit 1007 performs various corrections and compressions as needed and outputs the image data. The signal processing unit 1007 may be formed on the same semiconductor layer as the imaging device 1004, or it may be formed on a different semiconductor layer from the imaging device 1004. Alternatively, the imaging device 1004 and the signal processing unit 1007 may be formed on the same semiconductor layer.

[0119] The photoelectric conversion system further includes a memory unit 1010 for temporarily storing image data, and an external interface unit (external I / F unit) 1013 for communicating with an external computer or the like. Furthermore, the photoelectric conversion system includes a recording medium 1012 such as a semiconductor memory for recording or reading imaging data, and a recording medium control interface unit (recording medium control I / F unit) 1011 for recording or reading data from the recording medium 1012. The recording medium 1012 may be built into the photoelectric conversion system or may be detachable.

[0120] Furthermore, the photoelectric conversion system includes an overall control / calculation unit 1009 that controls various calculations and the entire digital still camera, and a timing generation unit 1008 that outputs various timing signals to the imaging device 1004 and the signal processing unit 1007. Here, the timing signals and the like may be input from an external source, and the photoelectric conversion system only needs to have at least the imaging device 1004 and the signal processing unit 1007 that processes the output signals output from the imaging device 1004.

[0121] The imaging device 1004 outputs the imaging signal to the signal processing unit 1007. The signal processing unit 1007 performs predetermined signal processing on the imaging signal output from the imaging device 1004 and outputs image data. The signal processing unit 1007 generates an image using the imaging signal.

[0122] Thus, according to this embodiment, a photoelectric conversion system can be realized by applying a photoelectric conversion device (imaging device) of any of the above embodiments.

[0123] (Eighth embodiment) The photoelectric conversion system and mobile unit of this embodiment will be described with reference to Figure 24. Figure 24 is a diagram showing the configuration of the photoelectric conversion system and mobile unit of this embodiment.

[0124] Figure 24(a) shows an example of a photoelectric conversion system related to an in-vehicle camera. The photoelectric conversion system 2300 has an imaging device 2310. The imaging device 2310 is a photoelectric conversion device as described in any of the embodiments above. The photoelectric conversion system 2300 has an image processing unit 2312 that performs image processing on a plurality of image data acquired by the imaging device 2310. The photoelectric conversion system 2300 also has a parallax acquisition unit 2314 that calculates parallax (phase difference of parallax images) from a plurality of image data acquired by the photoelectric conversion system 2300. Furthermore, the photoelectric conversion system 2300 has a distance acquisition unit 2316 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 2318 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 2314 and the distance acquisition unit 2316 are examples of distance information acquisition means that acquire distance information to an object. That is, distance information may be acquired not only by phase difference but also by ToF (Time Of Flight) technology. The collision determination unit 2318 may use any of this distance information to determine the possibility of a collision. The means for acquiring distance information may be implemented by specially designed hardware or by a software module. It may also be implemented by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination thereof.

[0125] The photoelectric conversion system 2300 is connected to the vehicle information acquisition device 2320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 2300 is also connected to the control ECU 2330, which is a control device (control unit) that outputs a control signal to generate braking force on the vehicle based on the judgment result of the collision judgment unit 2318. The photoelectric conversion system 2300 is also connected to the warning device 2340, which issues a warning to the driver based on the judgment result of the collision judgment unit 2318. For example, if the collision judgment unit 2318 determines that there is a high probability of collision, the control ECU 2330 performs vehicle control to avoid a collision or mitigate damage by applying the brakes, releasing the accelerator, or suppressing engine output. The warning device 2340 warns the user by sounding an alarm, displaying warning information on a screen such as a car navigation system, or vibrating the seat belt or steering wheel.

[0126] In this embodiment, the photoelectric conversion system 2300 images the area around the vehicle, for example, the front or rear. Figure 24(b) shows the photoelectric conversion system when imaging the area in front of the vehicle (imaging range 2350). The vehicle information acquisition device 2320 sends instructions to the photoelectric conversion system 2300 or the imaging device 2310. This configuration can further improve the accuracy of distance measurement.

[0127] The above example illustrates control to prevent collisions with other vehicles, but it can also be applied to control systems that automatically follow other vehicles or automatically drive to prevent vehicles from straying from their lanes. Furthermore, the photoelectric conversion system can be applied not only to vehicles such as the vehicle itself, but also to mobile objects (mobile devices) such as ships, aircraft, or industrial robots. In addition, it can be applied not only to mobile objects but also to a wide range of devices that utilize object recognition, such as intelligent transportation systems (ITS).

[0128] (Ninth Embodiment) The photoelectric conversion system of this embodiment will be explained with reference to Figure 25. Figure 25 is a block diagram showing an example configuration of a distance image sensor, which is a photoelectric conversion system.

[0129] As shown in Figure 25, the distance image sensor 1401 is configured to include an optical system 1402, a photoelectric converter 1403, an image processing circuit 1404, a monitor 1405, and a memory 1406. The distance image sensor 1401 receives light (modulated light or pulsed light) that is projected from the light source device 1411 toward the subject and reflected from the surface of the subject, thereby acquiring a distance image corresponding to the distance to the subject.

[0130] The optical system 1402 is composed of one or more lenses and guides the image light (incident light) from the subject to the photoelectric converter 1403, where it forms an image on the light-receiving surface (sensor part) of the photoelectric converter 1403.

[0131] As the photoelectric converter 1403, the photoelectric converter described in the above embodiment is applied, and a distance signal indicating the distance obtained from the received light signal output from the photoelectric converter 1403 is supplied to the image processing circuit 1404.

[0132] The image processing circuit 1404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric converter 1403. The distance image (image data) obtained through this image processing is then supplied to the monitor 1405 for display or supplied to the memory 1406 for storage (recording).

[0133] With the distance image sensor 1401 configured in this way, by applying the photoelectric conversion device described above, the characteristics of the pixels are improved, and for example, more accurate distance images can be acquired.

[0134] (Tenth embodiment) The photoelectric conversion system of this embodiment will be described with reference to Figure 26. Figure 26 is a diagram showing an example of a schematic configuration of an endoscopic surgical system, which is the photoelectric conversion system of this embodiment.

[0135] Figure 26 illustrates a surgeon (physician) 1131 performing surgery on a patient 1132 on a patient bed 1133 using an endoscopic surgical system 1103. As shown in the figure, the endoscopic surgical system 1103 consists of an endoscope 1100, surgical instruments 1110, and a cart 1134 equipped with various devices for endoscopic surgery.

[0136] The endoscope 1100 consists of a barrel 1101, the tip of which is inserted into the body cavity of the patient 1132 for a predetermined length, and a camera head 1102 connected to the base end of the barrel 1101. In the illustrated example, the endoscope 1100 is shown as a so-called rigid endoscope having a rigid barrel 1101, but the endoscope 1100 may also be configured as a so-called flexible endoscope having a flexible barrel.

[0137] An opening into which an objective lens is fitted is provided at the tip of the endoscope tube 1101. A light source device 1203 is connected to the endoscope 1100, and the light generated by the light source device 1203 is guided to the tip of the endoscope tube by a light guide extending inside the endoscope tube 1101, and is irradiated through the objective lens towards the object to be observed inside the body cavity of the patient 1132. The endoscope 1100 may be a straight-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0138] The camera head 1102 contains an optical system and a photoelectric converter. Reflected light from the object being observed (observation light) is focused by the optical system into the photoelectric converter. The photoelectric converter converts the observation light into electrical signals, generating an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The photoelectric converter can be the one described in the above embodiment. The image signal is transmitted as RAW data to the camera control unit (CCU) 1135.

[0139] The CCU1135 consists of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and other components, and comprehensively controls the operation of the endoscope 1100 and the display device 1136. Furthermore, the CCU1135 receives an image signal from the camera head 1102 and performs various image processing operations on that image signal, such as development processing (demosaic processing), to display the image based on that image signal.

[0140] The display device 1136 displays an image based on an image signal that has been processed by the CCU 1135, under control from the CCU 1135.

[0141] The light source device 1203 consists of a light source such as an LED (Light Emitting Diode) and supplies illumination light to the endoscope 1100 when photographing the surgical area, etc.

[0142] The input device 1137 is an input interface for the endoscopic surgical system 1103. The user can input various types of information and instructions to the endoscopic surgical system 1103 via the input device 1137.

[0143] The treatment instrument control device 1138 controls the driving of the energy treatment instrument 1112 for purposes such as tissue cauterization, incision, or blood vessel sealing.

[0144] The light source device 1203, which supplies illumination light to the endoscope 1100 when photographing the surgical area, can be composed of, for example, an LED, a laser light source, or a combination thereof. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so the white balance of the captured image can be adjusted in the light source device 1203. In this case, it is also possible to capture images corresponding to each of the RGB colors in time-division by irradiating the observation target with laser light from each of the RGB laser light sources in time-division and controlling the drive of the image sensor of the camera head 1102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter on the image sensor.

[0145] Furthermore, the light source device 1203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the drive of the image sensor of the camera head 1102 in synchronization with the timing of the change in light intensity, images can be acquired in time-division order, and these images can be combined to generate high dynamic range images without so-called black crushing and white clipping.

[0146] Furthermore, the light source device 1203 may be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the wavelength dependence of light absorption in body tissue is utilized. Specifically, by irradiating with narrowband light compared to the irradiation light used during normal observation (i.e., white light), predetermined tissues such as blood vessels on the surface of mucosa can be imaged with high contrast. Alternatively, in special light observation, fluorescence observation may be performed to obtain an image from fluorescence generated by irradiation with excitation light. In fluorescence observation, excitation light can be irradiated onto body tissue and fluorescence from the body tissue can be observed, or a reagent such as indocyanine green (ICG) can be injected into body tissue and excitation light corresponding to the fluorescence wavelength of the reagent can be irradiated onto the body tissue to obtain a fluorescence image. The light source device 1203 may be configured to supply narrowband light and / or excitation light corresponding to such special light observation.

[0147] (11th embodiment) The photoelectric conversion system of this embodiment will be described with reference to Figure 27. Figure 27(a) is a diagram showing an example of the configuration of eyeglasses 1600 (smart glasses), which are a photoelectric conversion system. The eyeglasses 1600 have a photoelectric conversion device 1602. The photoelectric conversion device 1602 is the photoelectric conversion device described in the 12th embodiment above. In addition, a display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 1601. There may be one or more photoelectric conversion devices 1602. In addition, multiple types of photoelectric conversion devices may be used in combination. The arrangement position of the photoelectric conversion device 1602 is not limited to that shown in Figure 27(a).

[0148] The eyeglasses 1600 further include a control device 1603. The control device 1603 functions as a power source that supplies power to the photoelectric converter 1602 and the display device. The control device 1603 also controls the operation of the photoelectric converter 1602 and the display device. The lens 1601 has an optical system formed therein for focusing light onto the photoelectric converter 1602.

[0149] Figure 27(b) illustrates a pair of glasses 1610 (smart glasses) according to one application example. The glasses 1610 have a control device 1612, which is equipped with a photoelectric converter equivalent to a photoelectric converter 1602 and a display device. The lens 1611 has an optical system formed therein for projecting light emitted from the photoelectric converter in the control device 1612 and from the display device, and an image is projected onto the lens 1611. The control device 1612 functions as a power supply that provides power to the photoelectric converter and the display device, and also controls the operation of the photoelectric converter and the display device. The control device may have a gaze detection unit that detects the wearer's gaze. Gaze detection may use infrared light. The infrared light emitter emits infrared light towards the eyeball of the user who is fixating on the displayed image. An imaging unit having a photoelectric conversion element detects the reflected light from the eyeball of the emitted infrared light, thereby obtaining an image of the eyeball. By having a reduction means that reduces the light from the infrared light emitter to the display unit in planar view, the deterioration of image quality is reduced.

[0150] The user's gaze towards a displayed image is detected from an image of the eyeball obtained by imaging with infrared light. Any known method can be applied to gaze detection using an image of the eyeball. As an example, a gaze detection method based on the Purkinje image obtained by the reflection of the irradiated light from the cornea can be used.

[0151] More specifically, gaze detection processing is performed based on the pupil-corneal reflection method. Using the pupil-corneal reflection method, a gaze vector representing the orientation (rotation angle) of the eyeball is calculated based on the pupil image and Purkinje image contained in the captured image of the eyeball, thereby detecting the user's gaze.

[0152] The display device of this embodiment includes a photoelectric conversion device having a photoelectric conversion element, and may control the display image of the display device based on the user's gaze information from the photoelectric conversion device.

[0153] Specifically, the display device determines a first field of view that the user is fixated on, and a second field of view other than the first field of view, based on gaze information. The first and second field of view may be determined by the control device of the display device, or they may be determined by an external control device and received by the display device. Within the display area of ​​the display device, the display resolution of the first field of view may be controlled to be higher than the display resolution of the second field of view. In other words, the resolution of the second field of view may be lower than that of the first field of view.

[0154] Furthermore, the display area has a first display area and a second display area different from the first display area, and a higher priority area may be determined from the first and second display areas based on gaze information. The first and second view areas may be determined by the control device of the display device, or they may be determined by an external control device and received. The resolution of the higher priority area may be controlled to be higher than the resolution of the areas other than the higher priority area. In other words, the resolution of areas with relatively lower priority may be set lower.

[0155] AI may be used to determine the first field of view area and high-priority areas. The AI ​​may be a model configured to estimate the angle of gaze and the distance to the target object at the end of the line of sight from the image of the eye, using the image of the eye and the direction the eye was actually looking in that image as training data. The AI ​​program may be installed in the display device, the photoelectric converter, or an external device. If installed in an external device, it will be transmitted to the display device via communication.

[0156] When display control is based on visual detection, this method is preferably applicable to smart glasses that further include a photoelectric converter for capturing images of the surrounding environment. The smart glasses can display the captured external information in real time.

[0157] The embodiments described above can be modified as appropriate without departing from the technical concept. Furthermore, examples in which some components of one embodiment are added to other embodiments, or in which some components of other embodiments are replaced, are also included as embodiments of the present invention.

[0158] Furthermore, the disclosure of this embodiment includes the following configurations and methods.

[0159] (Composition 1) A photoelectric conversion element that receives photons, An exposure control unit that generates signals defining a plurality of second exposure periods that are included in a first exposure period corresponding to one frame and are shorter than the first exposure period, A timing generation unit that generates a pulse signal defining time information within the second exposure period, A measurement unit measures the number of pulse signals since the first detection of a photon during the second exposure period, based on the pulse signal generated by the timing generation unit. A pulse control unit controls the number of pulse signals in the second exposure period following the second exposure period based on the value of the measurement unit, A photoelectric converter having

[0160] (Configuration 2) The photoelectric conversion apparatus according to configuration 1, characterized in that the pulse control unit selects a pulse signal from among a plurality of pulse signals and controls the number of pulse signals during the next second exposure period.

[0161] (Composition 3) The photoelectric conversion device according to configuration 1 or 2, characterized in that when the value of the measurement unit is smaller than a predetermined value, the number of pulse signals in the next second exposure period is made greater than the number of pulse signals in the second exposure period.

[0162] (Composition 4) The photoelectric conversion device according to any one of configurations 1 to 3, characterized in that when the value of the measurement unit is greater than a predetermined value, the number of pulse signals in the next second exposure period is reduced to less than the number of pulse signals in the second exposure period.

[0163] (Composition 5) The timing generation unit generates pulse signals at equal intervals, The photoelectric converter according to any one of configurations 1 to 4, wherein the pulse signal is configured to be input to the measuring unit at least during the second exposure period.

[0164] (Composition 6) The timing generation unit generates non-equally spaced pulse signals, The photoelectric converter according to any one of configurations 1 to 4, wherein the pulse signal is configured to be input to the measuring unit at least during the second exposure period.

[0165] (Composition 7) The photoelectric conversion apparatus according to configuration 6, wherein the non-equally spaced pulse signals are configured such that their period increases in proportion to the elapsed time of the second exposure period.

[0166] (Composition 8) The photoelectric converter according to configuration 6 or 7, characterized in that the intervals between the non-equally spaced pulse signals are set to be logarithmically compressed in real time.

[0167] (Composition 9) It has a correction circuit that corrects the signal output from the measurement unit, The photoelectric converter according to any one of configurations 1 to 8, characterized in that the correction circuit changes the correction means based on the number of pulse signals.

[0168] (Composition 10) The photoelectric conversion device according to configuration 9, characterized in that the correction circuit changes the value output from the measurement unit to a larger value when the number of pulse signals in the next second exposure period is less than the number of pulse signals in the second exposure period.

[0169] (Composition 11) The photoelectric conversion device according to any one of configurations 1 to 10, characterized in that the measurement unit includes a waveform shaping circuit that converts the signal from the photoelectric conversion element into a pulse signal.

[0170] (Composition 12) The measurement unit includes a timing determination circuit, The photoelectric conversion apparatus according to any one of configurations 1 to 11, characterized in that the timing determination circuit outputs a pulse signal generated by the timing generation unit after the timing at which a photon is first detected within the second exposure period.

[0171] (Composition 13) The measurement unit includes a counter circuit, The photoelectric converter according to configuration 12, wherein the counter circuit counts each pulse signal output from the timing determination circuit during the second exposure period to obtain a count value.

[0172] (Composition 14) The photoelectric conversion device according to configuration 13, characterized in that the counter circuit adds up and outputs the count values ​​for the plurality of second exposure periods.

[0173] (Composition 15) The photoelectric conversion device according to any one of claims 1 to 14, characterized in that the photoelectric conversion element is an avalanche photodiode.

[0174] (Composition 16) The photoelectric conversion device according to configuration 16, wherein a switch for performing a charging operation is provided between the avalanche photodiode and a power supply that applies a reverse bias to the avalanche photodiode.

[0175] (Composition 17) The photoelectric converter according to configuration 16, wherein the second exposure period is the period from the timing of the charge operation to the timing of the next charge operation.

[0176] (Composition 18) The measurement unit has a selection circuit, The plurality of second exposure periods each have one second exposure period and the other second exposure period. The photoelectric converter according to any one of configurations 1 to 13, wherein the selection circuit selects whether or not to output a signal corresponding to the pulse signal that defines the time information within the other second exposure period, depending on whether or not a photon is detected during the one second exposure period.

[0177] (Composition 19) The photoelectric conversion device according to configuration 18, characterized in that the selection circuit outputs a predetermined value regardless of the time from the start of the other second exposure period until the first detection of a photon if no photons are detected during the one second exposure period.

[0178] (Composition 20) The first substrate and the second substrate are stacked together. The first substrate has the photoelectric conversion element, The photoelectric conversion apparatus according to any one of configurations 1 to 19, wherein the second substrate comprises the exposure control unit, the timing generation unit, the measurement unit, and the pulse control unit.

[0179] (Composition 21) The first substrate, the second substrate, and the third substrate are stacked together. The first substrate has the photoelectric conversion element, The second substrate comprises the waveform shaping circuit, the timing determination circuit, the counter circuit, and the pulse control unit. The photoelectric conversion device according to configuration 7, wherein the third substrate comprises a second timing determination circuit, a second counter circuit, and a second pulse control unit.

[0180] (Composition 22) The pulse signal generated by the timing generation unit is input to the timing determination circuit provided on the second substrate, The photoelectric converter according to configuration 21, wherein the pulse signal generated by the timing generation unit input to the second timing determination circuit provided on the third substrate is different from the pulse signal generated by the timing generation unit.

[0181] (Composition 23) A photoelectric conversion device as described in any of configurations 1 to 22, A photoelectric conversion system comprising a signal processing unit that generates an image using the signal output by the aforementioned photoelectric conversion device.

[0182] (Composition 24) A mobile body including a photoelectric conversion device as described in any of configurations 1 to 22, A mobile body having a control unit that controls the movement of the mobile body using a signal output by the photoelectric converter. [Explanation of Symbols]

[0183] 1. Photoelectric converter 2. Exposure Control Unit 3 Timing generation unit 4. Measurement Unit 5. Pulse control unit 201 Signal Processing Circuit 202 Timing Determination Circuit 203 Counter Circuit 204 Quench element 205 Waveform shaping circuit 206 Latch Circuit 207 Signal Selection Circuit 208 Signal holding circuit 209 Pixel-based pulse generation circuit 210 Logic Circuits 211 Selection Circuit

Claims

1. A photoelectric conversion element that receives photons, An exposure control unit that generates signals defining a plurality of second exposure periods that are included in a first exposure period corresponding to one frame and are shorter than the first exposure period, A timing generation unit that generates a pulse signal defining time information within the second exposure period, A measurement unit measures the number of pulse signals since the first detection of a photon during the second exposure period, based on the pulse signal generated by the timing generation unit. A pulse control unit controls the number of pulse signals in the second exposure period following the second exposure period based on the value of the measurement unit, A photoelectric converter having

2. The photoelectric conversion apparatus according to claim 1, characterized in that the pulse control unit selects a pulse signal from among a plurality of pulse signals and controls the number of pulse signals during the next second exposure period.

3. The photoelectric conversion device according to claim 1, characterized in that when the value of the measuring unit is smaller than a predetermined value, the number of pulse signals in the next second exposure period is made greater than the number of pulse signals in the second exposure period.

4. The photoelectric conversion device according to claim 1, characterized in that when the value of the measurement unit is greater than a predetermined value, the number of pulse signals in the next second exposure period is reduced to the number of pulse signals in the second exposure period.

5. The timing generation unit generates pulse signals at equal intervals, The photoelectric conversion device according to claim 1, characterized in that the pulse signal is configured to be input to the measurement unit at least during the second exposure period.

6. The timing generation unit generates non-equally spaced pulse signals, The photoelectric conversion device according to claim 1, characterized in that the pulse signal is configured to be input to the measurement unit at least during the second exposure period.

7. The photoelectric conversion device according to claim 6, wherein the non-equally spaced pulse signals are configured such that their period increases in proportion to the elapsed time of the second exposure period.

8. The photoelectric converter according to claim 7, characterized in that the intervals between the non-equally spaced pulse signals are set to be logarithmically compressed in real time.

9. It has a correction circuit that corrects the signal output from the measurement unit, The photoelectric converter according to claim 1, characterized in that the correction circuit changes the correction means based on the number of pulse signals.

10. It has a correction circuit that corrects the signal output from the measurement unit, The photoelectric converter according to claim 4, characterized in that the correction circuit changes the value output from the measurement unit to a larger value when the number of pulse signals in the next second exposure period is less than the number of pulse signals in the second exposure period.

11. The photoelectric conversion device according to claim 1, characterized in that the measurement unit includes a waveform shaping circuit that converts the signal from the photoelectric conversion element into a pulse signal.

12. The measurement unit includes a timing determination circuit, The photoelectric conversion apparatus according to claim 11, characterized in that the timing determination circuit outputs a pulse signal generated by the timing generation unit after the timing at which a photon is first detected within the second exposure period.

13. The measurement unit includes a counter circuit, The photoelectric converter according to claim 12, characterized in that the counter circuit counts each pulse signal output from the timing determination circuit during the second exposure period and uses that count as the count value.

14. The photoelectric conversion device according to claim 13, characterized in that the counter circuit adds up and outputs the count values ​​for the plurality of second exposure periods.

15. The photoelectric conversion device according to claim 1, characterized in that the photoelectric conversion element is an avalanche photodiode.

16. The photoelectric conversion device according to claim 15, characterized in that a switch for performing a charging operation is provided between the avalanche photodiode and a power supply that applies a reverse bias to the avalanche photodiode.

17. The photoelectric converter according to claim 16, characterized in that the second exposure period is the period from the timing of the charging operation to the timing of the next charging operation.

18. The measurement unit has a selection circuit, The plurality of second exposure periods each have one second exposure period and the other second exposure period. The photoelectric converter according to claim 1, characterized in that the selection circuit selects whether or not to output a signal corresponding to the pulse signal that defines the time information within the other second exposure period, depending on whether or not a photon is detected during one of the second exposure periods.

19. The photoelectric converter according to claim 18, characterized in that the selection circuit outputs a predetermined value regardless of the time from the start of the other second exposure period until the first detection of a photon if no photon is detected during the one second exposure period.

20. The first substrate and the second substrate are stacked together. The first substrate has the photoelectric conversion element, The photoelectric conversion apparatus according to claim 1, characterized in that the second substrate has the exposure control unit, the timing generation unit, the measurement unit, and the pulse control unit.

21. The first substrate, the second substrate, and the third substrate are stacked together. The first substrate has the photoelectric conversion element, The second substrate comprises the waveform shaping circuit, the timing determination circuit, the counter circuit, and the pulse control unit. The photoelectric conversion device according to claim 13, wherein the third substrate comprises a second timing determination circuit, a second counter circuit, and a second pulse control unit.

22. The pulse signal generated by the timing generation unit is input to the timing determination circuit provided on the second substrate, The photoelectric converter according to claim 21, wherein the pulse signal generated by the timing generation unit, which is input to the second timing determination circuit provided on the third substrate, is different from the pulse signal generated by the timing generation unit.

23. A photoelectric conversion device according to any one of claims 1 to 22, A photoelectric conversion system comprising a signal processing unit that generates an image using the signal output by the aforementioned photoelectric conversion device.

24. A mobile body comprising a photoelectric converter according to any one of claims 1 to 22, A mobile body having a control unit that controls the movement of the mobile body using a signal output by the photoelectric converter.

Images