Photoelectric conversion equipment, photoelectric conversion system and mobile body
By introducing a timing controller and integrator into the photoelectric conversion device and setting weight values according to the photon detection timing, the problems of insufficient signal-to-noise ratio and dynamic range in the prior art are solved, and the signal-to-noise ratio is improved and the dynamic range is expanded under different light intensity conditions.
Patent Information
- Application Number
- CN202610644809.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-05-15
- Filing Date
- 2026-05-12
- Publication Date
- 2026-11-17
AI Technical Summary
Existing photoelectric conversion devices struggle to effectively improve the signal-to-noise ratio (SNR) and extend the dynamic range when detecting weak light at the single-photon level, especially under high light conditions where gain settings can lead to a decrease in SNR.
By introducing a timing controller into the photoelectric conversion device, weight values are set according to the timing of photon detection, and high gain is set under low light conditions and low gain is set under high light conditions. Combined with an integrator, weighted counting is performed in multiple sub-frame periods to achieve signal weighted processing.
By dynamically adjusting the gain and weight values under different light intensity conditions, the signal-to-noise ratio (SNR) was improved and the dynamic range was expanded, avoiding the decrease in SNR caused by gain settings under high light intensity conditions.
Smart Images

Figure CN123123750A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to photoelectric conversion devices, photoelectric conversion systems, and mobile bodies. Background Technology
[0002] A photoelectric conversion device capable of detecting weak light at the single-photon level using avalanche multiplication is known. US-2017-0186798 discloses a photoelectric conversion device in which a sensor chip with multiple pixels and a circuit chip having circuitry for performing signal processing are stacked. Japanese Patent Publications 2025-009968 and 2025-009969 describe a weighted counting method for weighting and counting signals detected by an avalanche photodiode. Summary of the Invention
[0003] This disclosure provides a technique for performing weighting based on gain.
[0004] This disclosure provides a photoelectric conversion device including a pixel array comprising multiple pixels, a readout circuit configured to read signals from the pixel array, and a timing controller. The timing controller can define multiple sub-frame periods forming each frame period and multiple detection periods forming each sub-frame period. Each of the multiple pixels may include a detector configured to detect photons and an integrator, the integrator being configured to add a weight value corresponding to a detection period in the multiple detection periods to the integrated value in each sub-frame period. The timing controller can set the weight value based on the gain of the signal multiplied by a processor configured to process the signal output from the readout circuit.
[0005] The features of this disclosure will become clear from the following description of embodiments with reference to the accompanying drawings. The following description of embodiments is by way of example. Attached Figure Description
[0006] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with this description, serve to explain the principles of the embodiments.
[0007] Figure 1 This is a view schematically illustrating an example configuration of a photoelectric conversion device according to an embodiment; Figure 2 This is a view schematically illustrating the configuration of the first substrate of the photoelectric conversion device according to an embodiment; Figure 3 This is a view schematically illustrating the configuration of the second substrate of the photoelectric conversion device according to an embodiment; Figure 4 This is a view illustrating an example of the pixel configuration according to an embodiment; Figures 5A to 5C This is a view used to explain the photon detection operation; Figures 6A to 6C This is a view used to interpret weighted counts; Figure 7 This is a view illustrating the relationship between gain and saturation count value according to the first embodiment; Figure 8 This is a view showing an example of the pixel configuration according to the first embodiment; Figure 9 This is a timing diagram illustrating an example of the operation of the pixel and timing controller according to the first embodiment; Figure 10 This is an example of a graph showing the probability of photon detection relative to the amount of light. Figure 11A and Figure 11B This is a view used to explain the second embodiment; Figure 12 This is a view used to explain the second embodiment; Figure 13 This is a view showing a photoelectric conversion system according to a first application example; Figure 14 This is a view showing a photoelectric conversion system according to a second application example; Figure 15A and Figure 15B This is a view showing a photoelectric conversion system according to a third application example; Figure 16 This is a view showing a photoelectric conversion system according to the fourth application example; Figure 17 This is a view showing a photoelectric conversion system according to the fifth application example; and Figure 18A and Figure 18B This is a view showing a photoelectric conversion system according to the sixth application example. Detailed Implementation
[0008] In the following, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are necessary, and multiple such features may be combined where appropriate. Furthermore, in the drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions thereof are omitted.
[0009] Embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that in the following description, terms indicating specific directions or positions (e.g., "up," "down," "right," "left," and other terms including these terms) are used where necessary. These terms are used to facilitate understanding of the embodiments with reference to the accompanying drawings, and the meaning of the terms does not limit the technical scope of the present disclosure.
[0010] In this specification, plan views are taken from a direction perpendicular to the light-incident surface of the semiconductor layer. Cross-sectional views indicate a plane in a direction perpendicular to the light-incident surface of the semiconductor layer. Note that if the light-incident surface of the semiconductor layer is microscopically rough, the plan view is defined with reference to the light-incident surface of the semiconductor layer when viewed macroscopically.
[0011] In the example described below, the detector for detecting photons is formed by an avalanche photodiode (APD). A fixed potential is applied to the anode of the APD, and the signal is extracted from the cathode side. Therefore, the semiconductor region containing charges of the same polarity as the signal charge as the majority carriers of the first conductivity type is an n-type semiconductor region, and the semiconductor region containing charges of the opposite polarity as the majority carriers of the signal charge as the second conductivity type is a p-type semiconductor region. Note that a fixed potential can be applied to the cathode of the APD, and the signal can be extracted from the anode side. In this case, the semiconductor region containing charges of the same polarity as the signal charge as the majority carriers of the first conductivity type is a p-type semiconductor region, and the semiconductor region containing charges of the opposite polarity as the majority carriers of the second conductivity type is an n-type semiconductor region. In the following description, the case where one node of the APD is set to a fixed potential will be used; however, the potentials of both nodes can vary.
[0012] Figure 1 An example configuration of a photoelectric conversion device 100 according to an embodiment of the present disclosure is illustrated schematically. The photoelectric conversion device 100 may have a structure in which, for example, a first substrate 11 and a second substrate 21 are stacked. The first substrate 11 may include, for example, a photoelectric converter array 12 comprising a plurality of photoelectric converters (APDs). The second substrate 21 may include processing circuitry 22 for processing signals output from the plurality of photoelectric converters.
[0013] Figure 2An example configuration of the first substrate 11 is schematically shown. The photoelectric converter array 12 of the first substrate 11 may include a plurality of photoelectric converters 102 arranged to form multiple rows and columns. Each photoelectric converter 102 is a constituent element of a detector for detecting photons and also a constituent element of a pixel 101. In the following explanation, the photoelectric conversion device 100 has aspects as part of an image capture device. However, the photoelectric conversion device 100 may be formed as part of another device. For example, the photoelectric conversion device 100 may be formed as a distance measuring device (e.g., a focus detection device or a distance measuring device using time-of-flight (TOF), a photometer (e.g., a device for measuring the amount of incident light), etc. Note that the plurality of pixels 101 may be arranged linearly, and in this case, the photoelectric conversion device 100 may form a line sensor.
[0014] Figure 3 The configuration of the processing circuitry 22 on the second substrate 21 is schematically shown. The processing circuitry 22 may include, for example, a pixel circuit array 120 containing multiple pixel circuits 103, a readout circuit 112, a controller 115, a horizontal scanning circuit 111, multiple signal lines 113, a vertical scanning circuit 110, a processor 122, an output unit 114, a gain setter 118, etc. Each pixel circuit 103 and its corresponding photoelectric converter 102 together form a pixel 101 and process the signal output from the photoelectric converter 102. A pixel circuit 103 may be provided corresponding to one photoelectric converter 102, or a pixel circuit 103 may be provided corresponding to two or more photoelectric converters 102. Even in the latter case, each pixel 101 can be considered to include a photoelectric converter 102 and a pixel circuit 103. In the example, a photoelectric converter 102 and a pixel circuit 103 may be electrically connected via a connector. The photoelectric converter array 120 and the pixel circuit array 120 form a pixel array.
[0015] Gain setter 118 can set the gain (digital gain) according to, for example, user operation or instructions from a host controller (not shown). Setting the gain may include receiving information indicating the gain from a host controller or similar device and storing it in memory (e.g., a latch circuit, flip-flop, or SRAM). Processing circuitry 22 may also include sensitivity setter 119 for setting ISO sensitivity. Sensitivity setter 119 sets the ISO sensitivity according to, for example, user operation or instructions from a host controller (not shown). Setting the ISO sensitivity may include receiving information indicating the ISO sensitivity from a host controller or similar device and storing it in memory (e.g., a latch circuit, flip-flop, or SRAM).
[0016] The vertical scanning circuit 110 can generate row control signals 116 according to, for example, control signals supplied from the controller 115, to sequentially select multiple rows of the photoelectric converter array 12. The vertical scanning circuit 110 and the controller 115 can form a timing controller TC, which generates timing for controlling the operation of the multiple pixels 101. The vertical scanning circuit 110 may include at least one of, for example, a shift register and an address decoder. The readout circuit 112 reads signals from the pixel array according to, for example, control signals supplied from the controller 115. More specifically, the readout circuit 112 reads signals output from pixel circuits 103 corresponding to pixels 101 in the rows selected by the vertical scanning circuit 110 via multiple signal lines 113.
[0017] The horizontal scanning circuit 111 can control the readout circuit 112, such that signals, for example, a row read out by the readout circuit 112, are output to the processor 122 in a predetermined order. The processor 122 processes the signals output from the readout circuit 112 and supplies the processed signals to the output unit 114. The processor 122 can perform processing, for example, multiplying the signals output from the readout circuit 112 by a gain set by the gain set by the gain setter 118. The processor 122 can be provided externally to the photoelectric conversion device 100, and the functions of the processor 122 can be implemented by a first processor provided inside the photoelectric conversion device 100 and a second processor provided externally to the photoelectric conversion device 100. Moreover, a device incorporating the photoelectric conversion device 100 can be referred to as a photoelectric conversion device.
[0018] Figure 4 An example configuration of a pixel 101 formed by a photoelectric converter 102 and a pixel circuit 103 is shown. Figure 4 In the example shown, a pixel circuit 103 is assigned to a photoelectric converter 102. The photoelectric converter 102 includes an APD 201. At least one of the plurality of constituent elements of the pixel circuit 103 may be arranged on the first substrate 11. Alternatively, all of the plurality of constituent elements of the pixel circuit 103 may be arranged on the first substrate 11.
[0019] A first voltage VH can be applied to the cathode of APD 201. A second voltage VL is applied to the anode of APD 201, and the potential of the first voltage VH is higher than the potential of the second voltage VL. The potential difference between the first voltage VH and the second voltage VL is applied between the anode and cathode of APD 201. This potential difference is the reverse bias voltage that causes APD 201 to perform avalanche multiplication operation. The charge generated by photons entering APD 201 causes avalanche multiplication, thereby generating an avalanche current. The mode in which a voltage higher than the breakdown voltage of APD 201 is applied between the anode and cathode is called Geiger mode. The mode in which a voltage near or below the breakdown voltage is applied between the anode and cathode of APD 201 is called linear mode. An APD operating in Geiger mode is called a SPAD. In the example, the first voltage VH is 1V and the second voltage VL is -30V.
[0020] The pixel circuit 103 may include a quenching element 202 connected between the cathode of the APD 201 and a terminal supplied with a first voltage VH. The quenching element 202 can be understood as an element that supplies the first voltage VH to the cathode of the APD 201. The quenching element 202 has the function of converting changes in the avalanche current generated in the APD 201 into a voltage signal. The quenching element 202 acts as a load circuit (quenching circuit) during signal multiplication via avalanche multiplication and is used to suppress avalanche multiplication by suppressing the voltage applied to the APD 201. This is referred to as quenching operation.
[0021] The pixel circuit 103 may additionally include at least one of, for example, a waveform shaping circuit 210, a signal processing circuit 211, and a selection circuit 212. The waveform shaping circuit 210 can output a pulse signal by shaping the potential change of the cathode of the APD 201 during photon detection. The waveform shaping circuit 210 may include, for example, an inverter circuit. Figure 4 In the example shown, the waveform shaping circuit 210 is formed by an inverter, but the waveform shaping circuit 210 can be formed by connecting multiple inverters in series or by another circuit that has a waveform shaping effect.
[0022] Signal processing circuit 211 can be configured to generate and maintain a count signal indicating the number of incident photons based on a pulse signal output from waveform shaping circuit 210. The count signal maintained in signal processing circuit 211 can be reset when a first control pulse of an effective level is supplied from vertical scanning circuit 110 via drive line 213. When a second control pulse of an effective level is supplied from vertical scanning circuit 110 via drive line 214, selection circuit 212 can electrically connect signal processing circuit 211 and signal line 113. Selection circuit 212 may include, for example, a buffer circuit.
[0023] Figures 5A to 5C This is a view used to explain the photon detection operation performed by the photoelectric conversion device 100. Figure 5A It shows from Figure 4 An excerpted view of the APD 201, quenching element 202, and waveform shaping circuit 210. The input side of the waveform shaping circuit 210 is node A, and the output side is node B. Figure 5B It shows Figure 5A The waveform at node A in the middle, and Figure 5C It shows Figure 5A The waveform at node B. From time t0 to time t1, the potential difference between VH and VL is applied to APD 201. When a photon enters at time t1, avalanche multiplication occurs in APD 201. This causes an avalanche multiplication current to flow into quenching element 202, and the voltage (potential) at node A drops. When the voltage drop increases and the potential difference applied to APD 201 decreases, avalanche multiplication in APD 201 stops. Thereafter, current compensating for the voltage drop from the second voltage VL flows into node A, and node A stabilizes at the original potential level at time t3. At this time, the portion of the output waveform at node A that exceeds a given threshold is shaped by waveform shaping circuit 210, and a pulse signal appears at node B.
[0024] Based on the principle of weighted counting in this disclosure, reference will be made to Figures 6A to 6C To describe. Here, Figure 6A This is a view showing the change in the detection probability of the first photon with respect to time under each light intensity condition and an example of the weighting coefficients in each time period. Figure 6B This is a graph showing the probability of photon detection relative to the amount of light. Figure 6C This is a graph showing the number of photon detection signal counts relative to the amount of light. Here, the photon first detected in each subframe period in which the counting operation is performed is referred to as the first photon. Note that in this specification, the amount of light means the amount of light entering the photoelectric conversion device 100 (pixel 101) per unit time.
[0025] like Figure 6A As shown, under high light intensity conditions, the probability of detecting the first photon is higher in the early part of the subframe period and decreases over time. If the subframe period is divided into a first period and a second period following the first period, then under high light intensity conditions, the probability of detecting the first photon is high during the first period. Under medium light intensity conditions, the photon detection probability also decreases over time. However, compared to the high light intensity case, the time dependence of the first photon detection probability is low, and the first photon can be detected even during the second period. Under low light intensity conditions, the detection probability of the first photon is almost independent of time.
[0026] exist Figure 6B In the diagram, dashed line A indicates the probability of detecting the first photon during the first time period at each light intensity, and dashed line B indicates the probability of detecting the first photon during the second time period at each light intensity. The solid line (A+B), which is the sum of dashed lines A and B, indicates the probability of detecting the first photon during the detection time period at each light intensity. As indicated by dashed line A, the higher the light intensity, the higher the probability of detecting the first photon during the first time period, and this probability saturates under high light intensity conditions. In particular, under low to medium light intensity conditions, the probability of detecting the first photon during the first time period increases exponentially with increasing light intensity. On the other hand, as indicated by dashed line B, under low to medium light intensity conditions, the higher the light intensity, the higher the probability of detecting the first photon during the second time period, and this probability decreases under medium to high light intensity conditions. This is because under high light intensity conditions, the first photon is detected in many cases during the first time period, while the photon detected in the second time period is the one that entered after the first photon. As for the probability of detecting the first photon under each light intensity, as indicated by the solid line (A+B) which is the sum of the dashed lines A and B, under low to medium light intensity conditions, the higher the light intensity, the higher the probability of photon detection, and this probability converges to a predetermined value under medium to high light intensity conditions.
[0027] Figure 6C A graph showing the count values at each light intensity is presented. Figure 6B The diagram shows the similar detection probability of the first photon, and the second and subsequent photons entering the APD. That is, under low to medium light intensity conditions, the higher the light intensity, the greater the number of photons counted. Under medium to high light intensity conditions, the count value saturates with increasing light intensity. The saturation value of the count value is called the saturation count value. Under high light intensity conditions, the increase in the count value does not correspond one-to-one with the increase in the number of incident photons, and it can be said that the weight of each count increases. In other words, it can be said that under low to medium light intensity conditions, the expected number of detected photons depends on the light intensity, but under medium to high light intensity conditions, the weight of the count value depends on the light intensity.
[0028] In the embodiment described below, a counting operation is performed while the weighting coefficients (weight values) are changed according to the timing of detected photons. In this embodiment, each frame period is divided into multiple sub-frame periods, and each sub-frame period is further divided into multiple detection periods. Each sub-frame period is also referred to as a recharge period. A frame is a still image, and a frame period is the period for acquiring one still image. For example, when 30 frames of still images are continuously reproduced, a one-minute moving image is generated. By setting the weighting coefficients (weight values) according to the timing of detected photons, both the signal-to-noise ratio (SNR) and the dynamic range can be improved simultaneously.
[0029] Figure 7 An example illustrates the relationship between the gain and saturation count value when a processor 122, configured to process the signal output from readout circuit 112, multiplies the signal. When capturing an image of a subject with low illumination, processor 122 multiplies the signal output from the pixel array by a gain (digital gain) greater than 1, thereby obtaining an image with proper exposure. Generally, processor 122 can be designed based on a saturation count value at the optimal SNR state as a reference. When capturing an image of a subject with illumination lower than the illumination corresponding to the reference saturation count value, processor 122 can multiply the signal output from readout circuit 112 by a gain greater than 1. For this reason, as... Figure 7 As illustrated, the saturation count value and the gain can be inversely proportional.
[0030] based on Figures 6A to 6C The example shown illustrates this: under high light intensity conditions, the gain of processor 122 is set to low, and under low light intensity conditions, the gain of processor 122 is set to high. Therefore, it can be said that under high light intensity conditions, i.e., when low gain is set, the weighting coefficients (weight values) depend on the light intensity of the subject, and under low light intensity conditions, i.e., when high gain is set, the unweighted photon count depends on the light intensity of the subject. Therefore, if low light intensity conditions are set, i.e., high gain, it can be said that the SNR decreases when weighting is performed.
[0031] The timing controller TC can set or determine the weighting coefficient (weight value) based on the gain set by the gain setter 118. In other words, the timing controller TC can set or determine the weighting coefficient (weight value) based on the gain by which the processor 122, configured to process the signal output from the readout circuit 112, multiplies the signal. Here, setting the weighting coefficient (weight value) to 1 means that no weighting is performed. Setting the weighting coefficient (weight value) according to the gain can be performed to suppress the reduction in SNR caused by weighting.
[0032] In this embodiment, such as Figure 7 As illustrated, if the gain set by the gain setter 118 is greater than a predetermined gain (threshold), then the timing controller TC does not perform weighting (i.e., sets the weighting coefficient (weight value) to 1). This suppresses the reduction in SNR caused by weighting. On the other hand, if the gain set by the gain setter 118 is less than the predetermined gain (threshold), then the timing controller TC performs weighting (i.e., sets the weighting coefficient (weight value) to a value greater than 1). The predetermined gain can be determined according to the required specifications. The photoelectric conversion device 100 may include a programming unit 121 that programs (sets) the predetermined gain (threshold) according to instructions from a user or the like.
[0033] Note that the predetermined gain (threshold) can be a value that the gain setter 118 never sets (e.g., if the gain setter 118 can set gains of 2, 4, 8, 16, 32, ..., then the predetermined gain is set to 20). Alternatively, the predetermined gain (threshold) can be a value that can be set by the gain setter 118. In the latter case, the timing controller TC can be configured to perform weighting if the gain set by the gain setter 118 equals the predetermined gain (threshold), or it can be configured not to perform weighting.
[0034] Figure 8 An example of the configuration of one pixel 101 according to the first embodiment is shown. Figure 8 In the example shown, the quenching element 202 is formed by a PMOS transistor, and the waveform shaping circuit 210 is formed by an inverter. The APD 201, quenching element 202, and waveform shaping circuit 210 can form a detector 710 for detecting photons. The signal processing circuit 211 can form an integrator 720 that adds a weighted value to the integral value in each of the plurality of subframe periods forming each frame period, based on the detection period in which photons were detected by the detector 710. The integrator 720 can include a counter 702 and a counter controller 701 that controls the counter 702. The counter controller 701 can be formed, for example, by an AND circuit. The output signal from the waveform shaping circuit 210 is input to one of the two input terminals of the AND circuit forming the counter controller 701, and a clock signal TCLK can be input to the other input terminal.
[0035] In addition to the clock signal TCLK, the enable signal EN, the reset signal RES, and the select signal SEL can be supplied to the pixel 101 from the vertical scan circuit 110. The clock signal TCLK, the enable signal EN, and the reset signal RES can be supplied from the controller 115. The vertical scan circuit 110 and the controller 115 can form a timing controller TC, which generates timing for controlling the operation of the multiple pixels 101. The timing controller TC can drive the multiple pixels 101 using either a global shutter method or a rolling shutter method. In the global shutter method, the exposure period for detecting photons is common to the multiple pixels 101, while in the rolling shutter method, the exposure period for measuring the number of photons entering each pixel 101 varies between rows.
[0036] The timing controller TC can cause each of the clock signal TCLK, enable signal EN, reset signal RES, and selection signal SEL to transition at a timing point according to the gain set by the gain setter 118. Furthermore, the timing controller TC can set or determine the weight value corresponding to the detection period in which photons are detected among multiple detection periods, based on the gain set by the gain set by the gain setter 118.
[0037] Figure 9 An example of the operation of pixel 101 and timing controller TC according to the first embodiment is shown. Figure 9 In the example shown, in the k-th frame, the gain is set to less than by the gain setter 118. Figure 7 The gain shown is the predetermined gain (threshold), and in the (k+1)th frame, it is set by the gain setter 118 to be greater than... Figure 7 The gain is the predetermined gain (threshold) shown. Therefore, the maximum value of the weighting coefficient (weight value) set by the timing controller TC for the k-th frame is different from the maximum value of the weighting coefficient (weight value) set by the timing controller TC for the (k+1)-th frame. More specifically, the maximum value of the weighting coefficient (weight value) set by the timing controller TC for the k-th frame is 3, and the maximum value of the weighting coefficient (weight value) set by the timing controller TC for the (k+1)-th frame is 1. Here, the maximum value of the weighting coefficient (weight value) is 1, which means that no weighting is performed.
[0038] In the following text, a low logic level for a signal is represented as L, and a high logic level for a signal is represented as H. As mentioned above, each frame period comprises multiple subframe periods (recharge periods). For example, in Figure 9 In this context, the time interval from time t0 to time t5 is a subframe interval (recharge interval). The timing controller TC defines multiple detection intervals that form each subframe interval using the clock signal TCLK (in... Figure 9 In the example, there are three detection periods. The clock signal TCLK has aspects that include multiple pulse signals defining multiple detection periods, and the intervals between the multiple pulse signals can be unequal in each subframe and can, for example, monotonically decrease. The first detection period is the period in which the weighting coefficient is expressed as "+3", the second detection period is the period in which the weighting coefficient is expressed as "+2", and the third detection period is the period in which the weighting coefficient is expressed as "+1". The clock signal TCLK is also the signal that causes the counter 702 to perform a counting operation during the period in which the voltage Vc of the output node of APD 201 is at L (the period in which the output of waveform shaping circuit 210 is at H).
[0039] If a photon is detected by detector 710 during the first detection period ("+3"), then Vc becomes L (the output of waveform shaping circuit 210 becomes H), and thereafter, the clock signal TCLK becomes H three times. In response, counter 702 performs three counting operations, and thus the value "3" corresponding to the first detection period is added to the integral value of the count value Count of integrator 720 (which becomes 0 immediately after being reset by reset signal RES). If a photon is detected by detector 710 during the second detection period ("+2"), then Vc becomes L (the output of waveform shaping circuit 210 becomes H), and thereafter, the clock signal TCLK becomes H twice. In response, counter 702 performs two counting operations, and thus the value "2" corresponding to the second detection period is added to the integral value of the count value Count of integrator 720. If a photon is detected by detector 710 during the third detection period ("+1"), then Vc becomes L (the output of waveform shaping circuit 210 becomes H), and thereafter, the clock signal TCLK becomes H only once. In response, counter 702 performs a counting operation only once, and therefore the value "1" corresponding to the third detection period is added to the integral value of integrator 720 as the count value Count. As described above, the method of adding the weight value corresponding to the detection period in which the photon was detected by detector 710 in the multiple detection periods to the integral value in each subframe period is called weighted addition.
[0040] The operation of the k-th frame starting from time t0 will be described illustratively below. As mentioned above, in the k-th frame, the gain setter 118 sets the gain to less than... Figure 7 The gain is the predetermined gain (threshold) shown. Accordingly, the timing controller TC sets the maximum value of the weighting coefficients (weight values) to 3. The timing of photon entry is exemplified as "incident photon," and... Figure 9 The operation illustrated depends on the timing of photon entry. At time t0, the logic level of the enable signal EN changes from L to H. Therefore, the exposure period in the k-th frame begins. Also, at time t0, the logic level of the clock signal VR changes to L and then immediately returns to H. Therefore, the quenching element 202 is turned on and Vc is reset (recharged). At time t1, the logic level of the clock signal TCLK changes to H. The clock signal TCLK is the signal that defines the timing of changing the weighting coefficients (in other words, the detection period), and it is also the signal that causes the counter 702 to perform a counting operation during the period in which Vc is at L (the period in which the output of the waveform shaping circuit 210 is at H).
[0041] At time t1, since Vc is at H (the output of waveform shaping circuit 210 is at L), counter 702 does not perform a counting operation even if the clock signal TCLK becomes H. At time t2, a photon enters APD 201. Therefore, the logic level of the output signal Vc becomes L, and the logic level of the output of waveform shaping circuit 210 becomes H. At time t3, if the level of clock signal TCLK becomes H, then the logic level of the output of the AND circuit used as counter controller 701 becomes H. Therefore, counter 702 performs a counting operation and adds 1 to the integral value, which is the count value Count. At time t4, the level of clock signal TCLK becomes H again, and the logic level of the output of the AND circuit used as counter controller 701 thus becomes H. Therefore, counter 702 performs a counting operation and adds 1 to the integral value, which is the count value Count. That is, during the period from time t1 to time t4, 2 is added as a weight value to the integral value, which is the count value Count. In other words, during a subframe period starting from time t0, 2 is added as a weight value to the integral value, which is the count value Count.
[0042] During the period from time t4 to time t5, the logic level of the enable signal EN changes to L. Counter 702 is therefore disabled. Photons entering APD 201 during this period are not detected. The period from time t1 to time t5 is a subframe period, and a new subframe period begins at time t5.
[0043] During the subframe period starting at time t5, the level of the clock signal TCLK changes to H at times t6 and t7, but since Vc is at H (the output from the waveform shaping circuit 210 is at L), the counter 702 does not perform a counting operation. A photon enters the APD 201 at time t8, and in response, the level of the output signal Vc changes to L, and the logic level of the output of the waveform shaping circuit 210 changes to H. At time t9, the level of the clock signal TCLK changes to H, and the counter 702 performs a counting operation, adding 1 to the integral value of the count value Count. That is, during the period from time t5 to time t9, 1 is added to the integral value of the count value Count.
[0044] During the period from time t14 to time t15, the selection signal SEL changes to H. Therefore, the selection circuit 212 outputs the integral value held in the signal processing circuit 211 to signal line 113. At time t16, a photon enters the APD 201, but because the logic level of the enable signal EN is L, the signal based on this photon is not counted. During the period from time t17 to time t18, the level of the reset signal RES changes to H, and the signal held in the signal processing circuit 211 is reset.
[0045] At time t19, the enable signal EN changes from L to H, and the exposure period (and the first subframe period) in frame (k+1) begins. As described above, in frame (k+1), the gain is set by gain setter 118 to be greater than... Figure 7 The gain is the predetermined gain (threshold) shown. Accordingly, the timing controller TC sets the maximum value of the weighting coefficient (weight value) to 1. This means that no weighting is performed. At times t20 and t22 in the (k+1)th frame corresponding to times t1 and t3 in the kth frame, the clock signal TCLK remains unchanged at H. Therefore, even at time t22 when the level of the output signal Vc is at L, the counter 702 does not perform a counting operation.
[0046] At time t23, if the level of the clock signal TCLK changes to H, then the logic level of the output from the AND circuit used as the counter controller 701 changes to H, and therefore, the counter 702 performs a counting operation and adds 1 to the integral value as the count value Count. During the period from time t23 to time t24, the logic level of the enable signal EN changes to L. The counter 702 is therefore controlled to be disabled. During this period, photons entering the APD 201 are not detected. The period from time t19 to time t24 is a subframe period, and during the subframe period starting from time t19, weighting and counting are performed.
[0047] The new subframe period begins at time t24. During this subframe period, photons enter APD 201 at time t27, and in response, the level of the output signal Vc changes to L, and the logic level of the output of waveform shaping circuit 210 changes to H. At time t28, the level of clock signal TCLK changes to H, counter 702 performs a counting operation, and adds 1 to the integral value of the count value Count. That is, during the period from time t24 to time t28, 1 is added to the integral value of the count value Count. Weighting and counting are performed during the subframe period starting at time t24.
[0048] During the period from time t33 to time t34, the selection signal SEL changes to H. Therefore, the selection circuit 212 outputs the integral value held in the signal processing circuit 211 to signal line 113. At time t35, a photon enters the APD 201, but since the logic level of the enable signal EN is L, the signal based on this photon is not counted. During the period from time t36 to time t37, the level of the reset signal RES changes to H, and the signal held in the signal processing circuit 211 is reset.
[0049] At time t38, the enable signal EN changes from L to H, and the exposure period in the (k+2)th frame begins.
[0050] In the above configuration, the clock signal TCLK generated by the timing controller TC changes according to the gain set by the gain setter 118. However, for example, an AND circuit can be added to pixel 101. Thus, the clock signal TCLK can be generated by the AND circuit calculating the mask signal generated by the timing controller TC according to the gain set by the gain setter 118 and the AND of the clock signal independent of the gain generation. Alternatively, the signal supplied to the counter 702 can be masked by the mask signal generated by the timing controller TC according to the gain set by the gain setter 118, or another configuration can be used.
[0051] Integrator 720 adds a weight value corresponding to the detection period in the multiple detection periods where the detector detected photons to the integrated value (count value) in each of the multiple subframe periods. In the example above, the summation of the weight values corresponding to the detection periods in the multiple detection periods is given by the number of clock signals TCLK, which are pulse signals. However, instead, for example, the weight values corresponding to the detection periods in the multiple detection periods where the detector detected photons can be provided to integrator 720 from the timing controller TC, and integrator 720 can add the weight values to the integrated value. The number of detection periods forming each subframe is not limited to 3, and can be any number of 2 or more. The weight values corresponding to the detection periods in the multiple detection periods where the detector detected photons can be any value.
[0052] The gain setter 118 can set the gain based on the ISO sensitivity set by the sensitivity setter 119. Here, the required saturation count value decreases as the ISO sensitivity increases. For this reason, the higher the ISO sensitivity, the greater the gain can be set.
[0053] In the example above, if the gain set by gain setter 118 is less than a predetermined gain (threshold), then the timing controller TC performs weighting; and if the gain set by gain setter 118 is greater than the predetermined gain, then no weighting is performed. However, depending on the value of the predetermined gain, if the gain set by gain setter 118 is less than the predetermined gain, then the timing controller TC may not perform weighting; and if the gain set by gain setter 118 is greater than the predetermined gain, then weighting may be performed. Moreover, when weighting is performed, the timing controller TC can change the maximum value of the weighting coefficient (weight value) according to the gain.
[0054] A second embodiment, a modification of the first embodiment, will be described below. Matters not mentioned in the description of the second embodiment may be consistent with the first embodiment. Figure 10 It involves adding an ultra-low light intensity condition, which has a lower light intensity than the low light intensity condition, to... Figure 6B The view shows that under light intensity conditions equal to or less than the ultra-low light intensity condition, the probability of detecting the first photon in the first time period, indicated by the dashed line A, is so low that it can be essentially ignored. For this reason, even when performing a counting operation while performing weighting, the probability of a photon entering a time period with a weight greater than 1 is very low, and there is almost no difference between the SNR with and without weighting.
[0055] On the other hand, even under low light conditions, there is a possibility that a flash of light may enter the photoelectric conversion device 100, and such events may occur. Figure 11A and Figure 11B The example shown is as follows. Figure 11A This illustrates the change in the count value when weighting is applied. The flash is instantaneous but has a large amount of light, and photons are highly likely to enter during the first detection period. For this reason, it is more likely that the count will be performed with the maximum weight value. Figure 11B This example illustrates the change in the count value without weighting. Figure 11B In the example shown, although the photon enters during the first detection period, like Figure 11A As shown in the example, but because no weighting is performed, these are simply counted as one photon. Therefore, it can be said that if a flash enters under low light intensity conditions, then by performing the count while performing weighting, a substantially accurate light intensity count value can be obtained.
[0056] In the second embodiment, an example will be described in which the gain set by the gain set by the gain setter 118 improves the detection accuracy of flashes that may occur under ultra-low light intensity while suppressing the reduction of SNR.
[0057] Figure 12 An example illustrates the relationship between the gain and saturation count value when a processor 122, configured to process a signal output from readout circuit 112, multiplies the signal, and gain thresholds 1 and 2. In a second embodiment, gain threshold 1 (first threshold) and gain threshold 2 (second threshold) are provided. Here, gain threshold 2 is greater than gain threshold 1. The photoelectric conversion device 100 may include a programming unit 121 that programs (sets) gain threshold 1 (first threshold) and gain threshold 2 (second threshold) according to instructions from a user or the like.
[0058] If the gain set by gain setter 118 is less than a first threshold and greater than a second threshold, then timing controller TC performs weighting (i.e., sets the weighting coefficient (weight value) to a value greater than 1). Conversely, if the gain set by gain setter 118 is greater than gain threshold 1 and less than gain threshold 2, then timing controller TC does not perform weighting (i.e., sets the weighting coefficient (weight value) to 1). Note that gain thresholds 1 and 2 can be values that gain setter 118 never sets. Alternatively, gain thresholds 1 and 2 can be values that can be set by gain setter 118. In the latter case, timing controller TC can be configured to perform weighting if the gain set by gain setter 118 equals gain threshold 1 or 2, or it can be configured not to perform weighting. Moreover, when weighting is performed, between the case where the gain set by gain setter 118 is less than the first threshold and the case where the gain is greater than the second threshold, timing controller TC can change the maximum value of the weighting coefficient (weight value).
[0059] According to the second embodiment, the pixel 101 of the photoelectric conversion device 100 can have the same... Figure 8 The same configuration is shown for pixel 101. The photoelectric conversion device 100 according to the second embodiment can be configured as follows... Figure 9 Operate as illustrated in the example. Here, in Figure 9 In the context of the Kth frame, the gain is indicated as being less than the gain threshold 1 or greater than the gain threshold 2, and the (k+1)th frame indicates the gain as being greater than the gain threshold 1 and less than the gain threshold 2.
[0060] The following will describe several application examples of the photoelectric conversion device 100.
[0061] Figure 13 This is a block diagram illustrating a schematic configuration of a photoelectric conversion system according to a first application example. A photoelectric conversion device 100 is incorporated into the photoelectric conversion system according to the first application example. The photoelectric conversion system according to the first application example may include, for example, a controller 401, a timing generator 402, an image acquirer 403, a readout unit 404, a gain adjuster 405, a nonlinear corrector 406, a defect corrector 407, a data compressor 408, and a storage unit 409. The gain adjuster 405, the nonlinear corrector 406, the defect corrector 407, the data compressor 408, and the storage unit 409 may form a processor 122.
[0062] Image acquisition unit 403 is, for example, a pixel array in which a plurality of pixels 101 are arranged, and readout unit 404 is, for example, readout circuit 112. Timing generator 402 is, for example, a timing controller TC. Controller 401 may be arranged inside photoelectric conversion device 100 or may be arranged externally. Image acquisition unit 403 is controlled by timing generator 402 controlled by controller 401. Image data generated by image acquisition unit 403 undergoes correction processing and is then input to storage unit 409. Note that the order of correction processing is not limited to... Figure 13 The order shown.
[0063] A gain adjuster 405 is provided, for example, between a readout unit 404 and a nonlinear corrector 406, and the gain adjuster 405 multiplies the image data generated by the image acquirer 403 by a digital gain. The data used for image correction often has fractional values, but if the image output is an integer value, the correction accuracy will be reduced due to quantization errors. When the image data is multiplied by a gain beforehand, the effects of quantization errors can be suppressed, and the correction accuracy can be improved. Here, if the quantization error can be suppressed to 1 / 4 or less of the single-photon signal level, the corrected image will appear visually natural. For this reason, the digital gain multiplied by the image data is preferably, for example, 4 times or greater.
[0064] A nonlinear corrector 406 can be arranged, for example, between a gain adjuster 405 and a defect corrector 407. The nonlinear corrector 406 can correct image data under the control of, for example, a controller 401. As described in the above embodiments, if the image acquirer 403 is a photon-counting type detector, the optical response is often nonlinear due to the influence of the dead time. If a correction assuming a linear response is performed under the influence of a nonlinear optical response, overcorrection may occur. Therefore, when nonlinear correction is performed on the image data before the arithmetic processing in the defect corrector 407, overcorrection can be prevented, and appropriate nonlinear correction based on the driving timing can be performed. Nonlinear correction can be performed using, for example, a lookup table.
[0065] Defect corrector 407 corrects data including defective pixels in the image data. Defect corrector 407 may, for example, extract the output value of the defective pixel, specify the location information and output value of the defective pixel, and replace that value with the average or median of the outputs of pixels surrounding the specified defective pixel, or it may divide the estimated defective image data, or it may perform correction by another method. Data compressor 408 compresses the corrected image data. In the photoelectric conversion system according to the first application example, a huge amount of image data corresponding to a high dynamic range can be generated. Therefore, by providing data compressor 408, the data is preferably compressed before it is stored in storage unit 409 of a subsequent stage. Storage unit 409 may hold at least a portion of the image data generated in the previous stage. Storage unit 409 may include at least one of, for example, SRAM, DRAM, and non-volatile memory.
[0066] Reference Figure 14 The photoelectric conversion system according to the second application example is described. The photoelectric conversion device 100 can be applied to various photoelectric conversion systems. Examples of photoelectric conversion systems include digital still cameras, digital video cameras, surveillance cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, and observation satellites. Camera modules, including optical systems such as lenses and image capture devices, are also included in the photoelectric conversion system. As examples of these, Figure 14 A block diagram of a digital still camera is shown.
[0067] Figure 14 The photoelectric conversion system illustrated includes an image capturing device 1004, exemplified as a photoelectric conversion device 100, and a lens 1002 that forms an optical image of a subject on the image capturing device 1004. The photoelectric conversion system also includes an aperture 1003 capable of changing the amount of light passing through the lens 1002, and a barrier 1001 for protecting the lens 1002. The lens 1002 and aperture 1003 form an optical system that converges light onto the image capturing device 1004. The image capturing device 1004 converts the optical image formed by the lens 1002 into an electrical signal.
[0068] The photoelectric conversion system includes a signal processor 1007, which is an image generator configured to generate an image by processing an output signal from an image capture device 1004. The signal processor 1007 performs various corrections or compressions and outputs image data as needed. The signal processor 1007 can be formed in a semiconductor layer where the image capture device 1004 is located, or it can be formed in a different semiconductor layer than the semiconductor layer of the image capture device 1004. Furthermore, the image capture device 1004 and the signal processor 1007 can be formed in the same semiconductor layer.
[0069] The photoelectric conversion system also includes a memory unit 1010 configured to temporarily store image data, and an external interface unit (external I / F unit) 1013 configured to communicate with an external computer or the like. Furthermore, the photoelectric conversion system includes a recording medium 1012 configured to record or read image capture data, such as a semiconductor memory, and a recording medium control interface unit (recording medium control I / F unit) 1011 configured to perform recording or reading from the recording medium 1012. Note that the recording medium 1012 can be integrated into the photoelectric conversion system or can be detachable.
[0070] The photoelectric conversion system also includes a general control / arithmetic unit 1009 that controls various arithmetic processes and the entire digital still camera, and a timing generator 1008 that outputs various timing signals to the image capture device 1004 and the signal processor 1007. Here, the timing signals, etc., can be input from the outside, and it is sufficient for the photoelectric conversion system to include at least the image capture device 1004 and the signal processor 1007 that processes the output signals from the image capture device 1004.
[0071] Image capture device 1004 outputs an image capture signal to signal processor 1007. Signal processor 1007 performs predetermined signal processing on the image capture signal output from image capture device 1004 and outputs image data. Signal processor 1007 uses the image capture signal to generate an image.
[0072] Reference Figure 15A and Figure 15B Describes a photoelectric conversion system and a moving body according to a third application example. Figure 15AAn example of a photoelectric conversion system 2300 involving an in-vehicle camera is shown. The photoelectric conversion system 2300 includes an image capture device 2310. The image capture device 2310 is the photoelectric conversion device 100 described in any of the above embodiments. The photoelectric conversion system 2300 includes an image processor 2312 that performs image processing on multiple image data acquired by the image capture device 2310. Furthermore, the photoelectric conversion system 2300 includes a disparity acquirer 2314 that calculates disparity (phase difference of the disparity image) from the multiple image data acquired by the photoelectric conversion system 2300. The photoelectric conversion system 2300 also includes a distance measuring device 2316 that calculates the distance to a target object based on the calculated disparity, and a collision determiner 2318 that determines the possibility of a collision based on the calculated distance. Here, the disparity acquirer 2314 or the distance acquirer 2316 is an example of a distance information acquisition component for acquiring distance information to a target object. That is, distance information can be acquired using not only phase difference but also TOF technology. The collision determiner 2318 can use one of multiple distance information pieces to determine the probability of a collision. The distance information acquisition component can be implemented by specially designed hardware or by a software module. The distance information acquisition component can be implemented using a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a combination thereof.
[0073] The photoelectric conversion system 2300 is connected to the vehicle information acquisition device 2320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 2300 is also connected to a control ECU 2330, which is a control device (controller) configured to generate a control signal for braking force to the vehicle based on the determination result output of the collision determiner 2318. Furthermore, the photoelectric conversion system 2300 is connected to an alarm device 2340 that generates an alarm to the driver based on the determination result of the collision determiner 2318. For example, if the collision probability is high according to the determination result of the collision determiner 2318, the control ECU 2330 performs braking, releases the accelerator pedal, or suppresses engine output, thereby controlling the vehicle to avoid a collision and reduce damage. The alarm device 2340 issues an alarm, displays alarm information on a screen such as a car navigation system, or applies vibration to the seat belt or steering wheel, thereby alerting the user.
[0074] In this application example, the vehicle's perimeter (e.g., the front or rear) can be captured by the photoelectric conversion system 2300. Figure 15BAn example is shown of a photoelectric conversion system arranged to capture the front side of a vehicle (image capture range 2350). The vehicle information acquisition device 2320 sends instructions to the photoelectric conversion system 2300 or the image capture device 2310. This configuration can further improve the accuracy of distance measurement.
[0075] The example of implementing control to avoid collisions with other vehicles has already been explained above. Photoelectric conversion systems can also be applied to implement autonomous driving control to follow another vehicle or to implement autonomous driving control to avoid deviating from the lane. Furthermore, photoelectric conversion systems can be applied not only to vehicles such as this car, but also to mobile bodies (mobile devices) such as ships, airplanes, or industrial robots. Moreover, photoelectric conversion systems can be applied not only to mobile bodies, but also to equipment that widely uses object recognition, such as Intelligent Transportation Systems (ITS).
[0076] Reference Figure 16 The photoelectric conversion system according to the fourth application example is described, which is configured as a distance image sensor. For example... Figure 16 As illustrated, the distance image sensor 1401 can be configured to include an optical system 1402, a photoelectric conversion device 1403, an image processing circuit 1404, a monitor 1405, and a memory 1406. The distance image sensor 1401 can receive light (modulated light or pulsed light) projected from the light source device 1411 toward the subject and reflected by the surface of the subject, thereby acquiring a distance image corresponding to the distance to the subject.
[0077] The optical system 1402 is formed by including one or more lenses and guides image light (incident light) from the subject to the photoelectric conversion device 1403, forming an image on the light-receiving surface (sensor portion) of the photoelectric conversion device 1403. The photoelectric conversion device 1403 is the same as the one described in the above embodiments, and a distance signal indicating the distance, obtained from the light-receiving signal output from the photoelectric conversion device 1403, is supplied to the image processing circuit 1404.
[0078] The image processing circuit 1404 performs image processing to create a distance image based on the distance signal supplied from the photoelectric conversion device 1403. The distance image (image data) obtained through image processing is then supplied to the monitor 1405 and displayed thereon, or supplied to the memory 1406 and stored (recorded) therein. The distance image sensor 1401 with this configuration can, for example, acquire a more accurate distance image as the characteristics of the pixels improve by applying the aforementioned photoelectric conversion device.
[0079] Reference Figure 17 The photoelectric conversion system according to the fifth application example is described, which is formed into an endoscopic surgical system. Figure 17 This illustrates the state in which an operator (doctor) 1131 uses an endoscopic surgical system 1103 to perform procedures on a patient 1132 on a bed 1133. (As shown) Figure 17 As shown, the endoscopic surgery system 1103 consists of an endoscope 1100, surgical instruments 1110, and a trolley 1134 on which various devices for endoscopic surgery are mounted.
[0080] Endoscope 1100 includes a lens tube 1101 comprising a region extending a predetermined length from the distal end, which is inserted into a body cavity of patient 1132, and a camera head 1102 connected to the proximal end of the lens tube 1101. Figure 17 The example shown illustrates an endoscope 1100 that is configured as a so-called rigid endoscope including a rigid lens barrel 1101, but the endoscope 1100 can be configured as a so-called flexible endoscope including a soft lens barrel.
[0081] An opening for mounting an objective lens is provided at the distal end of the lens barrel 1101. A light source device 1203 is connected to the endoscope 1100, and the light generated by the light source device 1203 is guided by a light guide extending inside the lens barrel 1101 to the distal end of the lens barrel 1101, and then emitted through the objective lens to the observation target in the body cavity of the patient 1132. Note that the endoscope 1100 can be a forward-looking endoscope, a forward-oblique endoscope, or a lateral endoscope.
[0082] An optical system and a photoelectric conversion device are provided in the camera head 1102, and reflected light (observation light) from the observed target is converged by the optical system to the photoelectric conversion device. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observed image. As the photoelectric conversion device, the photoelectric conversion device described in any of the above embodiments can be used. The image signal is transmitted as RAW data to the camera control unit (CCU) 1135.
[0083] The CCU 1135 comprises a central processing unit (CPU), a graphics processing unit (GPU), and other components, and comprehensively controls the operation of the endoscope 1100 and the display device 1136. Furthermore, the CCU 1135 receives image signals from the camera head 1102 and performs various image processing operations on the image signals, such as image development processing (de-mosaicing) for displaying images based on the image signals.
[0084] Under the control of the CCU 1135, the display device 1136 displays an image based on the image signal that has undergone image processing by the CCU 1135. The light source device 1203 is formed by a light source such as a light-emitting diode (LED) and supplies illumination light to the endoscope 1100 when imaging the operating parts, etc. The input device 1137 is an input interface to the endoscopic surgical system 1103. Users can input various information or commands into the endoscopic surgical system 1103 via the input device 1137. The treatment tool control device 1138 controls the drive of the energy treatment tool 1112 for purposes such as tissue ablation or incision, and sealing of blood vessels.
[0085] The light source device 1203 that supplies illumination light to the endoscope 1100 during imaging of the operating part can be formed, for example, a white light source formed by LEDs, a laser light source, or a combination thereof. If the white light source is formed by a combination of RGB laser light sources, then the output intensity and output timing of each color (each wavelength) can be accurately controlled, and therefore the light source device 1203 can adjust the white balance of the captured image. In this case, the target being observed is time-division irradiated by laser beams from RGB laser light sources respectively, and the drive of the image sensor of the camera head 1102 is controlled synchronously with the irradiation timing, thereby enabling the time-division capture of images corresponding to R, G, and B respectively. In this method, color images can be obtained without providing color filters in the image sensor.
[0086] The drive of the light source device 1203 can be controlled to change the intensity of the light to be output for each predetermined time interval. Images can be acquired in time division by controlling the drive of the image sensor of the camera head 1102 in synchronization with the timing of the change in light intensity, and the images can be combined to generate a high dynamic range image without so-called loss of shadow detail or highlight detail.
[0087] The light source device 1203 can be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the wavelength dependence of light absorption in body tissue is used. More specifically, by performing illumination with light in a narrow band, a predetermined tissue (such as blood vessels in the surface layer of a mucosa) is captured with high contrast compared to the illumination light during normal observation (i.e., white light). Alternatively, in special light observation, fluorescence observation can be performed to obtain an image by using fluorescence generated by performing radiation with excitation light. In fluorescence observation, for example, body tissue can be irradiated with excitation light and fluorescence from the body tissue can be observed, or a fluorescence image can be obtained by locally injecting the body tissue with excitation light corresponding to the fluorescence wavelength of a reagent such as indocyanine green (ICG) while simultaneously irradiating the body tissue with excitation light corresponding to the fluorescence wavelength of such a reagent. The light source device 1203 can be configured to supply narrow band light and / or excitation light corresponding to such special light observation.
[0088] Reference Figure 18A and Figure 18B The photoelectric conversion system according to the sixth application example is described, which is formed into smart glasses. Figure 18A This is a view illustrating an example of glasses 1600 (smart glasses) as a photoelectric conversion system. Glasses 1600 includes a photoelectric conversion device 1602. The photoelectric conversion device 1602 is the photoelectric conversion device described in any of the above embodiments. A display device including a light-emitting device such as an OLED or LED can be provided on the rear surface side of the lens 1601. One or more photoelectric conversion devices 1602 can be provided. Alternatively, multiple photoelectric conversion devices can be used in combination. The arrangement position of the photoelectric conversion device 1602 is not limited to... Figure 18A The location shown.
[0089] The glasses 1600 also include a control device 1603. The control device 1603 serves as a power source for supplying power to the photoelectric conversion device 1602 and the aforementioned display device. Furthermore, the control device 1603 controls the operation of the photoelectric conversion device 1602 and the display device. An optical system configured to focus light onto the photoelectric conversion device 1602 is formed on a lens 1601.
[0090] Figure 18B Glasses 1610 (smart glasses) according to another configuration example are described. Glasses 1610 includes a control device 1612, and a photoelectric conversion device corresponding to a photoelectric conversion device 1602 and a display device are mounted on the control device 1612. The photoelectric conversion device in the control device 1612 and an optical system configured to project light emitted from the display device are formed in a lens 1611, and an image is projected onto the lens 1611. The control device 1612 serves as a power source for supplying power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device may include a gaze detector for detecting the wearer's gaze. The gaze detection can be accomplished using infrared light. An infrared emitter emits infrared light towards the eyeball of a user gazing at the displayed image. An image capture unit including a light receiving element detects the reflected light from the emitted infrared light from the eyeball, thereby obtaining a captured image of the eyeball. A reducer is provided to reduce the light from the infrared emitter to the display unit in a planar view, thereby reducing image quality degradation.
[0091] The user's gaze at the displayed image is detected from a captured image of the eye obtained by capturing infrared light. Any known method can be applied to gaze detection using the captured image of the eye. As an example, a gaze detection method based on a Purkinje image obtained from light reflected from the cornea can be used. More specifically, gaze detection processing based on central pupillary corneal reflection is performed. Using the central pupillary corneal reflection, a gaze vector representing the direction (rotation angle) of the eye is calculated based on the image of the pupil included in the captured image of the eye and the Purkinje image, thereby detecting the user's gaze.
[0092] The display device according to this embodiment may include a photoelectric conversion device comprising a light receiving element, and control the displayed image of the display device based on the user's gaze information from the photoelectric conversion device. More specifically, the display device determines a first field of view region and a second field of view region other than the first field of view region based on the gaze information. The first and second field of view regions may be determined by a control device of the display device, or may be received from an external control device. Within the display area of the display device, the display resolution of the first field of view region may be controlled to be higher than the display resolution of the second field of view region. That is, the resolution of the second field of view region may be lower than the resolution of the first field of view region.
[0093] Furthermore, the display area includes a first display area and a second display area different from the first display area, and a higher priority area can be determined from the first and second display areas based on viewing information. The first and second field of view areas can be determined by the control device of the display device, or they can be received from an external control device. The resolution of the higher priority area can be controlled to be higher than the resolution of all other areas. That is, the resolution of the relatively low priority area can be low.
[0094] It's important to note that AI can be used to determine the primary field of view or higher priority areas. AI can be a model configured to use images of the eye and the actual direction the eye is looking in those images as supervisory data to estimate the angle of the gaze and the distance to a target object in front of the gaze. The AI program can be held by a display device, a photoelectric conversion device, or an external device. If an external device holds the AI program, it is transmitted to the display device via communication.
[0095] When performing gaze-based display control, smart glasses that also include a photoelectric conversion device configured to capture external images can preferably be applied. The smart glasses can display the captured external image information in real time.
[0096] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.
Claims
1. An optical-electric conversion apparatus comprising a pixel array including a plurality of pixels, a readout circuit configured to read out a signal from the pixel array, and a timing controller, wherein the timing controller defines a plurality of subframe periods forming each frame period and a plurality of detection periods forming each subframe period, each of the plurality of pixels includes a detector configured to detect a photon and an integrator configured to add, in each of the plurality of subframe periods, a weight value corresponding to a detection period in which the photon is detected by the detector among the plurality of detection periods to an integration value, and the timing controller sets the weight value in accordance with a gain with which a signal output from the readout circuit is multiplied by a processor configured to process the signal.
2. The apparatus according to claim 1, wherein in a case where the gain is smaller than a threshold value, the timing controller sets the weight value to a value larger than 1.
3. The apparatus according to claim 2, wherein in a case where the gain is larger than the threshold value, the timing controller sets the weight value to 1.
4. The apparatus according to claim 2, further comprising a programming unit configured to program the threshold value.
5. The apparatus according to claim 1, wherein in a case where the gain is smaller than a first threshold value and the gain is larger than a second threshold value larger than the first threshold value, the timing controller sets the weight value to a value larger than 1.
6. The apparatus according to claim 5, wherein in a case where the gain is larger than the first threshold value and smaller than the second threshold value, the timing controller sets the weight value to 1.
7. The apparatus according to claim 5, further comprising a programming unit configured to program the first threshold value and the second threshold value.
8. The apparatus according to claim 1, further comprising a gain setter configured to set the gain, wherein the timing controller sets the weight value in accordance with the gain set by the gain setter.
9. The apparatus according to claim 8, further comprising a sensitivity setter configured to set an ISO sensitivity, wherein the gain setter sets the gain in accordance with the ISO sensitivity set by the sensitivity setter.
10. The apparatus according to claim 1, further comprising the processor and a gain setter configured to set the gain, wherein the timing controller sets the weight value in accordance with the gain set by the gain setter.
11. The apparatus according to claim 10, further comprising a sensitivity setter configured to set an ISO sensitivity, wherein the gain setter sets the gain in accordance with the ISO sensitivity set by the sensitivity setter.
12. The apparatus according to claim 1, wherein the timing controller generates a plurality of pulse signals defining the plurality of detection periods.
13. The apparatus according to claim 12, wherein the integrator includes a counter and a counter controller configured to control the counter, and the counter controller causes the counter to perform a counting operation in each subframe period in accordance with a pulse signal following a pulse signal in which the photon is detected by the detector among the plurality of pulse signals.
14. The apparatus according to claim 13, wherein in each subframe period, intervals between the plurality of pulse signals are unequal intervals.
15. The apparatus according to claim 14, wherein In each subframe period, the interval between the plurality of pulse signals decreases monotonically.
16. The device according to any one of claims 1 to 15, wherein The detector includes an avalanche photodiode.
17. A photoelectric conversion system, comprising: The photoelectric conversion device according to any one of claims 1 to 16; as well as The image processing circuit is configured to generate an image using a signal output from a photoelectric conversion device.
18. A mobile body comprising the photoelectric conversion device according to any one of claims 1 to 16, comprising: A timing controller is configured to control the movement of a moving body using signals output from a photoelectric conversion device.
Citation Information
Patent Citations
Stacked SPAD image sensor
US20170186798A1
