Photoelectric conversion device

Abstract

To provide a photoelectric conversion device that can suppress the increase in consumption power of SPAD pixels.SOLUTION: A photoelectric conversion device 100 acquires images ranging from high sensitivity to low sensitivity once at the same time by setting a plurality of virtual sensitivity channels in one light exposure unit on the basis of a time aperture imaging algorithm. Specifically, information about multiple bits regarding incident photons is acquired and from the count value of the sensitivity channels resulting from the synthesis thereof, the number of incident photons is calculated. By adjusting the number of sensitivity channels and the time aperture ratio in the algorithm, the images in the high dynamic range can be acquired with fewer recharging times than a conventional method, and accordingly, the consumption power taken at the avalanche operation of SPAD can be suppressed largely.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device. [Background technology]

[0002] In recent years, photoelectric conversion devices (image sensors) using single-photon avalanche diodes (SPADs) have been attracting attention. SPADs utilize avalanche amplification, in which a large number of carriers are generated when a single photon is incident on a PN junction of an avalanche photodiode (hereinafter also referred to as "APD") to which a reverse bias exceeding the breakdown voltage is applied. A photoelectric conversion device using SPADs includes an APD for each of multiple pixels arranged in the row and column directions. A photoelectric conversion device using SPADs captures images by counting pulses generated based on carrier signals generated by avalanche amplification in these APDs.

[0003] Photoelectric conversion devices using SPADs have the advantage of being able to capture good images even in dark places (low illuminance) because they capture images by counting single incident photons.However, conventional photoelectric conversion devices using SPADs may increase the circuit size and power consumption as the number of counts increases in bright places (high illuminance).

[0004] In this regard, for example, Non-Patent Document 1 below discloses a photoelectric conversion device with a clock-synchronized recharge system, which aims to suppress increases in circuit size and power consumption when capturing images in bright locations. Generally, the appearance of photons within a unit time of natural light or artificial light that has undergone multiple reflections and diffusions, i.e., the time at which photons enter a photoelectric conversion device, is random due to their uncorrelated, non-interfering, and memoryless properties. Therefore, the probability density of photon appearance is constant, and the frequency of photon generation increases or decreases in proportion to the illuminance. Furthermore, the variation in this frequency of generation is known to follow a Poisson distribution, which is the root cause of so-called optical shot noise. Meanwhile, the interval at which photons arrive at a photoelectric conversion device is known to follow an exponential distribution, regardless of the illuminance intensity. The photoelectric conversion device in Non-Patent Document 1 is configured to sequentially set time segments corresponding to low, medium, and high illuminance within one frame, and to widen the dynamic range between the medium and high illuminance levels, the recharge operation interval is shortened as the illuminance increases. The recharge operation is an operation of supplying the APD with an excess voltage necessary to operate the APD as a SPAD. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] T. Takatsuka et al., “A 3.36 μm-pitch SPAD photon-counting image sensor using clustered multi-cycle clocked recharging technique with intermediate most-significant-bit readout,” 2023 IEEE Symposium on VLSI Technology and Circuits, C15-2, Kyoto, 2023. Summary of the Invention [Problem to be solved by the invention]

[0006] However, the photoelectric conversion device described in Non-Patent Document 1 still faces the problem of increased power consumption due to the need for more recharge operations in time segments assuming high illuminance. Furthermore, the recharge cycle becomes very short in time segments assuming high illuminance, resulting in significant fluctuations in power consumption and making it difficult to expand to a larger number of pixels. Furthermore, in time segments assuming high illuminance, a large number of counts occur in a very short period, necessitating an increase in the number of counter bits within the pixel to ensure complete counting. This also poses the problem of making pixel miniaturization difficult. Furthermore, this method, which sequentially scans pixels at different sensitivities serially along the time axis, creates a strong trade-off between expanding the dynamic range and the frame rate, making it difficult to output low-bit images at a high frame rate. This makes it difficult to apply image processing techniques, for example, to improve the image quality of moving images.

[0007] The present invention has been made to solve the above-mentioned problems, and therefore, a main object of the present invention is to provide a photoelectric conversion device that can suppress an increase in power consumption of pixels.

[0008] Another object of the present invention is to provide a photoelectric conversion device capable of suppressing fluctuations in pixel power consumption.

[0009] Another object of the present invention is to provide a photoelectric conversion device that can suppress an increase in the circuit scale of a pixel.

[0010] Another object of the present invention is to provide a photoelectric conversion device that can output low-bit images including binary data at a high frame rate, regardless of the degree to which the dynamic range for illuminance is expanded. [Means for solving the problem]

[0011] The above object of the present invention can be achieved by the following.

[0012] (1) A photoelectric conversion device having an avalanche photodiode including an anode and a cathode in each pixel, characterized in that the photoelectric conversion device has: a recharge unit that recharges the anode or the cathode once per unit exposure time; a gating unit that generates pulse signals based on the output of the avalanche photodiode at multiple different timings within the unit exposure time; and a counting unit that counts the pulse signals from the gating unit.

[0013] (2) The photoelectric conversion device according to (1), wherein the gating section receives the output signal of the avalanche photodiode and the plurality of determination signals at different timings to generate a pulse signal.

[0014] (3) The photoelectric conversion device described in (1) or (2) above, further comprising a normalization processing unit that adds up the count values ​​of the counting unit over multiple unit exposure times and normalizes the sum to calculate the normalized value as the number of photons that contributed to the occurrence of avalanche action in a specified frame.

[0015] (4) The photoelectric conversion device according to (3) above, further comprising a linearization processing unit that executes linearization processing on a nonlinear region range of the calculated number of photons.

[0016] (5) A photoelectric conversion device described in any one of (1) to (4) above, further comprising an arithmetic processing unit that adds up the count values ​​of the counting unit over multiple unit exposure times to generate a first intermediate image, and generates a second intermediate image by performing motion correction processing on the generated first intermediate image.

[0017] (6) A photoelectric conversion device described in any one of (1) to (5) above, which has a plurality of virtual sensitivity channels having different time lengths within the unit exposure time, and the plurality of virtual sensitivity channels are configured such that one virtual sensitivity channel encompasses other virtual sensitivity channels on the time axis.

[0018] (7) The photoelectric conversion device according to (6), wherein the different timings are the terminals of each of the virtual sensitivity channels.

[0019] (8) The photoelectric conversion device according to (6), wherein the plurality of virtual sensitivity channels are started at the same timing.

[0020] (9) The photoelectric conversion device according to any one of (1) to (8) above, wherein the pixel includes a latch section for holding the count value of the count section.

[0021] (10) A photoelectric conversion device according to any one of (1) to (9) above, comprising a first substrate and a second substrate stacked on the first substrate, a pixel having the avalanche photodiode on the first substrate, and the recharge unit, the gating unit, and the count unit on the second substrate.

[0022] (11) The photoelectric conversion device according to (10) above, further comprising a third substrate stacked on the second substrate, the third substrate comprising an upper counter unit connected in series with the count unit.

[0023] (12) The photoelectric conversion device according to any one of (1) to (11) above, further comprising a frame memory for storing an added value obtained by adding up the count values ​​of the counting section over a plurality of unit exposure times.

[0024] (13) The photoelectric conversion device according to any one of (1) to (12), wherein the gating section generates a pulse signal based on the output of the avalanche photodiode only at at least one timing selected from the plurality of different timings.

[0025] (14) A photoelectric conversion device described in any one of (1) to (12) above, wherein the gating unit generates a pulse signal based on the output of the avalanche photodiode at a timing corresponding to the end of a virtual sensitivity channel that is the most sensitive among the plurality of different timings.

[0026] (15) The photoelectric conversion device according to any one of (1) to (12) above, further comprising a sensitivity determining section that enables or disables a predetermined low-sensitivity virtual sensitivity channel for each pixel.

[0027] (16) A photoelectric conversion device as described in (15) above, having an upper counter unit connected in series with the count unit, and the sensitivity determination unit enables or disables the specified low-sensitivity virtual sensitivity channel based on the count state of the upper counter unit.

[0028] (17) The photoelectric conversion device described in (15) above, wherein the sensitivity determination unit determines the incident rate of photons from the count value of the counting unit and enables or disables the predetermined low-sensitivity virtual sensitivity channel based on the determination result.

[0029] (18) The photoelectric conversion device described in (15) above, wherein the sensitivity determination unit enables or disables the predetermined low-sensitivity virtual sensitivity channel based on a sum value obtained by adding up the count values ​​of the count unit over multiple unit exposure times.

[0030] (19) The photoelectric conversion device described in (12) above, wherein the frame memory has a high-sensitivity frame memory that holds count values ​​from a predetermined high-sensitivity virtual sensitivity channel, and a low-sensitivity frame memory that holds count values ​​from a predetermined low-sensitivity virtual sensitivity channel.

[0031] (20) The photoelectric conversion device according to (19), wherein the predetermined high-sensitivity virtual sensitivity channel is a virtual sensitivity channel with the highest sensitivity.

[0032] (21) The photoelectric conversion device according to (19) above, comprising a first converter for converting pixel data output from each pixel into a high-sensitivity count value, and a second converter for converting pixel data into a low-sensitivity count value. [Effects of the Invention]

[0033] The photoelectric conversion device of the present invention can suppress increases and fluctuations in pixel power consumption and can suppress increases in pixel circuit scale. Furthermore, the photoelectric conversion device of the present invention can output low-bit images at a high frame rate, regardless of the degree to which the dynamic range for illuminance is expanded. [Brief explanation of the drawings]

[0034] [Figure 1] 1 is a schematic block diagram illustrating the configuration of a photoelectric conversion device according to a first embodiment. [Figure 2] 2 is a functional block diagram illustrating functions of the pixel shown in FIG. 1. FIG. [Figure 3] FIG. 3 is a conceptual diagram for explaining a time aperture imaging algorithm in the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of the amount of information regarding the number of photons per recharge according to the sensitivity channel used. [Figure 5] FIG. 1 is a schematic diagram illustrating a conceptual model of incident photon detection using a temporal aperture imaging algorithm that achieves multiple sensitivities. [Figure 6] FIG. 10 is a diagram illustrating an example of a count value according to illuminance. [Figure 7] 10A and 10B are diagrams illustrating examples of dynamic range expansion ratios and in-pixel counter bit numbers corresponding to various combinations of time aperture ratios for sensitivity channels. [Figure 8] 10 is a graph illustrating a sensitivity channel transfer function and a normalized cumulative transfer function. [Figure 9] 10 is a diagram illustrating count values ​​according to illuminance and dominant sensitivity channels for each illuminance; FIG. [Figure 10] 10 is a graph showing an example of an SNR and a DR expansion rate. [Figure 11] 10 is a graph showing another example in which an SNR Dip occurs in the SNR and the DR expansion rate. [Figure 12] 10A and 10B are schematic diagrams for explaining switching of time aperture imaging parameters according to imaging illumination conditions. [Figure 13]FIG. 10 is a diagram illustrating an example of setting sensitivity ratios between sensitivity channels. [Figure 14] 10A and 10B are schematic diagrams illustrating a case where an image is reconstructed by performing correction processing on pixel data. [Figure 15] FIG. 10 is a schematic diagram illustrating a case where an image is reconstructed by simply adding pixel data. [Figure 16] 2 is a block diagram illustrating a schematic configuration of a signal processing unit shown in FIG. 1. [Figure 17] FIG. 1 is a circuit diagram illustrating a case where pixel hardware is configured on a two-layer substrate. [Figure 18] 18 is a timing chart for explaining the operation of the pixel circuit shown in FIG. 17. [Figure 19] FIG. 10 is a circuit diagram illustrating an equivalent circuit for calculating the power consumption at the cathode node of a SPAD. [Figure 20] 20 is a timing chart illustrating a charging and discharging operation in the equivalent circuit shown in FIG. 19. [Figure 21] 10A and 10B are schematic diagrams illustrating variations in power consumption during imaging using a temporal aperture imaging algorithm; [Figure 22] FIG. 10 is a schematic diagram illustrating, as a comparative example, fluctuations in power consumption during imaging using a conventional clock-synchronized recharge method. [Figure 23] 10 is a schematic diagram illustrating the count number of an in-pixel counter per exposure time unit in imaging using a time aperture imaging algorithm. FIG. [Figure 24] FIG. 10 is a schematic diagram illustrating, as a comparative example, the count number of a counter in a pixel per cluster in imaging using a conventional clock-synchronized recharge method. [Figure 25] FIG. 10 is a circuit diagram illustrating a case where pixel hardware is configured on a three-layer substrate in a photoelectric conversion device according to a second embodiment. [Figure 26] 26 is a timing chart for explaining the operation of the pixel circuit shown in FIG. 25. [Figure 27] FIG. 11 is a circuit diagram illustrating pixels and a data storage unit in a two-layer configuration in a third embodiment. [Figure 28] FIG. 2 is a schematic block diagram illustrating the layer configuration of a pixel. [Figure 29] FIG. 13 is a circuit diagram showing a modified example of the gating circuit in the fourth embodiment. [Figure 30] 30 is a timing chart illustrating the operation of the recharge circuit and the gating circuit shown in FIG. 29. [Figure 31] FIG. 13 is a schematic diagram for explaining detection of photons by a low-sensitivity channel under low illuminance in the fifth embodiment. [Figure 32] FIG. 1 is a circuit diagram illustrating a SPAD pixel for enabling a low sensitivity channel in a three-layer configuration. [Figure 33] 33 is a timing chart illustrating the operation of the pixel circuit of the SPAD pixel shown in FIG. 32. [Figure 34] FIG. 33 is a circuit diagram showing a first gating circuit using a logic circuit different from the example of FIG. 32. [Figure 35] FIG. 10 is a circuit diagram illustrating a SPAD pixel having a function of adaptively determining whether or not to use a low sensitivity channel in a two-layer configuration. [Figure 36] 10 is a timing chart for explaining determination of the amount of incident light based on the rate at which photons are incident. [Figure 37] FIG. 10 is a circuit diagram showing a modified example of a threshold determination circuit in a SPAD pixel that enables a low sensitivity channel. [Figure 38] 10 is a block diagram illustrating a configuration for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data. [Figure 39] FIG. 10 is a schematic diagram illustrating a method for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data. [Figure 40] 10A and 10B are schematic diagrams illustrating the relationship between pixel values ​​and high-sensitivity count values ​​and low-sensitivity count values; DETAILED DESCRIPTION OF THE INVENTION

[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following drawings, the same reference numerals refer to the same components, and the dimensions of each component in the drawings are not drawn to scale for clarity and convenience of explanation. Meanwhile, the embodiments described below are merely examples, and various modifications are possible from such embodiments.

[0036] In the following, the terms "upper" and "above" may include not only something that is directly above in contact with something, but also something that is above without contacting something.

[0037] An element expressed in the singular includes a plurality of elements unless the context clearly dictates otherwise. Furthermore, when a part "includes" or "has" a certain element, it does not mean excluding other elements, but means that other elements may also be included, unless specifically stated to the contrary.

[0038] Also, use of the term "said" and similar referents applies to both the singular and the plural.

[0039] Unless explicitly stated or stated to the contrary, steps constituting a method may be performed in any suitable order, and are not necessarily limited to the order of steps described. Any use of examples or exemplary terms (such as, for example, etc.) is intended to merely illustrate the technical idea, and the scope of the invention is not limited by said examples or exemplary terms, except as limited by the scope of the claims.

[0040] (First embodiment) <Configuration example of photoelectric conversion device> Fig. 1 is a schematic block diagram illustrating the configuration of a photoelectric conversion device according to the first embodiment, and Fig. 2 is a functional block diagram illustrating the functions of the pixel shown in Fig. 1.

[0041] As shown in FIG. 1, the photoelectric conversion device 100 includes a control unit 110, a pixel array unit 120, a data storage unit 140, and a signal processing unit .

[0042] The control unit 110 supplies various signals to the pixel array unit 120 to control the pixel array unit 120. The control unit 110 generates or inputs various signals such as a clock signal, a reset signal, and a control signal. The control signals include, for example, an RSTB signal, a recharge signal, a SEL signal, a CHK signal, a BITSEL signal, an EN_LS signal, a CHK_LS signal, a CHK_HS signal, an HLD signal, an FBK signal, and an EN_ADP signal. Details of the various signals will be described later.

[0043] The pixel array section 120 has a plurality of (for example, M) pixels 200 arranged in a two-dimensional lattice pattern (row and column directions) when viewed in a plan view perpendicular to the substrate surface of the photoelectric conversion device 100. Here, M is a natural number of 2 or more.

[0044] The signal processing unit 130 performs predetermined image processing on the pixel data from the pixel array unit 120 and outputs the processing result. The predetermined image processing may be, for example, a linearization process of a cumulative transfer function of the present invention, which will be described later, as well as a demosaic process, an AWB (Auto-White-Balance) process, a noise reduction process, a lens shading correction process, etc.

[0045] [Overview of the Pixel 200 configuration] As shown in FIG. 2, the pixel 200 includes a light receiving unit 210, a pixel circuit 220, and a sensitivity determination unit (arithmetic processing unit) 230. In this embodiment, an APD is used as the light receiving element, and therefore the pixel 200 is also referred to as a SPAD pixel 200. The light receiving unit 210 includes an APD 211 (see, for example, FIG. 17 ). The APD 211 is an avalanche photodiode. The pixel circuit 220 includes a recharge circuit (recharge unit) 221, a gating circuit (gating unit) 222, an in-pixel counter (count unit) 223, a readout latch circuit (latch unit) 224, a selection circuit (hereinafter also referred to as “SEL”) 225, an output bus 228, and the like. The sensitivity determination unit 230 performs a predetermined sensitivity determination on pixel data to select a sensitivity channel. The sensitivity determination unit 230 can be omitted as necessary. Details of the configurations of the pixel circuit 220 and the sensitivity determination unit 230 will be described later.

[0046] [Unit exposure time] The recharge circuit 221 recharges the cathode of the APD 211 once for each unit exposure time in response to a recharge signal received from the control unit 110. In other words, a unit exposure time is the time from one recharge to the next. The unit exposure time is a fixed period regardless of the amount of incident light. One recharge period is called one exposure time unit. In other words, the total exposure period is a collection of multiple exposure time units. The recharge signal is a one-shot trigger signal with a period (e.g., 4 μs) corresponding to the unit exposure time. In this embodiment, for example, the anode of the APD 211 is set to a fixed potential, and the cathode is recharged, thereby setting the potential difference between the anode and cathode to a predetermined voltage that allows the APD 211 to operate in Geiger mode. Specifically, a potential difference calculated by combining a fixed avalanche breakdown voltage built into the device and an externally controllable excess voltage is applied between the anode and cathode. An APD 211 operating in Geiger mode is called a SPAD 211. The configuration is not limited to one in which the anode has a fixed potential, but conversely, the cathode may have a fixed potential and the anode may be recharged.

[0047] The gating circuit 222 generates pulse signals at different CHK signal timings within a unit exposure time based on the output signal of the APD 211. The timing of generating pulse signals and the specific hardware configuration of the gating circuit 222 will be described later.

[0048] The in-pixel counter 223 counts pulse signals from the gating circuit 222 to detect and count photons that are incident on one pixel 200 and contribute to the occurrence of an avalanche action, but not all photons that are incident on the pixel 200 necessarily contribute to the counting. A specific hardware configuration of the in-pixel counter 223 will be described later (see, for example, FIG. 17).

[0049] <Counting the number of photons per unit exposure time> [Subframe] FIG. 3 is a conceptual diagram for explaining the time aperture imaging algorithm in this embodiment. In this embodiment, one frame can include multiple subframes. In the example shown in FIG. 3, one frame includes four subframes. When one frame (e.g., 16.7 ms) includes four subframes, one subframe corresponds to a period of 4.2 ms. Furthermore, one subframe can include multiple exposure time units. FIG. 3 illustrates an example in which one subframe includes 1024 exposure time units. Therefore, in this case, 1024 recharge operations are performed in one subframe.

[0050] [Sensitivity Channel] The number of photons reaching the APD 211 increases or decreases depending on the level of illuminance, but the average arrival interval of photons at the APD 211 is known to follow an exponential distribution, regardless of the level of illuminance. On the other hand, since photons arrive at the APD 211 randomly, it is thought that there is no bias in the probability density of photons reaching the APD 211 at a certain time during the recharge period. Utilizing this randomness, the time aperture imaging algorithm specifies the exposure time using a time aperture, and sets multiple virtual sensitivity channels in one pixel.

[0051] In this embodiment, multiple virtual sensitivity channels T i (i=1, 2, ...) is set. In FIG. 3, each sensitivity channel T i In the figure, exposure starts at the "Start of Exposure (SOE)" indicated by a black downward triangle, and ends at the "End of Exposure (EOE)" indicated by a black upward triangle. The end of exposure is also the timing for confirming photon detection. In other words, photon detection is confirmed by multiple virtual sensitivity channels T i The control unit 110 controls the sensitivity of each channel T i The CHK signal (determination signal) is made active at a plurality of different timings at the end of the exposure.

[0052] Each sensitivity channel T i When the recharge signal is low active, the exposure starts when the recharge signal becomes LO (low), and the exposure time is i The exposure ends when the time aperture period determined for each sensitivity channel (i=1, 2, ...) has elapsed. Therefore, the timing of confirming photon detection differs from one another. On the other hand, the start of exposure is the same for all sensitivity channels T i In other words, multiple sensitivity channels T i Therefore, each sensitivity channel T iThe time aperture periods of the channels overlap each other from the start of exposure during a period corresponding to the time aperture period of the least sensitive channel (T7 in FIG. 3).

[0053] In this manner, in this embodiment, the photoelectric conversion device 100 generates a plurality of virtual sensitivity channels T i It has multiple sensitivity channels T i The exposure starts at the same start timing, ends at different end timings, and photon detection is confirmed. i is configured such that one virtual sensitivity channel encompasses other virtual sensitivity channels on the time axis.

[0054] FIG. 3 illustrates an example in which sensitivity channels T1 to T7 are set. Below, the case in which sensitivity channels T1 to T7 or T1 to T3 are set will be mainly illustrated and explained, but the present invention is not limited to these cases. Sensitivity channel T1 is the most sensitive sensitivity channel, having a time opening period that is 1 time the unit exposure time. In other words, sensitivity channel T1 has sensitivity to detecting (counting) photons over the same time as the unit exposure time. T2 is 1 / 2 (=1 / 2) of the unit exposure time. 1 ) times the unit exposure time. In other words, sensitivity channel T2 is sensitive to detecting photons for half the time of the unit exposure time. T3 is sensitive to detecting photons for half the time of the unit exposure time. 2 ) times the unit exposure time. In other words, sensitivity channel T1 is sensitive to detecting photons for a period of one-fourth the unit exposure time. The same is true for T4 to T7.

[0055] The ratio of the time aperture period to the unit exposure time of the sensitivity channels T1 to T7 (hereinafter referred to as "Time Aperture Ratio (TAR)") is not limited to the above-mentioned ratios such as 1 / 2 and 1 / 4. However, for convenience of the hardware configuration, the time aperture ratio is 1 / 2. k(k is preferably set to 0 or a natural number.) As will be described later, various time aperture ratios can be combined with the sensitivity channels T1 to T7.

[0056] [How to count the number of photons per unit exposure time] The higher the illuminance of light incident on the APD 211 (hereinafter also referred to as "incident illuminance"), the greater the number of photons incident on the APD 211. Furthermore, the start of exposure is common to each of the sensitivity channels T1 to T7, and photons reach the APD 211 randomly, so the higher the incident illuminance, the higher the probability that photons will be incident during one recharge period. In particular, the higher the incident illuminance, the higher the probability that photons will be incident on a low-sensitivity sensitivity channel (hereinafter also referred to as a "low-sensitivity channel") with a short time aperture period.

[0057] Also, the sensitivity channel T i The time aperture periods (i=1, 2, ...) partially overlap each other, so that a certain sensitivity channel T i When a photon is incident on the sensitivity channel T i Another sensitivity channel T j Photons are also incident on (j>i). As a result, regardless of the time dependency of the timing at which photons are incident, the time aperture period of each of the sensitivity channels T1 to T7 can be associated with the level of sensitivity of each of the sensitivity channels T1 to T7, i.e., the weighting related to the sensitivity. In other words, the time aperture period can be treated as synonymous with the weighting related to the sensitivity of the sensitivity channel.

[0058] In the time aperture imaging algorithm, the number of photons incident on the APD 211 is determined by the sensitivity channel T i Photon count value by CNT i The method for calculating the number of photons will be described in detail later.

[0059] The count value CNT is initialized to 0 before counting begins. For each sensitivity channel T1 to T7, it is determined whether or not a photon has been incident during the time aperture period, and if a photon has been incident, 1 is added to the count value CNT. That is, for each of the sensitivity channels T1 to T7, it is determined whether or not a photon has been incident from the start of exposure to the end of exposure, and if a photon has been incident, 1 is added to the count value CNT. On the other hand, if a photon has not been incident, nothing is added. For example, if a photon is incident on the sensitivity channel T7, 1 is added to the count value CNT for the sensitivity channel T7. Furthermore, since the start of exposure is common to all of the sensitivity channels T1 to T7, 1 is also added to the count value CNT for each of the sensitivity channels T1 to T6. Therefore, the final count value CNT is 7. Furthermore, if a photon is incident during a period when only the sensitivity channel T1 is set, the final count value CNT is 1. If no photons are incident on the sensitivity channel T1, the final count value CNT is 0. In this embodiment, the sensitivity channel T i Counting of photons by the pixel counter 223 is realized by the pixel counter 223, for example, as will be described later.

[0060] [Information amount regarding the number of photons per unit exposure time] FIG. 4 is a diagram illustrating an example of the amount of information regarding the number of photons per recharge according to the sensitivity channel used.

[0061] If the amount of information I related to the number of photons is defined as the logarithm log2(CNT+1) with base 2, when sensitivity channels T1 to T7 are used, the amount of information I is a maximum of 3 bits per recharge. i The larger the number, the more information is obtained about the incident illuminance in one recharge operation. CNT in the definition of the amount of information I above represents the final count value.

[0062] In this manner, in this embodiment, information regarding the amount of incident light is expressed using multiple bits (multi-bit) and acquired at once (referred to as "acquiring multi-bit information"). That is, the photoelectric conversion device 100 can acquire a large amount of information regarding the amount of incident light for each level of illuminance at once by utilizing the statistical properties of photons.

[0063] [Conceptual model of imaging using temporal aperture] Figure 5 is a schematic diagram illustrating a conceptual model for detecting incident photons using a temporal aperture imaging algorithm that achieves multiple sensitivities. Multi-bit information acquisition can be modeled using the conceptual model shown in Figure 5. In the conceptual model shown in Figure 5, a single pixel appears to have multiple SPAD elements, and an ND (Neural Density) filter corresponding to the temporal aperture ratio is placed in front of each SPAD element. Each ND filter has a different sensitivity characteristic depending on the temporal aperture ratio. Multi-bit information acquisition can be considered equivalent to counting all outputs from multiple SPAD elements in response to incident light attenuated by the ND filters (or calculating the sum of the outputs). Conventional CIS (CMOS Image Sensor) devices are designed to accumulate photoelectrically converted electrons once within the pixel regardless of the amount of light, making it impossible to model such multiple sensitivity characteristics.

[0064] [Dynamic range for illuminance] Fig. 6 is a diagram illustrating an example of count values ​​corresponding to the level of illuminance, and Fig. 7 is a diagram illustrating an example of dynamic range expansion ratios and in-pixel counter bit numbers corresponding to various combinations of time aperture ratios for sensitivity channels.

[0065] In this specification, the illuminance range in which the photoelectric conversion device 100 can detect (count) photons is referred to as the dynamic range (DR) of illuminance (hereinafter simply referred to as "dynamic range"). For ease of explanation, the following will exemplify a case in which the incident illuminance is divided into three categories: high illuminance, medium illuminance, and low illuminance. The incident illuminance may be divided into any number of categories.

[0066] High illuminance refers to extremely high illuminance, such as sunlight. At high illuminance, a large number of photons enter within a certain period of time. Medium illuminance refers to the level of illuminance that can be achieved with general indoor lighting. Low illuminance refers to the illuminance in a dark lighting environment, such as under moonlight at night. At low illuminance, it is possible that not even a single photon enters during a single recharge period.

[0067] As shown in Figure 6, the number of photons incident during a single recharge period increases with increasing illuminance. Therefore, the probability of photons being incident even during the short time aperture period set for the low-sensitivity channel increases, resulting in a larger final count value CNT. For example, the time aperture ratio between the least sensitive sensitivity channel T7 and the most sensitive sensitivity channel T1 is 64:1. This means that sensitivity channel T7 suppresses the increase in count value even when exposed to light with an illuminance 64 times higher than that of sensitivity channel T1. This means that counting can be performed without saturating the count value with a small number of in-pixel counter bits. This means that the illuminance range in which each sensitivity channel operates linearly can be controlled by the time aperture ratio, thereby easily expanding the dynamic range of pixel 200.

[0068] For each of the high, medium, and low illuminance ranges, the APD 211 measures (detects) the first incident photon during the recharge period. However, because the potential difference between the anode and cathode decreases due to the measurement of the first photon, the APD 211 does not measure the second or subsequent incident photons. In other words, the second or subsequent photons are ignored. In the high illuminance example shown in FIG. 6 , the first incident photon is measured during the time aperture period of sensitivity channel T6 after the time aperture period of sensitivity channel T7, and the second or subsequent incident photons are ignored. However, the time aperture imaging algorithm of this embodiment counts the photons from sensitivity channels T6 to T1, resulting in a count value of 6. In the medium illuminance example, the first incident photon is measured during the time aperture period of sensitivity channel T4 after the time aperture period of sensitivity channel T5, and the second or subsequent incident photons are ignored, but the photons are counted during the time aperture period of sensitivity channel T4 to T1, resulting in a count value of 4. In addition, in the example of low illuminance, the first incident photon in the time aperture period of sensitivity channel T2 after the time aperture period of sensitivity channel T3 is measured, and the second and subsequent incident photons are ignored, but are counted in sensitivity channels T2 and T1, resulting in a count value of 2.

[0069] The number of sensitivity channels to be set may be one or more. To obtain the effect of expanding the dynamic range, the sensitivity channel T i It is preferable to set multiple sensitivity channels T i The time aperture period of each sensitivity channel T must be set to include the time aperture period of the less sensitive sensitivity channel. By restricting the setting of such sensitivity channels, the time dependency is cancelled even for randomly arriving photons, and the time of each sensitivity channel T i It is possible to realize an input / output transfer function, which will be described later, in which the sensitivity ratio is maintained.

[0070] As shown in Figure 7, each sensitivity channel T iBy setting the time aperture ratio (TAR), which is the ratio of the sensitivity channel T1 to the dynamic range expansion ratio DR EXT follows the following formula (1):

[0071]

number

[0072] The figure illustrates the dynamic range (DR) expansion calculated for cases 1 to 3. For each case, the figure also illustrates the number of bits of the in-pixel counter required to count photons in hardware. This characteristic of easily controlling the expansion of the dynamic range is referred to in this specification as "DR expansion controllability."

[0073] As will be explained below, the number of sensitivity channels and the time aperture ratio can be used to suppress dips in the signal-to-noise ratio (SNR) and to design a trade-off between the SNR and the DR magnification rate (referred to as "ease of trade-off design"). Furthermore, by considering the trade-off design and providing modes according to the imaging scene, it is possible to optimize the power consumption of the entire imaging system and the signal processing within the imaging system.

[0074] As mentioned above, the photon mean arrival time interval follows an exponential distribution, so in general, in a SPAD pixel with a clock-synchronized recharge method, as the photon mean arrival time interval approaches the recharge period interval, the probability that multiple photons will be incident during one recharge period increases. Therefore, due to this deviation in the arrival time interval, the count value exhibits nonlinearity near the point where it saturates, but for simplicity's sake, we will assume that the count value has linear characteristics.

[0075] Figure 8 is a graph illustrating the sensitivity channel transfer functions obtained by adding up pixel count values ​​for a unit exposure time of one frame period, and the normalized cumulative transfer function. The cumulative transfer function is obtained by accumulating the sensitivity channel transfer functions. In Figure 8, the horizontal axis represents the logarithm of illuminance, and the vertical axis represents the logarithm of the count value.

[0076] Sensitivity channel T1 can perform counting operations from low illuminance I1 to medium illuminance I2. During the period from low illuminance I1 to medium illuminance I2, sensitivity channel T1 is linear, and sensitivity channel T1 dominates over the other sensitivity channels. Sensitivity channel T2 can perform counting operations from medium illuminance I2 to high illuminance I3. During the period from medium illuminance I2 to high illuminance I3, sensitivity channel T2 is linear, and during the period from medium illuminance I2 to high illuminance I3, sensitivity channel T2 dominates over the other sensitivity channels. Sensitivity channel T3 can perform counting operations up to a predetermined illuminance even after the incident illuminance becomes high illuminance I3. During the period until the incident illuminance reaches the high illuminance I3, the sensitivity channel T3 is linear, and during the period until the incident illuminance falls from the high illuminance I3 to the predetermined illuminance, the sensitivity channel T3 becomes dominant over the other sensitivity channels.

[0077] FIG. 9 is a diagram illustrating count values ​​according to illuminance and dominant sensitivity channels for each illuminance. For example, at low illuminance I1, the count value in sensitivity channel T1 is C1L, the count value in sensitivity channel T2 is C2L, and the count value in sensitivity channel T3 is C3L. At low illuminance I1, the dominant sensitivity channel is sensitivity channel T1. At medium illuminance I2, the count value in sensitivity channel T1 is Max (maximum value), the count value in sensitivity channel T2 is C2M, and the count value in sensitivity channel T3 is C3M. At medium illuminance I2, the dominant sensitivity channel is sensitivity channel T2. At high illuminance I3, the count value in sensitivity channel T1 is Max, the count value in sensitivity channel T2 is also Max, and the count value in sensitivity channel T3 is C3H. At high illuminance I3, the dominant sensitivity channel is sensitivity channel T3.

[0078] When the illuminance increases beyond medium illuminance I2, sensitivity channel T1 saturates and becomes nonlinear, and is no longer able to support counting, but sensitivity channel T2 remains linear even in that range and is more sensitive than sensitivity channel T3, so sensitivity channel T2 plays a central role in increasing the integrated count value obtained by integrating the counts. Also, when the illuminance increases beyond high illuminance I3, sensitivity channels T1 and T2 become nonlinear and then saturate, and are no longer able to support counting, but sensitivity channel T3 remains linear up to a certain illuminance and has a certain sensitivity, so sensitivity channel T3 plays a central role in increasing the integrated count value.

[0079] Because the sensitivity channels T1, T2, and T3 have a constant sensitivity ratio with respect to illuminance, the composite count value obtained by normalizing the integrated count value shows a monotonically increasing trend with respect to illuminance. Furthermore, by normalizing the integrated count value with a normalization factor NF (Normalization Factor), the number of incident photons can be accurately determined after correcting for the effects of the number of sensitivity channels and the time aperture ratio, i.e., after correcting for sensitivity multiplication. NF is expressed by the following equation (2):

[0080]

number

[0081] In the above formula (2), T i is the sensitivity channel, TAR is the temporal aperture ratio, and NC is the total number of sensitivity channels.

[0082] Furthermore, whether or not photons detected during the recharge period contribute to the count value is determined solely by quantum operation, which does not include readout noise, due to the extremely high photoelectric conversion gain of the SPAD 211. Furthermore, because the cumulative transfer function obtained by combining each sensitivity channel is uniquely determined, it is possible to accurately calculate the number of incident photons as a completely linear count value proportional to the incident illuminance through processing by the pre-processing unit 131, which will be described later. However, the calculated value also includes the optical shot noise inherent in photons.

[0083] In this manner, in this embodiment, a plurality of sensitivity channels T i is used. Because photons have an evenly distributed probability of appearing on average during one recharge period, any desired sensitivity can be achieved by changing the time aperture rate during the very short recharge period. For example, by making the time aperture very short, it is possible to achieve an ultra-low sensitivity channel, and to obtain images that do not saturate even when the subject is very bright. This means that even a bright subject that would normally require 1 million counts in one frame period can be represented with just 10,000 to 20,000 counts.

[0084] [Input / output transfer function] Assuming that the photons incident on the APD 211 vary on average during the charging period depending on the incident illuminance lx, the sensitivity of each of the incident photons to each sensitivity channel T i Count value (detection value) CNT(T iHowever, since the illuminance lx is proportional to the number of incident photons, the illuminance lx represents the number of photons in the following formula. Furthermore, the count value CNT(T i By integrating the counts (lx) for the number of sensitivity channels NC, the expected value E[CNT(lx)] of the count value in one recharge can be calculated using the following formula (3).

[0085]

number

[0086] The expected value includes both the nonlinearity caused by the clock-synchronized recharge operation and the dimming characteristics due to the time aperture method. By integrating the expected value for the number of recharges NR, it is possible to calculate the integrated count value at a certain incident illuminance lx, i.e., the transfer function. Furthermore, by multiplying the calculated transfer function by the normalization coefficient NF, it is possible to calculate the normalized aggregated transfer function (Equation (4) below).

[0087]

number

[0088] In addition, the expected count value E[C M (lx)] can be calculated using the following formula (5) from the expected value when no photons are incident during the recharge period and when one or more photons are incident in the Poisson distribution. ETU is the time in one exposure time unit.

[0089]

number

[0090] [SNR, dynamic range (DR) expansion rate] FIG. 10 is a graph showing an example of the SNR and the DR expansion ratio, and FIG. 11 is a graph showing another example of the SNR and the DR expansion ratio where an SNR Dip occurs.

[0091] For example, consider a case where three sensitivity channels T1, T2, and T3 are used. If the time-aperture ratio of sensitivity channels T2 and T3 is set to T1:T2:T3 = 1:1 / 4:1 / 4, the SNR increases monotonically, but the resulting DR magnification is limited to 16x. On the other hand, as shown in Figure 11, if the time-aperture ratio of sensitivity channels T2 and T3 is set to T1:T2:T3 = 1:1 / 4:1 / 16, the count value of T3 is low when T2 saturates. This results in a lower SNR than when the illuminance range (DR magnification) is small. In other words, an SNR dip occurs. However, even if the SNR dip occurs, it is known that humans cannot see anything above a certain level. By utilizing this characteristic, a high DR magnification—64x in this case—can be achieved while maintaining a sufficiently high SNR.

[0092] Furthermore, if the SNR is already sufficiently high and the occurrence of a dip has almost no effect on image quality, a wider dynamic range can be achieved using a smaller number of sensitivity channels by relatively increasing the temporal aperture ratio. In other words, a smaller number of sensitivity channels can reduce the number of counter bits and power consumption. As such, this embodiment is characterized by its ability to easily design a performance trade-off between SNR and DR magnification.

[0093] [Switching temporal aperture imaging parameters] Fig. 12 is a schematic diagram for explaining switching of time aperture imaging parameters according to imaging illumination conditions, and Fig. 13 is a diagram showing an example of setting the sensitivity ratio between each sensitivity channel.

[0094] As shown in FIG. 12 , the photoelectric conversion device 100 can be configured to switch time-aperture imaging parameters depending on the imaging illumination conditions. The imaging illumination conditions are conditions related to the illumination at the imaging location, such as a scene with standard illumination, a dark scene, or a scene with a large difference between light and dark (high dynamic range). The time-aperture imaging parameters are parameters used in time-aperture imaging, and in this embodiment, are, for example, the sensitivity ratio between each sensitivity channel. The user prepares a set of time-aperture imaging parameters depending on each imaging illumination condition and the corresponding expected subject and use case. For example, the user inputs the set of time-aperture imaging parameters into the photoelectric conversion device 100 when capturing an image, or selects pre-registered time-aperture imaging parameters. This allows the photoelectric conversion device 100 to acquire (capture) an image under optimal operating conditions. Alternatively, the photoelectric conversion device 100 may be configured to determine the illumination range from the currently acquired image and automatically switch the time-aperture imaging parameters.

[0095] 13, for example, in case A, the image is captured under a standard illumination environment, so the sensitivity ratios of the four sensitivity channels T1 to T4 are set to be uniformly reduced to 1 / 4. This results in a dynamic range expansion ratio of 4 x 4 x 4 = 64 times.

[0096] In case B, since the image is taken under moonlight at night and sufficient illumination is not available, more sensitivity channels are allocated to the high sensitivity side. This improves the image quality in low illumination. Sensitivity channel T6 is also set to be used. This allows people and objects walking at night to be recorded brightly without being saturated with white.

[0097] Additionally, in case C, because it is a high dynamic range scene in which high and low illumination exist in the same scene, all seven sensitivity channels T1 to T7 are used, and the sensitivity ratio between T2 and T3 is reduced to improve image quality in the low illumination side as well. This expands the dynamic range to the high illumination side while maintaining low illumination image quality, so that, for example, it is possible to capture details of passengers inside a car at the same time as capturing an object illuminated by the bright lights of the car.

[0098] [Configuration of data storage unit 140] Fig. 14 is a schematic diagram illustrating a case where an image is reconstructed by performing correction processing on pixel data, and Fig. 15 is a schematic diagram illustrating a case where an image is reconstructed by simply adding pixel data.

[0099] Unlike CIS imaging, SPAD imaging does not generate readout noise or quantization noise due to its extremely high photoelectric conversion gain. In other words, SPAD imaging has zero readout noise and quantization noise, and images can be obtained during the imaging process. Therefore, even when SPAD imaging is performed by accumulating images during the accumulation process, there is no degradation of SNR due to noise. In fact, the image quality improves with increasing accumulation, as the shot noise ratio to the signal decreases. Thus, SPAD imaging has excellent characteristics in terms of SNR and image quality.

[0100] As shown in FIG. 14 , the data accumulation unit 140 first generates a first intermediate image by simply adding together several one-exposure time units as a first process. The data accumulation unit 140 then performs motion correction processing on some of the first intermediate images to generate a combined second intermediate image. The data accumulation unit 140 then performs motion blur correction processing on the second intermediate images. Then, as a second process, the data accumulation unit 140 generates a final subframe image by again performing integration and motion correction processing on the second intermediate images that have undergone blur correction processing. Note that the second process does not necessarily have to be performed. The motion correction processing and blur correction processing may also be combined with denoising processing. When the amount of motion is small, motion blur correction may be performed on a subframe-by-subframe basis.

[0101] In this embodiment, the temporal aperture imaging algorithm already obtains information from high to low illumination levels at the time of one exposure time unit frame, so motion compensation can be performed for the entire illumination range, resulting in clear moving images across the entire illumination range.

[0102] In addition, the temporal aperture imaging algorithm outputs information from high to low illumination acquired during a single short recharge period as interim accumulated images at a high frame rate. This allows for the accumulation of interim accumulated images over a wide range of illumination levels, from low to high, while simultaneously aligning the subject in a series of scenes, even though the amount of information per interim accumulated image is small. Furthermore, by combining the accumulation of interim accumulated images with the first and second processes, motion blur caused by subject movement or camera movement can be suppressed, and high dynamic range video images with a good SNR, particularly for low-light subjects, can be generated. This improves the image quality of high dynamic range video images.

[0103] On the other hand, if a 3-bit frame image is acquired in one exposure time unit and the acquired frame images are simply added, an image can be acquired without motion blur even if the subject is moving quickly under conditions other than low illumination, as shown in Figure 15. However, if the subject is moving quickly, particularly under low illumination, sufficient contrast may not be obtained.

[0104] [Configuration of signal processing unit 130] Fig. 16 is a block diagram illustrating a schematic configuration of the signal processing unit 130 shown in Fig. 1. The signal processing unit 130 includes a pre-processing unit 131 and an image processing unit 132.

[0105] The pre-processing unit 131 performs normalization and linearization on the image (time-aperture raw image) generated by the time-aperture imaging algorithm (referred to as a "pre-correction processing flow"). The pre-processing unit 131 includes a time-aperture normalization unit (normalization processing unit) 21, an inverse function calculation unit 22, a look-up table (LUT) 23, and a transfer function linearization unit (linearization processing unit) 24.

[0106] The time aperture normalization unit 21 calculates a normalized count value (accurate number of photons) by performing a normalization process on the image (raw data of count values) generated by the time aperture imaging algorithm. More specifically, as shown in the graph at the bottom left of Fig. 16, the cumulative transfer function before normalization is in a state where the sensitivity has been multiplied by multiple sensitivity channels (gap G between arrows), so normalization restores the original sensitivity and calculates an accurate number of photons.

[0107] The inverse function calculation unit 22 calculates the cumulative transfer function TF agg,norm Calculate the inverse function of (lx). As mentioned above, the cumulative transfer function TF agg,norm(lx) has a characteristic that is a combination of the light attenuation associated with the time aperture imaging algorithm, the photon arrival interval that follows an exponential distribution, and the nonlinear characteristics associated with the clock-synchronized recharge method. The inverse function of the cumulative transfer function, Lin(lx), is a gain correction coefficient that returns the nonlinear region of the calculated photon number to linear, and is expressed by the following equation (6).

[0108]

number

[0109] where TF LIN (lx) is a linear function of a certain reference illuminance lx, including photon detection efficiency such as quantum efficiency and avalanche occurrence probability. The cumulative transfer function TF agg,norm Since the formula for (lx) is complicated, it is difficult to analytically determine the number of incident photons. Therefore, in the inverse function calculation unit 22, the cumulative transfer function TF agg,norm (lx) and TF LIN The input / output relationship is determined in advance from (lx), and the relationship between the output and the input is written into the LUT 23 as an inverse function Lin(lx) based on the input / output relationship, thereby realizing the calculation of the number of incident photons.

[0110] The transfer function linearizer 24 performs linearization processing on the nonlinear region (e.g., corresponding to a high illuminance region) of the calculated photon count. More specifically, the transfer function linearizer 24 refers to the LUT 23 and performs processing to replace the photon count (count value) calculated by the time aperture normalizer 21 with the corresponding illuminance, thereby linearizing the correspondence relationship between the illuminance and the count value. The transfer function linearizer 24 is used, for example, in the case of a color image in which high illuminance regions need to be linearized. Linearization by the transfer function linearizer 24 does not have to be used for monochrome images.

[0111] The image processing unit 132 uses a general image signal processing pipeline to perform various processes such as demosaic, AWB (Auto White Balance), high dynamic range tone mapping, color processing, etc. By performing a pre-correction processing flow using the pre-processing unit 131, it becomes possible to apply conventional general color reproduction and image processing to images captured using a time aperture imaging algorithm.

[0112] <Example of hardware configuration of pixel 200> [Example of a two-layer structure] 17 is a circuit diagram illustrating a case where the hardware of the pixel 200 is configured with a two-layer substrate. For example, the pixel 200 may have a light receiving unit 210 including an APD 211 mounted on a first layer (first substrate) and a pixel circuit 220 mounted on a second layer (second substrate). An output signal from the APD 211 on the first layer is transmitted to the pixel circuit 220 on the second layer using hybrid bonding stacking technology.

[0113] The pixel circuit 220 in the second layer has a recharge circuit 221 , a gating circuit 222 , an in-pixel counter 223 , a read latch circuit 224 , a SEL 225 , and an output bus 228 .

[0114] The recharge circuit 221 recharges the cathode of the APD 211 once for each unit exposure time based on the recharge signal.

[0115] The gating circuit 222 controls the clock input of the in-pixel counter 223, thereby controlling the count-up operation of the in-pixel counter 223. More specifically, the gating circuit 222 has a NOR element, which receives the cathode node potential Vc and the CHK signal and outputs an OUT signal.

[0116] The in-pixel counter 223 is a ripple counter having a 3-bit counter, and is initialized to 0 by the RSTB signal. The in-pixel counter 223 counts up (increments) at the rising edge of the OUT signal to detect the output (photons) of the APD 211. The in-pixel counter 223 is connected to the sensitivity channel T i Any number of bits can be used as long as it can count values ​​equal to or greater than the number of lines. Generally, the size of the SPAD pixel 200 is determined by the size of the in-pixel counter 223, so the smaller the number the better. Figure 17 shows a 3-bit configuration as an example.

[0117] The read latch circuit 224 holds each bit output of the in-pixel counter 223 in response to a recharge signal. The SEL 225 controls the readout of the bit value held in the read latch circuit 224 via the output bus 228 in response to the SEL signal. The width (number of bits) of the output bus 228 preferably matches the number of bits in the in-pixel counter, but they may differ. A smaller output bus width reduces the layout area required for wiring, making it easier to miniaturize the entire pixel. Alternatively, only the most significant bit of the in-pixel counter 223 may be held in the read latch circuit 224.

[0118] In this embodiment, a read latch circuit 224 is provided to read out the count value of the in-pixel counter 223 so that it overlaps with the time aperture imaging. Readout of all pixels is completed in one recharge period. In this case, the unit exposure time and the readout time can be overlapped, so no dead period for imaging due to readout occurs, and imaging can be performed without sensitivity loss. Note that instead of providing the read latch circuit 224, one or more extra bits may be implemented in the in-pixel counter 223, allowing for slow readout over one recharge period or more.

[0119] [Example of operation of a two-layer pixel circuit] Fig. 18 is a timing chart for explaining the operation of pixel circuit 220 shown in Fig. 17. However, to simplify the explanation of counter operation, values ​​different from those in actual circuit operation are used. Specifically, the value of intra-pixel counter 223 is decremented when a clock is input, but the operation itself can be described without inconsistency even if it is treated as being incremented.

[0120] (1) When the recharge signal becomes LO level, the cathode node potential Vc of the APD 211 is pulled up to HI level, causing the APD 211 to operate in Geiger mode and become ready to receive photons.

[0121] (2) When a photon is incident on the ADP 211, an avalanche breakdown occurs, and the cathode node potential Vc drops to the LO level.

[0122] (3) When RSTB of the in-pixel counter 223 becomes LO, the in-pixel counter 223 is cleared to 0. This prepares the counter for counting.

[0123] (4) When the CHK signal becomes LO, the logic state of the cathode node potential Vc is propagated to the OUT signal and confirmed. If the cathode node potential Vc is HI, the OUT signal becomes LO according to the NOR logic. If the cathode node potential Vc is LO, the OUT signal becomes HI according to the NOR logic. When the OUT signal becomes HI, a rising edge can be generated, which increments the in-pixel counter 223.

[0124] In this way, the gating circuit 222 propagates the logic state of the cathode node potential Vc to the OUT signal in accordance with the logic state of the CHK signal. i At a timing selected from the plurality of different end timings, the state of the cathode node potential Vc of the APD 211 (the output of the APD 211) is checked.

[0125] (5) Once avalanche breakdown occurs, the cathode node potential Vc maintains that state, so the count value increases each time the CHK signal becomes LO. This allows time aperture imaging. In the example of Figure 18, the three sensitivity channels T i is set, and in the first recharge period, the first photon is incident on sensitivity channel T3. Therefore, the CHK signals of sensitivity channels T3, T2, and T1 are counted three times. In the second recharge period, the first photon is incident on sensitivity channel T1. Therefore, the CHK signal of sensitivity channel T1 is counted once.

[0126] (6) When the recharge signal becomes LO, the output value of the in-pixel counter 223 is written to the read latch circuit 224 (LO-through-HI-Latch). That is, when the recharge signal is LO, the read latch circuit 224 passes through (does not hold) the output value of the in-pixel counter 223, and when the recharge signal is HI, it latches (holds) it. The value written to the read latch circuit 224 is read out over one recharge period. In the example of FIG. 18, when the recharge signal in the second recharge period changes from LO to HI, the count value "3" in the first recharge period is written to the read latch circuit 224 and is read out during the second recharge period. Note that subsequent incident photons within one recharge period are ignored and not counted.

[0127] [Low power consumption of photoelectric conversion device 100] Fig. 19 is a circuit diagram illustrating an equivalent circuit for calculating the power consumption at the cathode node of the SPAD 211. Fig. 20 is a timing chart illustrating the charge and discharge operations in the equivalent circuit shown in Fig. 19.

[0128] In FIG. 19, the power consumption due to charging and discharging at the cathode node is expressed by the following formula (7). In formula (7), V BD is the breakdown voltage, C totis the total capacitance at the cathode node, VEX is the voltage swing (which in this configuration is the same as the supply voltage), and f is the recharge frequency.

[0129]

number

[0130] Generally, to reduce pixel size, the SPAD 211, the recharge transistor of the recharge circuit 221, and the inverter circuit 229 are fabricated on separate wafers and then bonded using hybrid bonding, a type of lamination technology. The inverter circuit 229 is used to shape the cathode potential waveform and convert it into a digital voltage. Generally, hybrid bonding requires large electrodes, which increases the parasitic capacitance. For example, the parasitic capacitance is thought to be about 5 fF / ea. on average.

[0131] Furthermore, to operate the SPAD 211 in avalanche mode with high quantum sensitivity, a high voltage such as 3 V must be supplied as the excess voltage (VEX). However, the charge / discharge power is generated by a single charge proportional to the voltage amplitude of the cathode node, in this case VEX, and this charge flows into the anode terminal to which a large negative power supply of approximately -20 V is applied, which is the breakdown voltage for a silicon substrate. In other words, the charge / discharge power due to the recharge operation is discharged. Therefore, it can be considered that the charge / discharge power due to the recharge operation is far more dominant than the power consumed by the subsequent in-pixel counter 223, which generally uses a low power supply voltage of 1 V or less. Therefore, it can be said that most of the power consumed in the SPAD pixel 200 is consumed by the charge / discharge operation of the parasitic capacitance associated with the cathode node.

[0132] In this embodiment, a time-aperture imaging algorithm is used to obtain multi-bit information in a single recharge operation. Therefore, a dynamic range and SNR equivalent to or greater than that of the conventional clock-synchronized recharge method described in Non-Patent Document 1 can be achieved with, for example, 1 / 2.5 times fewer recharges. As a result, a high-dynamic-range scene can be captured with significantly less power consumption than the conventional clock-synchronized recharge method.

[0133] [Ease of increasing the number of pixels in the SPAD pixel 200] Fig. 21 is a schematic diagram illustrating fluctuations in power consumption during imaging using the temporal aperture imaging algorithm. Fig. 22 is a schematic diagram illustrating, as a comparative example, fluctuations in power consumption during imaging using a conventional clock-synchronized recharge method.

[0134] As shown in FIG. 21 , in this embodiment, the SPAD pixel 200 can always perform a recharge operation at a constant interval regardless of the amount of incident light. Therefore, fluctuations in power consumption at the cathode node are minimized regardless of whether the amount of incident light is very strong or very weak. As a result, fluctuations in the power supply to the pixel array section 120 are suppressed. Furthermore, peak currents at the cathode node are also suppressed. This simplifies the design of the power supply wiring for the pixel array section 120, making it easy to achieve a large number of pixels.

[0135] On the other hand, in imaging using the conventional clock-synchronized recharge method, the fluctuations in power consumption in SPAD pixels are large, as shown in Figure 22. Therefore, designing the power supply wiring for the pixel array is difficult, making it difficult to achieve a high pixel count.

[0136] [SPAD pixel 200 pixel miniaturization] Fig. 23 is a schematic diagram illustrating the count number of an in-pixel counter per exposure time unit in imaging using the temporal aperture imaging algorithm. Fig. 24 is a schematic diagram illustrating, as a comparative example, the count number of an in-pixel counter per cluster in imaging using a conventional clock-synchronized recharge method.

[0137] The temporal aperture imaging algorithm allows for constant recharge operation regardless of the amount of incident light, and for very strong light, a low-sensitivity channel with a low temporal aperture ratio is responsible for detecting photons. In other words, using a sensitive channel is equivalent to capturing images with reduced light, making it possible to perform counting operations with a reduced count increase rate within one recharge period. This characteristic allows the in-pixel counter 223 to be configured with a small number of bits, making it possible to miniaturize the pixel.

[0138] As shown in Figure 23, in the temporal aperture imaging algorithm method of this embodiment, when sensitivity channels T1 to T7 are used, only 7 counts increase during one exposure time unit, so a 3-bit in-pixel counter is sufficient. Since the size of the SPAD pixel 200 is determined by the size required for the in-pixel counter 223 rather than the size of the SPAD 211, pixel miniaturization can be easily achieved using a temporal aperture imaging algorithm that can perform imaging with a small number of bits. Furthermore, if three sensitivity channels are sufficient, a 2-bit counter can be used, making it possible to realize even smaller pixels.

[0139] On the other hand, as shown in Figure 24, with the conventional clock-synchronized recharge method, the shortest recharge cycle is approximately 25 ns, and it is estimated that this is executed in bursts of approximately 40 times. Also, in one cluster period consisting of multiple types of recharge cycles, it is estimated that the maximum count is approximately 125. Therefore, in order to accurately count incident photons, the in-pixel counter needs to have a large bit count, such as 8 bits.

[0140] Therefore, in the temporal aperture imaging algorithm method of this embodiment, it is possible to configure a pixel front-end circuit with a reduced number of bits in the in-pixel counter compared to the conventional method.

[0141] [High frame rate feasibility] In this method, the dynamic range expansion rate is determined only by the temporal aperture ratio, so unlike conventional methods, binary and low-bit images can be output at high frame rates regardless of the degree of dynamic range expansion.

[0142] The photoelectric conversion device 100 of the present embodiment described above can suppress increases and fluctuations in pixel power consumption, and can also suppress increases in the circuit scale of the pixel 200. Furthermore, the photoelectric conversion device 100 can output low-bit images at a high frame rate, regardless of the degree to which the dynamic range for illuminance is expanded.

[0143] (Second embodiment) In the second embodiment, a case where the hardware of the pixel 200 is configured with a three-layer substrate is illustrated. Fig. 25 is a circuit diagram illustrating a case where the hardware of the pixel in the photoelectric conversion device according to the second embodiment is configured with three layers. Note that to avoid duplication, detailed description of the same configuration as in the first embodiment will be omitted.

[0144] [Example of a three-layer structure] The configuration of the first and second layers is the same as that of the two-layer configuration in the first embodiment, except for the read latch circuit 224, SEL 225, and output bus 228. An in-pixel counter 323 (upper counter unit) is mounted on the third layer (third substrate), as on the second layer. The in-pixel counter 223 on the second layer and the in-pixel counter 323 on the third layer are connected in series, and the in-pixel counter 223 and the in-pixel counter 323 work together to operate as a single large in-pixel counter. A connection element such as Nano-TSV (nTSV) can be used to connect the in-pixel counter 223 on the second layer and the in-pixel counter 323 on the third layer.

[0145] The number of bits of the in-pixel counter 223 in the second layer and the in-pixel counter 323 in the third layer do not necessarily have to be the same. iThe number of bits must be at least 1. However, CEIL represents the round-up function.

[0146] In a three-layer configuration, the rate at which the final bit changes decreases. Therefore, all pixels in the pixel array section 120 only need to be transferred over multiple recharge periods, eliminating the need for the read latch circuit 224. Instead of the read latch circuit 224, a selector is provided to read all bits last, and the BITSEL signal controls whether or not to read them.

[0147] [Example of operation of a three-layer pixel circuit] Fig. 26 is a timing chart for explaining the operation of pixel circuit 220 shown in Fig. 25. However, for ease of explanation, Fig. 26 will be explained using an example in which counter 223 in pixel 223 in the second layer in Fig. 25 is configured with 2 bits, counter 323 in pixel 323 in the third layer is configured with 1 bit, and the number of sensitivity channels is three.

[0148] (1) When the second layer intra-pixel counter 223 reaches the full code, a carry is propagated to the third layer intra-pixel counter 323, and the third layer intra-pixel counter 323 is incremented.

[0149] (2) The shortest time for the counter 223 in the pixel in the second layer to reach full code is after one recharge time. Therefore, if the most significant bit of the counter 323 in the pixel in the third layer is read from the pixel 200 before that time, it is possible to accurately count outside the pixel 200 without missing any increment events carried over from the lower bits.

[0150] (3) After all the accumulation times have finished, the BITSEL signal is used to sequentially read out the lower-order bit information remaining in the in-pixel counter 223. This allows the same final count value to be obtained without having to read out all the bits every recharge time.

[0151] (Third embodiment) In the third embodiment, a case will be described in which each pixel 200 in a pixel array section 120 has a light receiving section 210 and a pixel circuit 220, and the photoelectric conversion device 100 has a data storage section 140 as a peripheral circuit of the pixel array section 120. Note that to avoid duplication, detailed description of the same configuration as in the first embodiment will be omitted.

[0152] [Example of pixel circuit and data storage section in a two-layer structure] 27 is a circuit diagram illustrating pixels 200 and a data storage unit 140 in a two-layer configuration in the third embodiment. In this embodiment, the pixel array unit 120 has a data storage unit 140 in addition to a plurality of pixels 200. The pixel 200 has a light receiving unit 210 and a pixel circuit 220. The configurations of the light receiving unit 210 and the pixel circuit 220 are the same as those in the first embodiment, so detailed description thereof will be omitted.

[0153] The data accumulation unit 140 is disposed outside the pixel array unit 120 and includes a column processor 241 and a frame memory 242. The column processor 241 integrates (adds) pixel data read from the read latch circuit 224 or the in-pixel counter 223 over multiple unit exposure times, and writes the integrated value (sum) to the frame memory 242. The column processor 241 is, for example, an adder circuit, and one column is provided for each column. The frame memory 242 is a memory that holds the integrated value by the column processor 241, and can be, for example, an SRAM (Static Random Access Memory). For example, when simply integrating pixel data, the column processor 241 adds data read from a corresponding address in the frame memory 242 to pixel data output from the pixel circuit 220, and writes the sum to the corresponding address in the frame memory 242. In this way, by providing the frame memory 242 around the pixel array section 120 and storing data digitally therein, the number of counter bits within the pixel can be reduced, which in turn makes it easier to miniaturize the pixel.

[0154] [Layer structure of pixel 200] FIG. 28 is a schematic block diagram illustrating an example of the layer configuration of a pixel 200. When the substrate has a two-layer configuration (left diagram), the second layer is provided with a pixel counter array 244 including pixel circuits 220 and sensitivity determination units 230, a column processor 241, and a frame memory 242. A plurality of pixel circuits 220 are arranged in an array in the pixel counter array 244. The column processor 241 and the frame memory 242 can be divided into multiple pairs in the same layer. FIG. 28 illustrates a case where pairs of the column processor 241 and the frame memory 242 are arranged one on each side of the pixel counter array 244. In the example shown in FIG. 28, the column processor 241 and the frame memory 242 are divided and arranged in the vertical direction of the figure, but they may also be divided and arranged in the horizontal direction. Furthermore, the number of pairs (divisions) of the column processor 241 and the frame memory 242 is not limited to two and may be four or more.

[0155] Furthermore, when the substrate has a three-layer structure (right diagram), a pixel counter array 244 may be arranged on the second layer, and multiple pairs of column processors 241 and frame memories 242 may be arranged directly below the pixel counter array 244 on the third layer. FIG. 28 illustrates an example in which four pairs of column processors 241 and frame memories 242 are arranged. The number of pairs (number of divisions) of column processors 241 and frame memories 242 is not limited to four and may be eight or more. The parasitic capacitance and resistance of the readout wiring decrease according to the number of divisions, thereby enabling high-speed readout operations.

[0156] (Fourth embodiment) In the fourth embodiment, a case where imaging is performed using only the high sensitivity channel as needed will be described. Note that to avoid duplication, detailed description of the same configuration as in the first embodiment will be omitted.

[0157] [Modification of gating circuit] FIG. 29 is a circuit diagram showing a modified example of a gating circuit in the fourth embodiment. FIG. 29 illustrates a case where a two-stage gating circuit having one logic element in each stage is used. Gating is performed using one gating signal in each stage of the gating circuit, i.e., a total of two gating signals in the two stages. In this embodiment, the gating signal in the first stage is a CHK signal, and the gating signal in the second stage is an EN_LS (Low Sensitivity) signal. This allows the in-pixel counter to count up using only the high-sensitivity channel (hereinafter also referred to as the "high-sensitivity channel") T1. While FIG. 29 illustrates a case where a NOR element 251 is used in the first stage and an AND element 252 in the second stage, other combinations of logic elements may be used as long as they can achieve the same logical operation.

[0158] 29, the cathode node potential Vc is input to one input terminal of the NOR element 251, and the CHK signal is input to the other input terminal. In addition, the output signal (OUT signal) of the NOR element is input to one input terminal of the AND element 252, and the EN_LS signal is input to the other input of the AND element 252.

[0159] The CHK signal is transmitted to each sensitivity channel T i When the CHK signal is enabled, the OUT signal outputs the same logic state as the cathode.

[0160] When the EN_LS signal is LO, the logical state of the OUT signal is not transmitted to OUT2 by the AND element 252. On the other hand, when the EN_LS signal is HI, the logical state of the OUT signal is transmitted to the OUT2 signal. The EN_LS signal is a signal for checking and controlling whether or not to count the low sensitivity channels (T2, T3), and is a control signal that does not count the low sensitivity channels (T2, T3) when LO, and counts the low sensitivity channels (T2, T3) when HI.

[0161] That is, depending on the logic level of the EN_LS signal, it is possible to control whether to check only the high-sensitivity channel T1 or to check the low-sensitivity channels (T2, T3) as well. Therefore, in scenes where a high dynamic range image is not required, the number of increments of the in-pixel counter 223 can be reduced by omitting checking the low-sensitivity channels (T2, T3). This reduces the power consumption of the in-pixel counter 223.

[0162] Note that EN_LS does not need to be common to all pixels 200 in the pixel array unit 120. For example, the pixel array unit 120 may be divided into a plurality of pixel regions, and a different EN_LS signal may be supplied to each of the divided pixel regions. In the following description of this embodiment, the high-sensitivity channel is designated as T1, and all other sensitivity channels are designated as low-sensitivity channels.

[0163] [Operations of the recharge circuit 221 and the gating circuit 250] FIG. 30 is a timing chart illustrating the operation of the recharge circuit 221 and the gating circuit 250 shown in FIG.

[0164] (1) When the CHK signal is LO and the cathode potential Vc is LO, the logic state of the OUT signal is HI.

[0165] (2) As shown in the dotted circle in Figure 30, by setting EN_LS to HI only when checking sensitivity channel T1, the logical state of the OUT2 signal and the logical state of the OUT signal become the same, which can be used as an event to increment the in-pixel counter 223 in the subsequent stage.

[0166] (3) On the other hand, at the timing when the sensitivity channels T2 and T3 are checked, EN_LS is LO, and the logic state of the OUT signal is not transmitted to the OUT2 signal, so the in-pixel counter 223 does not count up.

[0167] (Fifth embodiment) In the fifth embodiment, a method for detecting photons using a low-sensitivity channel in a low-illuminance state will be described. In order to avoid duplication, detailed descriptions of the same configuration as in the second embodiment will be omitted.

[0168] Fig. 31 is a schematic diagram for explaining the detection of photons by a low-sensitivity channel under low illuminance conditions, and Fig. 31 illustrates an example in which three channels (T1 to T3) are set.

[0169] Photon detection is accompanied by optical shot noise, which cannot be eliminated in principle. The standard deviation of optical shot noise is calculated by the square root of the optical signal. In other words, the higher the illumination intensity, the higher the signal-to-noise ratio (SNR), but the more optical shot noise is generated. In other words, when comparing artifacts generated in the time aperture imaging algorithm with optical shot noise, if the optical shot noise is sufficiently large, it can be considered indistinguishable from the artifacts generated in the time aperture imaging algorithm. In other words, when comparing artifacts with optical shot noise, if the optical shot noise is sufficiently large, the artifacts can be ignored.

[0170] Taking into consideration the nature of such artifacts and optical shot noise, this embodiment can be configured to detect photons in low illumination conditions using, for example, one of the following two methods.

[0171] Low-light SNR improvement method 1: Count photons only using the high-sensitivity channel T1 until the optical shot noise becomes large enough, and then count photons using the low-sensitivity channels T2 and T3 as well.

[0172] Low-illuminance SNR improvement method 2: The count value acquired by the high-sensitivity channel T1 and the count values ​​acquired by the low-sensitivity channels T2 and T3 are independently integrated and stored in frame memory 242, and two transfer functions corresponding to the high-sensitivity and low-sensitivity channels are connected together. For example, the two transfer functions are connected at the coordinate point of the incident illuminance where the signal-to-noise ratio (SNR) is a value (threshold) sufficiently higher than 30 dB. That is, the count value of the high-sensitivity channel T1 is used at illuminances where the SNR is 30 dB or less, and the count values ​​of the low-sensitivity channels T2 and T3 are used at illuminances where the SNR is higher than 30 dB.

[0173] In the extremely low illuminance range, the probability that a photon will be counted by the low-sensitivity channel is very low, but the probability is not zero, so there is a very rare possibility that a pixel (bright spot) that is slightly brighter than the surroundings may occur. In such cases, the absolute value of the luminance value of each pixel is small, so it does not pose a major problem in terms of image quality when the image is viewed as a whole, but using the above method can improve image quality.

[0174] [Example of a SPAD pixel 200 configuration that enables a low sensitivity channel] FIG. 32 illustrates a specific example of the low-illumination SNR improvement method 1 described above, and is a circuit diagram illustrating a SPAD pixel 200 in a three-layer configuration that enables a low-sensitivity channel. In this embodiment, exposure is initiated by enabling only the high-sensitivity channel (the low-sensitivity channel is disabled). After a certain period of time, if high illuminance is determined based on a determination condition described below, the low-sensitivity channel is enabled and photon counting continues using both the high-sensitivity channel and the low-sensitivity channel. The certain period of time may be, for example, the period until the count value of the counter 323 in the pixel in the third layer reaches a predetermined threshold. FIG. 32 illustrates, as an example, a case where the certain period of time is the period until all of the upper bits of the counter 323 in the pixel in the third layer become HI. FIG. 33 is a timing chart illustrating the operation of the pixel circuit 220 of the SPAD pixel 200 shown in FIG. 32.

[0175] The pixel circuit 220 has a recharge circuit 221, a first gating circuit 260, an in-pixel counter 223, a lock flag circuit 226, and a second gating circuit 227 in the second layer. The first gating circuit 260 has an AND element and a NOR element, receives the cathode node potential Vc, a CHK_HS signal, and an OUT2 signal, and outputs an OUT signal to the count enable of the in-pixel counter 223. The CHK_HS signal is a signal for checking the high-sensitivity channel and is an active LO signal. The lock flag circuit 226 generates a Lock signal for locking the low-sensitivity channel. The second gating circuit 227 has a NAND element, receives the Lock signal and the CHK_LS signal, and outputs an OUT2 signal. The CHK_LS signal is a signal for checking the low-sensitivity channel and is an active HI signal.

[0176] Furthermore, the pixel circuit 220 has, in the third layer, an in-pixel counter 323, a threshold value determination circuit 324, and a SEL 325. The threshold value determination circuit 324 sets an OV (overflow) signal to HI when the value of the in-pixel counter 323 reaches a predetermined threshold value.

[0177] 33, in the threshold determination circuit 324, when all of the upper bits of the in-pixel counter 323 in the third layer become HI (a state in which the illuminance has shifted from low to high), the OV signal becomes HI. The FBK signal, which becomes active at the same time as the recharge signal, writes the logical state of the OV signal into the lock flag (DFF) of the lock flag circuit 226 arranged in the second layer, and outputs it as a Lock signal. When the Lock signal is LO, it means that the low-sensitivity channel is locked, i.e., the low-sensitivity channel is disabled, and when it is HI, it means that the low-sensitivity channel is unlocked, i.e., the low-sensitivity channel is enabled.

[0178] When the lock is released, the gating operation of the second gating circuit 227 is enabled by the HI active signal of CHK_LS. The lock flag is first initialized to the LO level by RSTB and updated every recharge period by the recharge signal. After that, the gating operation of the first gating circuit 260 counts using all sensitivity channels from high to low sensitivity.

[0179] When the count value of the in-pixel counter 323 reaches a predetermined threshold value (in the example of FIG. 32, all upper bits are HI), the OV signal becomes HI. This signal is propagated from the third layer to the second layer by the FBK signal and is periodically captured in the lock flag. Once the lock flag is set to HI and released, it always maintains the unlocked state due to a feedback loop using an OR element.

[0180] The lock flags are initially set to the locked state by the RSTB signal for all sensitivity channels other than T1, i.e., T2 and T3. The HLD signal is used to capture and hold the previous output from the second layer using a latch circuit to prevent erroneous operation due to discontinuity in the input state when feeding back the OV signal from the third layer. Note that while the example shown in Figure 33 illustrates a three-layer configuration, it can also be applied to two-layer and other layer configurations.

[0181] In this way, the lock flag circuit 226 and the threshold determination circuit 324 are arranged for each pixel 200, and the sensitivity channel T i The sensitivity determination unit 230 functions as a sensitivity determination unit 230 that determines (selects) a low-sensitivity channel for each pixel 200. The sensitivity determination unit 230 disables or enables a predetermined low-sensitivity channel. The sensitivity determination unit 230 continues to enable or disable the low-sensitivity channel for each pixel 200 according to the count state of the intra-pixel counter 323. Furthermore, the configuration of the threshold determination circuit 324 is not limited to a binary configuration of whether or not an overflow occurs, but may be multi-valued as an application example, so that multiple low-sensitivity channels can be enabled in multiple stages.

[0182] [Modification of the first gating circuit] FIG. 34 is a circuit diagram illustrating a first gating circuit using a logic circuit different from that of the example of FIG. 32. In the example shown in FIG. 34, CHK_LS is an LO active signal, unlike the example of FIG. 32. The first gating circuit 260 includes a first NOR element 261, a second NOR element 262, and an OR element 263. The first NOR element 261 receives the cathode node potential Vc and CHK_LS and outputs the OUT_LS signal. The second NOR element 262 receives the cathode node potential Vc and CHK_HS and outputs the OUT_HS signal. The OR element 263 receives the OUT_LS and OUT_HS signals and outputs the OUT signal. This configuration of the present modification allows the number of stages in the circuit configurations for the CHK signals of the high-sensitivity and low-sensitivity channels to be the same, thereby aligning the signal delays due to the circuit configuration. This enables the generation of timing that achieves a highly accurate time aperture ratio.

[0183] [Example of pixel circuit configuration that adaptively determines the use of low sensitivity channels] FIG. 35 shows another specific means of the above-mentioned low-illumination SNR improvement method 1, and is a circuit diagram illustrating a SPAD pixel 200 having a function of adaptively determining whether or not to use a low-sensitivity channel in a two-layer configuration.

[0184] The second layer pixel circuit 220 includes a recharge circuit 221, a first gating circuit 260, an in-pixel counter 223, an adaptive determination circuit 270, a lock flag circuit 226, a second gating circuit 227, etc. The configurations of the recharge circuit 221, the first gating circuit 260, and the in-pixel counter 223 are as described above, so description thereof will be omitted.

[0185] The adaptive decision circuit 270 adaptively decides whether to use a low-sensitivity channel based on the tendency of the incident photon intervals. The adaptive decision circuit 270 includes a threshold decision circuit 271, a state memory 272, and a circuit 273 associated with the state memory.

[0186] The threshold determination circuit 271 determines whether the count value of the in-pixel counter 223 matches a predetermined threshold value. The status memory 272 is a memory for storing the state of the rate at which photons are incident on the pixel 200 after exposure has started (also simply referred to as the "incident rate"), that is, the state of the photon counting rate by the in-pixel counter 223. The number of bits of the status memory 272 is set to an appropriate required number of bits.

[0187] The adaptive decision circuit 270 determines the photon incidence rate from the start of exposure. More specifically, if the photon incidence rate is equal to or greater than a certain rate (high photon rate), the adaptive decision circuit 270 determines that the light irradiating the pixel 200 is high illuminance and immediately enables the low-sensitivity channel. In this case, optical shot noise becomes dominant, so artifacts caused by the time aperture imaging algorithm are not a problem, and a good high-illuminance image can be obtained.

[0188] On the other hand, if the adaptive decision circuit 270 determines that the incident rate of photons is low (low photon rate), it determines that the light irradiating the pixel 200 is low illuminance, and enables only the high sensitivity channel without enabling the low sensitivity channel. This prevents artifacts caused by the time aperture imaging algorithm, and allows for a good low illuminance image to be obtained.

[0189] The adaptive decision circuit 270 can make adaptive decisions at any time during the exposure period, but is most effective when made within a short period immediately after the start of exposure. The adaptive decision circuit 270 can be enabled by the EN_ADP signal.

[0190] In this way, the adaptive decision circuit 270 determines the rate at which photons are incident and, based on the result of the determination, estimates the amount of light incident on the pixel 200. Because the adaptive decision circuit 270 makes a determination based on the counting rate of incident photons, it is not limited to the maximum value that can be expressed by the number of bits of the in-pixel counter 223. Furthermore, because a necessary sensitivity channel is selected for each pixel 200, implementation is possible even when the number of bits of the in-pixel counter 223 is small.

[0191] The lock flag circuit 226 and the adaptive decision circuit 270 are arranged for each pixel 200 and correspond to the sensitivity channel T i The sensitivity determination unit 230 functions as a sensitivity determination unit 230 that determines (selects) for each pixel 200. The sensitivity determination unit 230 disables or enables a predetermined low-sensitivity channel. The sensitivity determination unit 230 calculates the photon incidence rate from the count value of the in-pixel counter 223, and continues to enable or disable the low-sensitivity channel based on the calculation result.

[0192] <Example of determining the amount of incident light based on the rate of incident photons> FIG. 36 is a timing chart for explaining a method for determining the amount of incident light based on the rate of incident photons.

[0193] When high-intensity light is incident on a pixel 200 in the pixel array unit 120, the time interval between photons arriving at the pixel 200 is short. In other words, photons are incident on the pixel 200 without interruption (almost continuously). In this case, the final count value of the in-pixel counter 223 is predicted to be large. The adaptive decision circuit 270 determines that a high photon rate exists when photons are incident on the pixel without interruption. The adaptive decision circuit 270 can determine that a high photon rate exists when a condition for a high photon rate is met. For example, a condition for a high photon rate is when the count value of the in-pixel counter 223 is 4 or greater three times in a row, as a state in which photons are incident on the pixel without interruption. Figure 36 shows that the values ​​of the in-pixel counter for the past three consecutive recharges are 6, 7, and 7. If the adaptive decision circuit 270 determines that a high photon rate exists, it sets the HRT signal to a HI level to enable the low-sensitivity channel. The HRT signal indicates a high photon rate.

[0194] On the other hand, when low-intensity light is incident on the pixel 200, the interval between photon arrival times is long. That is, photons are incident on the pixel 200 intermittently. In this case, the final count value of the in-pixel counter 223 is expected to be very small. The adaptive decision circuit 270 may determine a low photon rate when the condition for a high photon rate is not met. The adaptive decision circuit 270 may also be configured to determine a low photon rate when photons are incident on the pixel 200 intermittently. The adaptive decision circuit 270 may determine a low photon rate when, for example, the count value of the in-pixel counter 223 becomes approximately 0 or 1 during one recharge period, which is a state in which photons are incident on the pixel 200 intermittently. If the adaptive decision circuit 270 determines a low photon rate, it sets the HRT signal to the LO level to disable the low-sensitivity channel. Furthermore, instead of being limited to two stages of high photon rate and low photon rate, an adaptive decision circuit for multiple stages of photon rate may be provided as an application example, and multiple low sensitivity channels may be enabled in multiple stages.

[0195] <Modification of Threshold Determination Circuit in SPAD Pixel 200 for Enabling Low Sensitivity Channel> 37 is a circuit diagram illustrating another specific example of the low-illumination SNR improvement method 1, showing a modified example of the threshold determination circuit in the SPAD pixel 200 that enables the low-sensitivity channel. The data storage unit 240 includes a column processor 241, a frame memory 242, and a threshold determination circuit 243.

[0196] To perform threshold determination using the count value of the in-pixel counter 223, the in-pixel counter 223 needs to have a certain number of bits, which may increase the pixel size of the SPAD pixel 200. As described above, the pixel data output from the pixel circuit 220 of the SPAD pixel 200 is integrated by the column processor 241 arranged immediately before the frame memory 242 outside the pixel array unit 120, and stored in the frame memory 242. Figure 37 illustrates an example of a threshold determination circuit 243 that determines whether the integration result by the column processor 241 is equal to or greater than a threshold value, for example, 256 counts. If the integration result by the column processor 241 is equal to or greater than a threshold value, for example, 256 counts, the lock is released by clearing the lock flag of the lock flag circuit 226 via the feedback bus 291.

[0197] In this way, by providing the column processor 241 and the threshold value determination circuit 243 outside the pixel array unit 120, it is possible to use an arbitrarily large threshold value to be compared with the integrated value of the pixel data.

[0198] The lock flag circuit 226 and the data storage unit 240 are connected to the sensitivity channel T i The sensitivity determination unit 223 functions as a sensitivity determination unit that determines (selects) for each pixel 200. The sensitivity determination unit continues to enable or disable a predetermined low-sensitivity channel. The sensitivity determination unit enables or disables a predetermined low-sensitivity channel based on a sum obtained by adding up the count values ​​of the in-pixel counter 223 over multiple unit exposure times.

[0199] <Separation of high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data> Fig. 38 shows a specific example of the low-illumination SNR improvement method 2, and is a block diagram illustrating a configuration for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data. Fig. 39 is a schematic diagram illustrating a method for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data. Fig. 40 is a schematic diagram illustrating the relationship between pixel values ​​and high-sensitivity count values ​​and low-sensitivity count values.

[0200] The count value from the high-sensitivity channel T1 is referred to as the high-sensitivity count value, and the count values ​​from the low-sensitivity channels T2 and T3 are referred to as the low-sensitivity count value. As shown in Fig. 38, the photoelectric conversion device 100 has high-sensitivity and low-sensitivity frame memories 281 and 282, first and second converters 283 and 284, and first and second column processors 285 and 286 in a data storage unit 280. The first and second converters 283 and 284 correspond to the high-sensitivity and low-sensitivity frame memories 281 and 282, respectively.

[0201] As described above, the pixel circuit 220 outputs pixel data that combines the count value of photons detected through the high-sensitivity channel T1 and the count value of photons detected through the low-sensitivity channels T2 and T3 using a time-aperture imaging algorithm. For each of the high-sensitivity and low-sensitivity converters, the image data values ​​(pixel values) are converted into high-sensitivity count values ​​and low-sensitivity count values ​​and output the converted values. The first and second column processors 285 and 286 integrate the outputs of the first and second converters 283 and 284, respectively, and store the integrated values ​​in the high-sensitivity and low-sensitivity frame memories 281 and 282, respectively. In other words, the converters expand the pixel data that has been compressed within the pixel.

[0202] For example, assume that the count value of the 3-bit in-pixel counter 223 is output from the pixel circuit 220. As shown in Fig. 39, the high-sensitivity count value and the low-sensitivity count value can be calculated as follows.

[0203] [High sensitivity count value] When the count value of the intra-pixel counter 223 is 1 or greater, the first converter 283 outputs the high-sensitivity count value as 1 because the high-sensitivity channel T1 is always being counted. The first column processor 285 adds the high-sensitivity count value of 1 to the value read from the high-sensitivity frame memory 281 and writes the result into the high-sensitivity frame memory 281. That is, the value of the high-sensitivity frame memory 281 is incremented. On the other hand, when the count value of the intra-pixel counter 223=0, the first converter 283 outputs the high-sensitivity count value as 0. The first column processor 285 adds the high-sensitivity count value of 0 to the value read from the high-sensitivity frame memory 281 and writes the result into the high-sensitivity frame memory 281 (that is, it does not perform any operation on the sensitivity frame memory 281).

[0204] [Low sensitivity count value] When the count value of the intra-pixel counter 223 is 1 or greater, the second converter 284 decrements the count value of the intra-pixel counter 223 by 1 (the value minus the high-sensitivity channel T1) as the low-sensitivity count value. The second column processor 286 adds the count value of the intra-pixel counter 223 decremented by 1 to the value read from the low-sensitivity frame memory 282 and writes the result to the low-sensitivity frame memory 282. On the other hand, when the count value of the intra-pixel counter 223=0, the second converter 284 outputs the low-sensitivity count value as 0. The second column processor 286 adds the low-sensitivity count value of 0 to the value read from the low-sensitivity frame memory 282 and writes the result to the low-sensitivity frame memory 282 (i.e., it does not perform any operation on the low-sensitivity frame memory 282).

[0205] In this way, by separating the high-sensitivity count value and the low-sensitivity count value from the pixel data and connecting the transfer functions obtained from each at an arbitrary threshold, it becomes easy to realize a composite transfer function that is not affected by artifacts caused by the time aperture imaging algorithm.In addition, other image signal processing such as linearization can be efficiently realized.

[0206] As described above, the photoelectric conversion device of the present invention has been described in the embodiments. However, it goes without saying that those skilled in the art can appropriately add, modify, and omit parts to the present invention within the scope of the technical concept thereof.

[0207] For example, a configuration in which some of the configurations in one of the first to fifth embodiments described above are added to the configuration in another embodiment, or a configuration in which some of the configurations in one embodiment are deleted may be adopted.

[0208] Furthermore, in the first to fifth embodiments described above, a portion of the components arranged inside the pixel 200 may be arranged outside the pixel 200, or a portion of the components arranged outside the pixel 200 may be arranged inside the pixel 200. [Explanation of symbols]

[0209] 100 Photoelectric conversion device, 110 control section, 120 pixel array section, 130 signal processing section, 140 Data Storage Unit 200 pixels (SPAD pixels), 210 Light receiving section, 211 APD(SPAD), 220 pixel circuits, 210 Light receiving section, 221 recharge circuit, 222 gating circuit, 223 In-pixel counter, 224 read latch circuit, 225 selection circuit (SEL), 226 lock flag circuit, 227 second gating circuit; 228 output buses, 229 inverter circuits, 230 sensitivity determination unit, 241 column processor, 242 frame memory, 243 Threshold judgment circuit, 244 pixel counter array 250 gating circuits, 260 first gating circuit, 270 adaptive decision circuit, 280 Data storage unit, 281 Frame memory for high sensitivity, 282 Frame memory for low sensitivity, 283 first converter, 284 second converter, 285 first column processor, 286 second column processor, 323 In-pixel counter, 324 Threshold judgment circuit 325 SEL.

Claims

1. A photoelectric conversion device having an avalanche photodiode including an anode and a cathode in a pixel, a recharge unit that recharges the anode or the cathode once per unit exposure time; a gating unit that generates pulse signals based on the output of the avalanche photodiode at a plurality of different timings within the unit exposure time; a counting unit that counts the pulse signals from the gating unit.

2. 2. The photoelectric conversion device according to claim 1, wherein the gating section receives the output signal of the avalanche photodiode and the plurality of determination signals at different timings to generate a pulse signal.

3. 3. The photoelectric conversion device according to claim 1, further comprising a normalization processing unit that adds up the count values ​​of the counting unit over a plurality of the unit exposure times and normalizes the sum to calculate the normalized value as the number of photons that contributed to the occurrence of avalanche action in a predetermined frame.

4. The photoelectric conversion device according to claim 3 , further comprising a linearization processing unit that performs linearization processing on a nonlinear region range of the calculated number of photons.

5. The photoelectric conversion device according to claim 1 or 2, further comprising an arithmetic processing unit that adds up the count values ​​of the counting unit over a plurality of the unit exposure times to generate a first intermediate image, and generates a second intermediate image by performing motion correction processing on the generated first intermediate image.

6. 3. The photoelectric conversion device according to claim 1, wherein the unit exposure time has a plurality of virtual sensitivity channels having different time lengths, and the plurality of virtual sensitivity channels are configured such that one virtual sensitivity channel encompasses other virtual sensitivity channels on the time axis.

7. The photoelectric conversion device according to claim 6 , wherein the different timings are terminals of each of the virtual sensitivity channels.

8. The photoelectric conversion device according to claim 6 , wherein the plurality of virtual sensitivity channels are started at the same timing.

9. 3. The photoelectric conversion device according to claim 1, further comprising a latch unit in each pixel, the latch unit holding the count value of the count unit.

10. A first substrate and a second substrate stacked on the first substrate, 3. The photoelectric conversion device according to claim 1, wherein the pixel having the avalanche photodiode is provided on the first substrate, and the recharge section, the gating section, and the count section are provided on the second substrate.

11. further comprising a third substrate stacked on the second substrate; The photoelectric conversion device according to claim 10 , further comprising a higher-order counter unit connected in series to the count unit on the third substrate.

12. 3. The photoelectric conversion device according to claim 1, further comprising a frame memory for storing an added value obtained by adding up the count values ​​of said counting unit over a plurality of unit exposure times.

13. 3. The photoelectric conversion device according to claim 1, wherein the gating section generates a pulse signal based on the output of the avalanche photodiode only at at least one timing selected from the plurality of different timings.

14. 3. The photoelectric conversion device according to claim 1, wherein the gating unit generates a pulse signal based on the output of the avalanche photodiode at a timing corresponding to an end of a virtual sensitivity channel having the highest sensitivity among the plurality of different timings.

15. 3. The photoelectric conversion device according to claim 1, further comprising a sensitivity determination unit that enables or disables a predetermined low-sensitivity virtual sensitivity channel for each pixel.

16. a higher-order counter unit connected in series with the count unit; The photoelectric conversion device according to claim 15 , wherein the sensitivity determining section enables or disables the predetermined low-sensitivity virtual sensitivity channel based on the count state of the upper counter section.

17. The photoelectric conversion device according to claim 15 , wherein the sensitivity determination unit determines an incident rate of photons from the count value of the count unit, and enables or disables the predetermined low-sensitivity virtual sensitivity channel based on a determination result.

18. The photoelectric conversion device according to claim 15 , wherein the sensitivity determination unit enables or disables the predetermined low-sensitivity virtual sensitivity channel based on a sum obtained by adding up count values ​​of the count unit over a plurality of unit exposure times.

19. The photoelectric conversion device according to claim 12, wherein the frame memory has a high-sensitivity frame memory that holds count values ​​from a predetermined high-sensitivity virtual sensitivity channel, and a low-sensitivity frame memory that holds count values ​​from a predetermined low-sensitivity virtual sensitivity channel.

20. 20. The photoelectric conversion device according to claim 19, wherein the predetermined high-sensitivity virtual sensitivity channel is a virtual sensitivity channel with the highest sensitivity.

21. 20. The photoelectric conversion device according to claim 19, further comprising a first converter for converting pixel data output from each pixel into a high-sensitivity count value, and a second converter for converting pixel data into a low-sensitivity count value.

Images