Photoelectric conversion device

Abstract

The present disclosure aims to provide a photoelectric conversion device capable of suppressing an increase in power consumption of SPAD pixels. For example, a photoelectric conversion device may include an avalanche photodiode in each pixel of a pixel array, the avalanche photodiode including an anode and a cathode. The apparatus may include: a charger configured to charge an anode or a cathode once per unit exposure time; a gate circuit configured to generate a pulse signal based on an output of the avalanche photodiode at a plurality of different times within a unit exposure time; and a counter configured to count the pulse signal from the gate circuit.

Description

[0001] This application claims priority and benefit to Japanese Patent Application No. 2024-088692, filed on May 31, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to photoelectric conversion equipment (or photoelectric conversion device). Background Technology

[0003] Recently, photoelectric conversion devices (image sensors) using single-photon avalanche photodiodes (SPADs) have attracted attention. In an avalanche photodiode (hereinafter also referred to as "APD"), the SPAD utilizes avalanche amplification, in which a large number of charge carriers are generated when a single photon is incident on a PN junction subjected to a reverse bias voltage exceeding the breakdown voltage. Photoelectric conversion devices using SPADs have an APD in each of a plurality of pixels arranged along the row and column directions. Photoelectric conversion devices using SPADs acquire images by counting pulses generated based on a carrier signal produced by avalanche amplification in such APDs.

[0004] Photoelectric conversion devices using SPADs acquire images by counting incident single photons, which has the advantage of obtaining good images even in dark places (low illumination). Furthermore, conventional photoelectric conversion devices using SPADs can lead to increased circuit size and power consumption due to the increased counting in bright locations (high illumination).

[0005] In this regard, for example, Non-Patent Document 1 discloses a clock-synchronized charging type photoelectric conversion device designed to suppress the increase in circuit size and power consumption when shooting in bright environments. Generally, due to the interference-free and memoryless nature of photons, the occurrence of photons per unit time in artificial or natural light that has undergone multiple reflections or diffusions (i.e., the time it takes for photons to enter the photoelectric conversion device) is uncorrelated and random. Therefore, the probability density of photon occurrence becomes constant, and the frequency of photon occurrence increases or decreases proportionally with illuminance. Furthermore, it is known that the deviation of this occurrence frequency follows a Poisson distribution, which is the root cause of so-called optical shot noise. Moreover, regardless of the intensity of illuminance, it is known that the interval between photons arriving at the photoelectric conversion device follows an exponential distribution. In the photoelectric conversion device of Non-Patent Document 1, time divisions assuming low, medium, and high illuminance are sequentially set within one frame, and in order to extend the dynamic range on the medium and high illuminance sides, the interval of charging operations is configured to shorten as illuminance increases. The charging operation is the process of supplying the APD with the overvoltage required to operate the APD as a SPAD.

[0006] Non-Patent Literature 1: T. Takatsuka et al., “A 3.36μm-pitch SPAD photon-counting image sensor using clustered multi-cycle clocked recharging technique with intermediate most-significant-bit readout”, 2023 IEEE Symposium on VLSI Technology and Circuits, C15-2, Kyoto, 2023. Summary of the Invention

[0007] However, in the photoelectric conversion device of Non-Patent Document 1, the problem of increased power consumption still exists because more charging operations are required during periods of high illumination. Furthermore, during periods of high illumination, the charging operation cycle becomes very short, resulting in large power consumption fluctuations and difficulty in scaling to multiple pixels. Additionally, during periods of high illumination, a large number of counts occur within a very short time period, so the number of bits in the counter within the pixel usually needs to be increased to ensure accurate counting. Therefore, it is difficult to miniaturize the pixels. Moreover, in methods that continuously scan along the time axis with different sensitivities, a strong trade-off exists between expanding the dynamic range and the frame rate, making it difficult to output low-bit images at high frame rates. Therefore, it becomes difficult to apply image processing techniques to improve image quality, such as for moving images.

[0008] This disclosure has been made to address the aforementioned problems. Therefore, a key objective of this disclosure is to provide a photoelectric conversion device capable of suppressing the increase in pixel power consumption.

[0009] In addition, another object of this disclosure is to provide a photoelectric conversion device capable of suppressing fluctuations in the power consumption of pixels.

[0010] In addition, another object of this disclosure is to provide a photoelectric conversion device capable of suppressing the increase in the circuit size of pixels.

[0011] Furthermore, another object of this disclosure is to provide a photoelectric conversion device capable of outputting binary low-bit images at a high frame rate without depending on the extent to which the dynamic range is extended by illumination.

[0012] According to some embodiments, a photoelectric conversion device may include: an avalanche photodiode in each pixel of a pixel array, the avalanche photodiode including an anode and a cathode. The device may include: a charger configured to charge either the anode or the cathode once per unit exposure time; a gating circuit configured to generate pulse signals based on the output of the avalanche photodiode at multiple different times within the unit exposure time; and a counter configured to count the pulse signals from the gating circuit.

[0013] According to the photoelectric conversion device of this disclosure, it is feasible to suppress the increase and fluctuation of pixel power consumption while simultaneously suppressing the increase of pixel circuit size. Furthermore, according to the photoelectric conversion device of this disclosure, low-bit images can be output at a high frame rate regardless of the extent to which the dynamic range of illumination is extended. Attached Figure Description

[0014] Figure 1 A schematic block diagram illustrating an example configuration of a photoelectric conversion device according to a first embodiment is shown.

[0015] Figure 2 It shows the result of Figure 1 The pixel shown is a functional block diagram illustrating the example functionality provided by the example function.

[0016] Figure 3 A schematic diagram illustrating the time aperture imaging algorithm in the first embodiment is shown.

[0017] Figure 4 An example amount of information is shown regarding the number of photons per charge, depending on the sensitivity channel used.

[0018] Figure 5 A schematic diagram of an example conceptual model for incident photon detection using a time-aperture imaging algorithm that implements multiple sensitivities is shown.

[0019] Figure 6 Example count values ​​based on illuminance are shown.

[0020] Figure 7 Examples of dynamic range expansion and the number of in-pixel counter bits are shown, corresponding to various combinations of the time aperture ratio of the sensitivity channel.

[0021] Figure 8 Example graphs illustrating the sensitivity channel transfer function and the normalized aggregate transfer function are shown.

[0022] Figure 9 Example count values ​​based on illuminance and example dominant sensitivity channels for each illuminance are shown.

[0023] Figure 10 A graph showing an example of SNR and DR magnification is provided.

[0024] Figure 11 A graph showing another example of SNR Dip occurring in SNR and DR amplification is presented.

[0025] Figure 12 A schematic diagram is shown to illustrate the transformation of time aperture imaging parameters according to imaging illumination conditions.

[0026] Figure 13 An example of setting the sensitivity ratio between each sensitivity channel is shown.

[0027] Figure 14 The diagram illustrates an example of reconstructing an image by performing correction processing on pixel data.

[0028] Figure 15 The diagram illustrates an example of reconstructing an image by simply adding pixel data.

[0029] Figure 16 It shows Figure 1 The block diagram shows an example schematic configuration of the signal processor.

[0030] Figure 17 A circuit diagram is shown illustrating an example case where the pixel hardware is configured as a two-layer substrate.

[0031] Figure 18 A description is shown Figure 17 The timing diagram shows the operation of the pixel circuit.

[0032] Figure 19 A circuit diagram illustrating an example equivalent circuit for calculating power consumption at the cathode node of a SPAD is shown.

[0033] Figure 20 It shows Figure 19 Timing diagrams of example charging and discharging operations in the equivalent circuit shown.

[0034] Figure 21 A schematic diagram illustrating an example variation in power consumption during imaging using a time-aperture imaging algorithm is shown.

[0035] Figure 22 A schematic diagram is shown illustrating an example variation in power consumption during imaging using a conventional clock-synchronized charging method as a comparative example.

[0036] Figure 23 A schematic diagram is shown illustrating an example count of the counter within a pixel per exposure time unit in an image using a temporal aperture imaging algorithm.

[0037] Figure 24 A schematic diagram is shown illustrating an example count of the counter within a pixel for each cluster in an image obtained by using a conventional clock-synchronized charging method as a comparative example.

[0038] Figure 25 A circuit diagram is shown illustrating an example of a photoelectric conversion device according to the second embodiment where the pixel hardware is configured as a three-layer substrate.

[0039] Figure 26 A description is shown Figure 25 The timing diagram shows the operation of the pixel circuit.

[0040] Figure 27 A circuit diagram illustrating an example pixel and an example data accumulator configured in a two-layer manner according to a third embodiment is shown.

[0041] Figure 28 A schematic block diagram illustrating an example layer configuration for pixels is shown.

[0042] Figure 29 A circuit diagram illustrating a modified example of the gating circuit according to the fourth embodiment is shown.

[0043] Figure 30 It shows Figure 29 Timing diagram of example operation of the charging circuit and gating circuit shown.

[0044] Figure 31 A schematic diagram is shown to illustrate the detection of photons through a low-sensitivity channel under low illumination conditions according to the fifth embodiment.

[0045] Figure 32 A circuit diagram is shown illustrating an example SPAD pixel with a low-sensitivity channel enabled in a three-layer configuration.

[0046] Figure 33 It shows Figure 32 The timing diagram shows an example operation of the pixel circuitry for the SPAD pixel shown.

[0047] Figure 34 It shows the use of and Figure 32 The example logic circuits are the circuit diagrams of the first gating circuits of different logic circuits.

[0048] Figure 35 A circuit diagram is shown illustrating an example SPAD pixel configured in two layers, with a function for determining whether to adaptively use a low-sensitivity channel.

[0049] Figure 36 A timing diagram is shown to describe the determination of the amount of incident light based on the incident rate of photons.

[0050] Figure 37 A circuit diagram is shown illustrating a modified example of a threshold determination circuit that enables a low-sensitivity channel in a SPAD pixel.

[0051] Figure 38 A block diagram is shown illustrating an example configuration for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data.

[0052] Figure 39 A schematic diagram illustrating a method for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data is shown.

[0053] Figure 40 A schematic diagram illustrating an example relationship between pixel values, high-sensitivity count values, and low-sensitivity count values ​​is shown. Detailed Implementation

[0054] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same components, and for clarity and ease of description, the dimensions of each component in the drawings are shown at a different scale than in reality. Furthermore, the embodiments described below are merely exemplary, and various modifications are possible from such embodiments.

[0055] Below, the description of a component as “above” or “on top of” may include not only components that are directly above in a contact manner, but also components that are above in a non-contact manner.

[0056] Unless the context clearly indicates otherwise, a component referred to in the singular includes a plural component. Furthermore, unless specifically stated otherwise, when a component is referred to as "comprising" or "having" a component, this does not exclude other components, but rather may include other components. Therefore, throughout the specification, unless the context clearly indicates otherwise, when a component is described as "comprising" a particular element or group of elements, it should be understood that the component is formed solely by that element or group of elements, or that the element or group of elements may be combined with other elements to form the component. On the other hand, the term "composed of" indicates that the component is formed solely by one or more of the listed elements.

[0057] As can be seen, for example, in the accompanying drawings, items described herein in the singular may be provided in the plural. Therefore, unless the context otherwise indicates, a description of a single item provided in the plural should be understood to apply to the remaining multiple items. With respect to the steps constituting the method, the order is explicitly stated, or, if no contrary statement exists, the steps are performed in an appropriate order. However, the order is not necessarily limited to the order in which the steps are described. Unless otherwise limited by example or exemplary terminology, any use of such example or exemplary terminology (e.g., etc.) is intended only to illustrate a technical concept and not to limit the scope of the patent claims.

[0058] Terms such as “identical” or “equal” as used herein do not necessarily refer to exactly the same features when referring to characteristics such as orientation, layout, location, shape, size, composition, quantity, time, or other measures, but are intended to cover nearly identical features including typical variations that may occur due to conventional manufacturing processes. The term “substantially” may be used herein to emphasize this meaning.

[0059] (First embodiment) <Configuration Example of Photoelectric Conversion Equipment> Figure 1 A schematic block diagram illustrating an example configuration of a photoelectric conversion device according to the first embodiment is shown. Furthermore, Figure 2 It shows the result of Figure 1 The pixel shown is a functional block diagram illustrating the example functionality provided by the example function.

[0060] like Figure 1 As shown, the photoelectric conversion device 100 may include a controller 110, a pixel array unit 120, a data accumulator 140, and a signal processor 130.

[0061] The controller 110 can supply various signals to the pixel array unit 120 to control the pixel array unit 120. The controller 110 can generate or input various signals (such as clock signals, reset signals, and control signals). Control signals may include, for example, RSTB signals, recharge signals, SEL signals, CHK signals, BITSEL signals, EN_LS signals, CHK_LS signals, CHK_HS signals, HLD signals, FBK signals, EN_APD signals, etc. A detailed description of each of these signals will be provided later. Although not shown, the controller 110 and other controllers or processors described herein may include one or more of the following components: at least one central processing unit (CPU) configured to execute computer program instructions to perform various processes and methods; random access memory (RAM) and read-only memory (ROM) (such as, for example, RAM, ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) configured to access and store data and information and computer program instructions.

[0062] The pixel array section 120 may include a pixel array comprising a plurality (e.g., M) of pixels 200 arranged in a two-dimensional grid shape (in the row direction and in the column direction) in a planar view perpendicular to the substrate surface of the photoelectric conversion device 100. Here, M is a natural number greater than or equal to 2.

[0063] The signal processor 130 can perform predetermined image processing on pixel data from the pixel array unit 120 and output the processing results. In addition to the linearization processing of the aggregation transfer function described below, the predetermined image processing may also include, for example, demosaic processing, automatic white balance (AWB) processing, noise reduction processing, lens shading correction processing, etc.

[0064] [Overview of the 200 pixel configuration] like Figure 2 As shown, each pixel 200 may include a light receiver 210, pixel circuitry 220, and a sensitivity determiner (operation processor) 230. In this embodiment, an APD is used for the light receiving element, therefore pixel 200 is also referred to as SPAD pixel 200. The light receiver 210 may include an APD (211) (see example...) Figure 17APD 211 may be an avalanche photodiode. Pixel circuit 220 may include a charging circuit (charger) 221, a gating circuit 222, an in-pixel counter (counter) 223, a read latch circuit (latch circuit) (224), a selection circuit (hereinafter also referred to as "SEL") 225, and an output bus 228, etc. Sensitivity determiner 230 performs predetermined sensitivity determination processing on pixel data to select a sensitivity channel. Sensitivity determiner 230 may be omitted as appropriate. A detailed description of the configuration of pixel circuit 220 and sensitivity determiner 230 will be provided later.

[0065] [Unit Exposure Time] The charging circuit 221 can charge the cathode of the APD 211 once per unit exposure time (or exposure time unit) using a charging signal received from the controller 110. In other words, the unit exposure time can indicate the time period from when charging is performed until the next charging is performed. The unit exposure time can be a fixed time period and can be independent of the amount of incident light. A charging period can be referred to as an exposure time unit. That is, the entire exposure period can be a collection of multiple exposure time units. The charging signal can be a one-shot trigger signal with a period corresponding to the unit exposure time (e.g., 4 μs). In this embodiment, for example, the anode of the APD 211 can be set to a fixed potential and the cathode can be charged such that the potential difference between the anode and cathode can be changed to a predetermined voltage that enables the APD 211 to operate in Geiger mode. Specifically, a potential difference consisting of the sum of a device-fixed built-in avalanche breakdown voltage and an externally controllable overvoltage can be applied between the anode and cathode. The APD 211 operating in Geiger mode can also be referred to as SPAD 211. Then, this disclosure is not limited to the configuration where the anode is at a fixed potential; on the contrary, the APD 211 may have a configuration where the cathode is at a fixed potential and the anode is charged.

[0066] The gating circuit 222 can generate pulse signals based on the output signal of APD 211 at the times of multiple different CHK signals within a unit exposure time. The generation time of the pulse signals and the specific hardware configuration of the gating circuit 222 will be described later.

[0067] The in-pixel counter 223 can detect and count photons incident on pixel 200 that contribute to the avalanche operation by counting pulse signals from gating circuit 222, and not all photons incident on pixel 200 necessarily contribute to this count. The specific hardware configuration of the in-pixel counter 223 will be described later (see, for example...). Figure 17 ).

[0068] <Number of photons per unit exposure time> [Subframe] Figure 3 A schematic diagram illustrating the time aperture imaging algorithm in this embodiment is shown. In this embodiment, a frame may include multiple subframes. Figure 3 In one example shown, a frame comprises four subframes. If a frame (e.g., 16.7ms) comprises four subframes, then one subframe corresponds to a 4.2ms time interval. Furthermore, a subframe can include multiple exposure time units. Figure 3 An example is shown where 1024 exposure time units are included in a single subframe. Therefore, in this case, 1024 charging operations are performed within a single subframe.

[0069] [Sensitivity Channel] The number of photons arriving at APD 211 increases or decreases depending on the illuminance level (e.g., intensity), but regardless of the illuminance level, the average arrival interval of photons to APD 211 is known to follow an exponential distribution. Photons can arrive at APD 211 randomly, therefore the probability density of photons arriving at APD 211 at any time during the charging period can be expected to be unbiased. By utilizing this randomness, in temporal aperture imaging algorithms, exposure time can be determined via temporal aperture, and multiple virtual sensitivity channels can be set within a single pixel.

[0070] In this embodiment, multiple virtual sensitivity channels (or "sensitivity channels") T can be set for an exposure time unit (i=1, 2, ...). i Multiple virtual sensitivity channels T i This can be used to confirm multiple time periods of photon detection. Figure 3 In the middle, each sensitivity channel T i Exposure can be started by the "Start of Exposure (SOE)" indicated by a black inverted triangle, and ended by the "End of Exposure (EOE)" indicated by a black triangle. The end of exposure can also be used as a confirmation time for photon detection. For example, this can be set in each virtual sensitivity channel T. i At the end, confirmation of photon detection is performed. Controller 110 can perform this confirmation on each sensitivity channel T. i The CHK signal (determining signal) is activated at multiple different times when the exposure ends. In one example, the gating circuit 222 can generate a pulse signal based on the output signal of the APD 211 at one of the multiple different times corresponding to the end of the virtual sensitivity channel with the highest sensitivity.

[0071] If the charging signal is active low, then each sensitivity channel T iExposure can begin when the charging signal becomes LO (low), and can be performed after T for each sensitivity channel since the beginning. i The time aperture period (i=1,2,...) is set to end at the specified time interval. Therefore, the time for checking photon detection can be different. Meanwhile, for all sensitivity channels T... i Exposure can essentially begin simultaneously. That is to say, the sensitivity channel T... i They can start at essentially the same time. Sensitivity channel T i The time aperture period can be compared with the sensitivity channel with the lowest sensitivity from the start of exposure. Figure 3 The time periods corresponding to the time aperture time periods (T7) overlap with each other.

[0072] In this way, in this embodiment, the photoelectric conversion device 100 may include multiple virtual sensitivity channels T with different time lengths per unit exposure time. i Sensitivity channel T i Essentially, exposures can begin at the same start time and end at different end times, and photon detection checks can be performed. Sensitivity channel T i It can be configured to include another virtual sensitivity channel on the time axis.

[0073] exist Figure 3 Examples of sensitivity channels T1 to T7 being configured are given in the text. In the following description, although examples of sensitivity channels T1 to T7 or T1 to T3 being configured are provided, this disclosure is not limited to these cases. Sensitivity channel T1 may be the most sensitive channel (e.g., the highest sensitivity channel) having a time aperture period of 1 unit exposure time. Sensitivity channel T1 may have sensitivity for detecting photons (counting photons) over a period of time substantially equal to the unit exposure time (e.g., a time interval). T2 may be a time aperture period with a time aperture of 1 unit exposure time... A sensitivity channel with a time aperture period of times that of a given exposure time. That is, sensitivity channel T2 can have the sensitivity to detect photons during a period of half the exposure time. T3 can have the sensitivity of a time aperture period of times that of a given exposure time. A sensitivity channel with a time aperture period that is times larger than the time aperture. That is, sensitivity channel T3 can have the sensitivity to detect photons during a quarter of a unit exposure time. This may be suitable for T4 through T7.

[0074] The ratio of the time aperture period to the unit exposure time of the sensitivity channels T1 to T7 (hereinafter referred to as the "time aperture ratio (TAR)") is not limited to the ratios mentioned above (such as 1 / 2, 1 / 4, etc.). However, for hardware configuration convenience, it is desirable for the time aperture ratio to be set to... (k is a natural number). Furthermore, as described later, various time aperture ratios can be combined for sensitivity channels T1 to T7.

[0075] [Method for counting photons per unit exposure time] As the intensity of light incident on APD 211 (hereinafter referred to as "incident illuminance") increases, the number of photons incident on APD 211 can increase. Furthermore, the exposure start can be common for each of the sensitivity channels T1 to T7, and photons can arrive at APD 211 randomly; therefore, as the incident illuminance increases, the probability of photon incident during a charging period can increase. In particular, with higher incident illuminance, the probability of photons incident on a low-sensitivity sensitivity channel (hereinafter referred to as "low-sensitivity channel") with a short aperture period can be higher.

[0076] In addition, the sensitivity channel T i The time aperture periods (i=1,2,…) can partially overlap with each other, so when a photon is incident on a specific sensitivity channel T… i When the photon is incident on the channel T with higher sensitivity, the photon is also incident on the channel T with higher sensitivity. i The sensitivity of the other sensitivity channel T is high. j (j>i). Therefore, regardless of the time dependence of the photon incident time, the time aperture period of each of the sensitivity channels T1 to T7 can be matched with the sensitivity level (i.e., the sensitivity weight value) of each of the sensitivity channels T1 to T7. For example, the time aperture period can be considered equivalent to the sensitivity weight value for the sensitivity channel.

[0077] In temporal aperture imaging algorithms, the sensitivity channel T can be used to measure the time per unit exposure. i Photon Counting (CNT) i The total number of photons incident on the APD 211 is used to calculate the number of photons. A detailed description of the method used to calculate the number of photons is given below.

[0078] The count value CNT can be initialized to 0 before counting begins. For each of the sensitivity channels T1 to T7, it can be determined whether a photon is incident during the time aperture period, and if a photon is incident, 1 can be added to the count value CNT. That is, for each of the sensitivity channels T1 to T7, it can be determined whether a photon is incident from the beginning to the end of the exposure, and if so, 1 can be added to the count value CNT. On the other hand, if no photon is incident, nothing can be added. For example, when a photon is incident on sensitivity channel T7, 1 can be added to the count value CNT of sensitivity channel T7. Furthermore, the start of exposure is common for each of the sensitivity channels T1 to T7, so 1 can be added to the count value CNT of each of the sensitivity channels T1 to T6. Therefore, the final count value CNT can be 7. In addition, if a photon is incident during the period when sensitivity channel T1 is set, the final count value CNT can be 1. If no photon enters sensitivity channel T1, the final count value CNT can be 0. In this embodiment, through the sensitivity channel T i Counting photons can be achieved, for example, by using an in-pixel counter 223 as described later.

[0079] [Information related to the number of photons per unit exposure time] Figure 4 An example amount of information is shown regarding the number of photons per charge, depending on the sensitivity channel used.

[0080] If the information I related to the number of photons is defined as a logarithm with base 2... Therefore, when using sensitivity channels T1 to T7, the information quantity I can be increased to a maximum of 3 bits per charge. This varies depending on the sensitivity channel T used. i As the quantity increases, more information related to the incident irradiance can be obtained using a single charging operation. In the formula defining information quantity I, CNT can represent the final count value.

[0081] In this way, in this embodiment, information related to the amount of incident light can be represented using multiple bits (multi-bit) and acquired at once (referred to as "multi-bit information acquisition"). For example, the photoelectric conversion device 100 can acquire a large amount of information related to the amount of incident light at each illuminance height by utilizing the statistical properties of photons.

[0082] [A conceptual model visualized using time aperture] Figure 5 A schematic diagram of an example conceptual model for incident photon detection using a temporal aperture imaging algorithm that achieves multiple sensitivities is shown. The acquisition of multiple bits of information can be achieved through... Figure 5The conceptual model shown in the figure is used for modeling. In the conceptual model shown in the figure, for a pixel, there can be multiple SPAD elements, and a neutral density (ND) filter corresponding to the temporal aperture ratio can be placed in front of each SPAD element. Each ND filter can have different sensitivity characteristics depending on the temporal aperture ratio. Multi-bit information acquisition can be considered equivalent to counting (or summing) all outputs from multiple SPAD elements for incident light attenuated by the ND filter. In conventional CIS (CMOS image sensors), regardless of the amount of light, there can be a structure within the pixel that accumulates electrons that convert photoelectric signals once, making it infeasible to perform modeling with multiple sensitivity characteristics.

[0083] [Dynamic range of illuminance].

[0084] Figure 6 Example count values ​​based on illuminance height are shown. Figure 7 Examples of dynamic range expansion and the number of in-pixel counter bits are shown, corresponding to various combinations of the time aperture ratio of the sensitivity channel.

[0085] In this specification, the range of illuminance within which the photoelectric conversion device 100 can detect photons (count photons) is referred to as the dynamic range of illuminance (DR) (hereinafter also simply referred to as "dynamic range"). For better understanding and ease of description, an example is given below where incident illuminance is divided into three levels (high illuminance, medium illuminance, and low illuminance). Several types of incident illuminance may exist.

[0086] High illuminance can refer to very high illuminance (such as sunlight). Under high illuminance conditions, a large number of photons can arrive within a given time period. Medium illuminance can refer to illuminance achieved using, for example, typical indoor lighting. Low illuminance can refer to illuminance under dark lighting conditions (such as moonlight at night). Under low illuminance conditions, it is not uncommon for no photons to be received during a single charging period.

[0087] like Figure 6As shown, the number of photons incident during a charging period can increase with increasing illuminance. Therefore, even during short aperture periods set as low-sensitivity channels, the probability of photon incident can increase, thus increasing the final count value CNT. For example, the temporal aperture ratio of the lowest sensitivity channel T7 to the highest sensitivity channel T1 can be 64 times. That is, for high-intensity light, the increase in the count value of sensitivity channel T7 is suppressed by 64 times compared to the increase in the count value of sensitivity channel T1. Therefore, it is feasible to count using a small number of bits of the intra-pixel counter without saturating the count value. The illuminance range in which each sensitivity channel operates linearly can be controlled by the temporal aperture ratio, and thus, the dynamic range in pixel 200 can be easily extended.

[0088] For each of high, medium, and low illumination levels, the APD 211 can measure (detect) the first incident photon (or first photon) during the charging period. Because the potential difference between the anode and cathode decreases during the measurement of the first photon, the APD 211 does not need to measure photons incident after a second time interval. Thus, photons after the first photon can be ignored. Figure 6 In one example of high illumination, the first photon incident during the time aperture period of sensitivity channel T6 following the time aperture period of sensitivity channel T7 is measured, and photons incident after the second time are ignored; however, the count value can be changed to 6 from sensitivity channels T6 to T1 using the time aperture imaging algorithm of this embodiment. Furthermore, in one example of medium illumination, the first photon incident during the time aperture period of sensitivity channel T4 following the time aperture period of sensitivity channel T5 is measured, and photons incident after the second time are ignored; however, they can be counted from sensitivity channels T4 to T1, and the count value can be changed to 4. In one example of low illumination, the first photon incident during the time aperture period of sensitivity channel T2 following the time aperture period of sensitivity channel T3 is measured, and photons incident after the second time are ignored; however, they can be counted in sensitivity channels T2 to T1, and the count value can be changed to 2.

[0089] Furthermore, the number of sensitivity channels to be set can be one or more. To achieve the effect of extending the dynamic range, multiple sensitivity channels T can be set. i This is desirable. However, the time aperture period of each sensitivity channel Ti can be set to include the time aperture period of the lower sensitivity channels. By constraining the settings of these sensitivity channels, time dependence can be eliminated even for randomly arriving photons, and thus the time aperture period of each sensitivity channel T can be maintained. i While achieving a higher sensitivity ratio, the input-output transfer function described below is also implemented.

[0090] like Figure 7 As shown, by setting each sensitivity channel T i The time aperture ratio (TAR) allows for arbitrary extension of the dynamic range, and the dynamic range extension is greater than that of DR. EXT According to the following formula (1), the sensitivity channel T i The time aperture ratio (TAR) is the time aperture ratio (TAR) per sensitivity channel. i The ratio of the time aperture period to the time aperture period of the sensitivity channel T1.

[0091] [Formula 1]

[0092] The accompanying figure illustrates the expansion of the dynamic range (DR) calculated for cases 1 through 3. Furthermore, for each case, the number of bits of the in-pixel counter required to implement photon counting in hardware is shown. In this way, the characteristic that allows easy control over the dynamic range expansion is referred to in this specification as "DR expansion controllability".

[0093] Furthermore, as will be described below, it is feasible to suppress the occurrence of signal-to-noise ratio (SNR) dip, or to design a tradeoff between SNR and DR expansion rate (referred to as "easyness of design tradeoff"), depending on the number of sensitivity channels and the time aperture ratio. Moreover, by performing mode preparation based on the imaging scenario while considering the tradeoff design, optimizations can be performed, including power consumption of the entire imaging system and signal processing within the imaging system.

[0094] As mentioned above, the average photon arrival time interval can follow an exponential distribution. Therefore, typically in SPAD pixels with clock-synchronized charging, if the average photon arrival time interval is close to the charging interval, the probability of multiple photons being incident during a charging interval can increase. Consequently, due to this deviation in the arrival time interval, the count value can exhibit nonlinearity near saturation; however, for simplicity, the count value can be assumed to have linear characteristics.

[0095] Figure 8 Example graphs are shown, illustrating the sensitivity channel transfer function and the normalized aggregate transfer function, respectively, by summing the pixel counts over a unit exposure time over a frame period. The aggregate transfer function can be an aggregation of the sensitivity channel transfer functions. Figure 8 In the diagram, the horizontal axis represents the logarithm of illuminance (log), and the vertical axis represents the logarithm of the count value (log).

[0096] Sensitivity channel T1 can handle counting operations from low illuminance I1 to medium illuminance I2. During the period when the incident illuminance changes from low illuminance I1 to medium illuminance I2, sensitivity channel T1 can be linear, and it may become dominant over other sensitivity channels. Furthermore, when the incident illuminance changes from medium illuminance I2 to high illuminance I3, sensitivity channel T2 can handle counting operations. During the period when the incident illuminance changes from medium illuminance I2 to high illuminance I3, sensitivity channel T2 can be linear, and it may become dominant over other sensitivity channels. Moreover, even after the incident illuminance changes to high illuminance I3, sensitivity channel T3 can handle counting operations until a predetermined illuminance level. During the period when the incident illuminance changes from high illuminance I3 to the predetermined illuminance level, sensitivity channel T3 can be linear, and it may become dominant over other sensitivity channels.

[0097] Figure 9 Example count values ​​based on illuminance and example dominant sensitivity channels for each illuminance are shown. For example, in the case of low illuminance I1, the count values ​​of sensitivity channels T1, T2, and T3 can be C1L, C2L, and C3L, respectively. In the case of low illuminance I1, the dominant sensitivity channel can be sensitivity channel T1. For example, in the case of medium illuminance I2, the count values ​​of sensitivity channels T1, T2, and T3 can be Max (maximum value), C2M, and C3M, respectively. In the case of medium illuminance I2, the dominant sensitivity channel can be sensitivity channel T2. For example, in the case of high illuminance I3, the count values ​​of sensitivity channels T1, T2, and T3 can be Max, Max, and C3H, respectively. In the case of high illuminance I3, the dominant sensitivity channel can be sensitivity channel T3.

[0098] When the illuminance increases beyond the medium illuminance I2, sensitivity channel T1 may become saturated and nonlinear, and may be unable to handle counting operations. However, sensitivity channel T2 can remain linear in this region and is more sensitive than sensitivity channel T3. Therefore, sensitivity channel T2 plays a central role in increasing the accumulated count value. When the illuminance increases beyond the high illuminance I3, sensitivity channels T1 and T2 may become saturated after a nonlinear response and may be unable to handle counting operations. However, sensitivity channel T3 can remain linear up to a specific illuminance and also possess a specific sensitivity. Therefore, sensitivity channel T3 plays a central role in increasing the accumulated count value.

[0099] Sensitivity channels T1, T2, and T3 can have constant sensitivity to illuminance, so the composite count obtained by normalizing the accumulated counts can show a monotonically increasing illuminance. Furthermore, by normalizing the accumulated counts via a normalization factor NF, the number of incident photons can be accurately obtained, and the effects of the number of sensitivity channels or the time aperture ratio are corrected (i.e., sensitivity increases are corrected). NF can be expressed by the following Equation 2.

[0100] [Formula 2]

[0101] In Formula 2, T i The indicator shows the sensitivity channel; TAR indicates the time-aperture ratio; and NC indicates the total number of sensitivity channels.

[0102] Furthermore, due to the very high photoelectric conversion gain of SPAD 211, it is possible to determine whether a photon detected during the charging period contributes to the count value by quantum behavior excluding readout noise. Moreover, the aggregation transfer function obtained by synthesizing each sensitivity channel can be determined substantially uniquely, thus it is feasible to calculate the number of incident photons substantially accurately as a linear count value proportional to the incident illuminance (i.e., substantially completely linearized by processing of preprocessor 131, which will be described below). However, the calculated value may also include optical shot noise inherent in the original photons.

[0103] In this way, in this embodiment, multiple sensitivity channels T with varying time aperture periods within an exposure time unit can be used. i Photons can have an average non-deviating probability of occurrence during the charging period, thus allowing arbitrary sensitivity to be achieved by varying the temporal aperture over a very short charging period. For example, by making the aperture very short, it is possible to achieve a channel with very low sensitivity, and it is also possible to obtain an unsaturated image even for objects with very high illumination. Therefore, even for a high-illumination object that originally required 1 million counts, it can now be represented using approximately 10,000 to 20,000 counts during a single frame period.

[0104] [Input / Output Transfer Function] Assuming that photons incident on the APD 211 are scattered evenly according to the incident illuminance lx during the charging period, then the sensitivity channel T of the incident photons... i The count value (detection value) CNT (T) i (lx) can be calculated. However, here, illuminance lx can be proportional to the number of incident photons, therefore in the following formula, illuminance I xThis can represent the number of photons. Furthermore, the count value CNT(T) can be expressed using the following formula 3. i (lx) is accumulated according to the number of sensitivity channels NC to calculate the expected value E[CNT(lx)] of the count value during a charge.

[0105] [Formula 3]

[0106] The expected values ​​mentioned above can include both the nonlinearity caused by clock-synchronized charging operation and the sensitivity characteristics due to the time aperture method. The accumulated count value (i.e., the transfer function) for any incident illuminance lx can be calculated. Furthermore, a normalized aggregate transfer function can be calculated by multiplying the calculated transfer function by a normalization factor NF. (Formula 4 below).

[0107] [Formula 4]

[0108] Furthermore, the expected value of the count in the clock-synchronized charging method can be calculated using the following formula 5, based on the calculation of the expected value under the conditions of no photon incident during the charging period and under the conditions of one or more photons incident in a Poisson distribution. However, T ETU Indicates the time of one exposure time unit.

[0109] [Formula 5]

[0110] [SNR, Dynamic Range (DR) Spread Ratio] Figure 10 The graphs shown illustrate examples of SNR and DR magnification, and Figure 11 A graph showing another example of SNR Dip occurring in SNR and DR amplification is presented.

[0111] For example, suppose three sensitivity channels T1, T2, and T3 are used. When the time aperture ratio of sensitivity channels T1, T2, and T3 is set to T1:T2:T3 = 1:1 / 4:1 / 4, the SNR can increase monotonically, but the obtained DR amplification can be 16 times. Furthermore, as... Figure 11As shown, when the time aperture ratio of sensitivity channels T1, T2, and T3 is set to T1:T2:T3 = 1:1 / 4:1 / 16, sensitivity channel T3 can have a low count value when sensitivity channel T2 is saturated. Therefore, the SNR can be lower compared to when the illuminance range (DR amplification) is small. That is, a decrease in SNR (SNR Dip) can occur. However, even if the SNR decreases due to SNR Dip, it is known that when the SNR exceeds a certain level, the SNR cannot be detected by the human eye, and by utilizing this characteristic, it is feasible to achieve a high DR amplification (64x in this case) while maintaining a sufficiently high SNR.

[0112] Furthermore, if the SNR is already high enough and the image quality is not affected by degradation, a wider dynamic range can be achieved using fewer sensitivity channels by making the time aperture ratio relatively large. For example, by reducing the number of sensitivity channels, the number of bits in the counter can be reduced or power consumption can be decreased. In this way, this embodiment is characterized by the ability to easily implement performance trade-offs regarding SNR and DR amplification.

[0113] [Conversion of time aperture imaging parameters] Figure 12 A schematic diagram is shown to illustrate the transformation of time aperture imaging parameters according to imaging illumination conditions. Figure 13 An example of setting the sensitivity ratio between each sensitivity channel is shown.

[0114] like Figure 12 As shown, the temporal aperture imaging parameters can be configured to be converted according to imaging illumination conditions. Imaging illumination conditions may include conditions related to illumination at the imaging location, and these conditions may include, for example, a scene with standard illumination, a dark scene, a scene with large differences in brightness (high dynamic range), etc. The temporal aperture imaging parameters may include parameters used in imaging in temporal aperture mode, and in this embodiment, for example, the temporal aperture imaging parameters may include the sensitivity ratio between each sensitivity channel. The user can prepare a set of temporal aperture imaging parameters according to each imaging illumination condition and a correspondingly assumed subject or use case. For example, the user can input a set of temporal aperture imaging parameters into the photoelectric conversion device 100 at the time of imaging, or can select pre-registered temporal aperture imaging parameters. Therefore, the photoelectric conversion device 100 can acquire (capture) images under optimal operating conditions. Optionally, the photoelectric conversion device 100 can be configured to determine the illumination range from the currently acquired image and can automatically convert the temporal aperture imaging parameters.

[0115] like Figure 13As shown, for example, case A is considered to be imaging under standard illumination conditions, so the sensitivity ratio of the four sensitivity channels T1 to T4 can be set to be equally sensitive to 1 / 4 (i.e., all 1 / 4). Therefore, a dynamic range extension ratio of 4×4×4=64 times can be obtained.

[0116] Furthermore, since case B involves imaging under moonlight at night, insufficient illumination is assumed, allowing more sensitivity channels to be allocated to the high-sensitivity side. This improves image quality in low-light conditions. Additionally, sensitivity channel T6 can be used. Therefore, objects or people walking at night can be recorded brightly without white saturation.

[0117] Furthermore, in case C, a high dynamic range scene exists within the same scene, encompassing both high and low illumination. Therefore, all seven sensitivity channels T1 to T7 can be used, and the sensitivity ratio of T2 to T3 can be reduced to improve image quality on the low-illumination side. Consequently, the dynamic range can be extended to the high-illumination side while maintaining low-illumination image quality. Thus, for example, details of passengers in a car can be captured at essentially the same time as a subject illuminated by the car's bright headlights.

[0118] [Configuration of Data Accumulator 140] Figure 14 A schematic diagram illustrates an example of reconstructing an image by performing correction processing on pixel data. Furthermore, Figure 15 A schematic diagram is shown illustrating an example case of reconstructing an image by simply adding pixel data.

[0119] Unlike CIS imaging, SPAD imaging may be free of readout noise or quantization noise due to its very high photoelectric conversion gain. For example, readout noise and quantization noise can be zero in SPAD imaging, and the image can be acquired during the accumulation phase. Therefore, even though the image is accumulated and visualized during the accumulation phase in SPAD imaging, there is no SNR degradation due to noise; on the contrary, image quality can be improved as accumulation proceeds because the shot noise ratio of the signal is reduced. In this way, SPAD imaging can exhibit excellent characteristics in terms of both SNR and image quality.

[0120] like Figure 14As shown, the data accumulator 140 can first generate a first intermediate image as a first process by simply adding the count values ​​of a counter within a plurality of exposure time units. Then, the data accumulator 140 can perform motion correction processing on a plurality of the first intermediate images to generate a synthesized second intermediate image. Then, the data accumulator 140 can perform blur correction processing on the second intermediate image. Then, the data accumulator 140 can perform accumulation and motion correction processing again on the second intermediate image that has already undergone blur correction processing as a second process, thereby generating the final subframe image. Alternatively, the second processing may not be required. Furthermore, motion correction processing, blur correction processing, and denoising processing can be combined. When the amount of motion is small, motion blur correction can also be performed on a subframe basis.

[0121] In this embodiment, information from high to low illumination has been obtained at a single exposure time frame using a temporal aperture imaging algorithm, allowing motion correction processing to be performed across the entire illumination range. Therefore, clear motion images can be obtained across the entire illumination range.

[0122] Furthermore, in temporal aperture imaging algorithms, information acquired during a short charging period, ranging from high to low illumination, can be accumulated at a high frame rate and output as an image during this processing. Therefore, although the amount of image information during the accumulation period of a single scene is small, it is feasible to accumulate images during the accumulation period while aligning the subject's position across a wide range from low to high illumination, for example, in a series of scenes. Moreover, by combining the accumulation of intermediate accumulated images with the first and second processes, motion displacement caused by subject movement or camera movement itself can be suppressed, and in particular, it is feasible to generate high dynamic range motion pictures with good SNR for low-light subjects. Therefore, the image quality of high dynamic range motion pictures can be improved.

[0123] like Figure 15 As shown, for example, by acquiring 3-bit frame images within one exposure time unit and simply summing the acquired frame images, it is feasible to acquire images without causing motion blur, even when the subject is moving quickly, under conditions other than low light. However, there are situations where sufficient contrast cannot be obtained, especially when the subject is moving quickly in low light conditions.

[0124] [Configuration of Signal Processor 130] Figure 16 It shows Figure 1 The block diagram shows an example schematic configuration of the signal processor 130. The signal processor 130 may include a preprocessor 131 and an image processor 132.

[0125] The preprocessor 131 can perform normalization and linearization processing (referred to as the "preprocessing correction process") on the image generated by the temporal aperture imaging algorithm (temporal aperture generated image). The preprocessor 131 may include a temporal aperture (TA) normalizer device (normalization processor) 21 (TA NORM), an inverse function calculator 22, a lookup table (LUT) 23, and a transfer function linearizer (linearization processor) 24 (TF LIN).

[0126] The temporal aperture normalizer 21 performs normalization processing on the image (raw data of the count values) generated by the temporal aperture imaging algorithm, thereby calculating the normalized count value (the accurate number of photons). More specifically, as... Figure 16 As shown in the graph at the lower left, the aggregation transfer function before normalization is in a state where the sensitivity is multiplied by multiple sensitivity channels (the gap G between the arrows). Therefore, through normalization, the original sensitivity can be recovered and the accurate number of photons can be calculated. In one example, the normalization processor 21 can calculate the number of photons contributing to the occurrence of the avalanche operation in a predetermined frame by summing the counter counts over multiple unit exposure times and normalizing the sum.

[0127] The inverse function calculator 22 can calculate the aggregate transfer function TF. agg, norm (lx). As mentioned above, the aggregation transfer function TF agg, norm (lx) may have properties obtained by combining the following characteristics: sensitivity associated with the temporal aperture imaging algorithm, photon arrival interval of exponential distribution, and nonlinearity associated with the clock-synchronized charging method. The inverse function Lin(lx) of the aggregate transfer function may include a gain correction coefficient (or linearization factor) that restores the nonlinear region of the calculated photon number to a linear type, and may be expressed by Equation 6 below.

[0128] [Formula 6]

[0129] Here, TF LIN (lx) is a linear function of a specific reference illuminance lx, which includes photon detection efficiency (such as quantum efficiency or avalanche probability). Aggregate transfer function TF agg, norm (lx) can be expressed as a complex formula, making it difficult to obtain the number of incident photons analytically. Therefore, in the inverse function calculator 22, the number of incident photons can be obtained from the aggregation transfer function TF. agg, norm (lx) and TF LINThe input / output relationship is obtained in advance, and the number of incident photons can be calculated by writing the relationship between the output and the input as the inverse function Lin(lx) into LUT 23 based on the input / output relationship.

[0130] The transfer function linearizer 24 can perform linearization on a nonlinear range of photon counts generated (e.g., corresponding to a high illuminance range) by referring to the temporal aperture normalizer 21. More specifically, the transfer function linearizer 24 can linearize the correspondence between illuminance and counts by referring to the LUT 23, which replaces the number of photons (count values) calculated by the temporal aperture normalizer 21 with the corresponding illuminance. The transfer function linearizer 24 can be used, for example, in the case of color images where it is necessary to linearize high illuminance regions. The linearization performed by the transfer function linearizer 24 need not be used for black and white images. Figure 16 In this context, "MAX" can represent the maximum value of the normalized and linearized aggregate transfer function.

[0131] Image processor 132 can perform various processing steps (such as demosaicing, automatic white balance (AWB), tone mapping, high dynamic range, and color processing) via a typical image signal processing pipeline. It is even feasible to apply conventional color reproduction or image processing to images captured by temporal aperture imaging algorithms by performing a pre-correction processing stream via preprocessor 131.

[0132] <Example of hardware configuration for 200 pixels> [Example of a two-layer structure] Figure 17 A circuit diagram is shown illustrating an example configuration of the hardware of pixel 200 as a two-layer substrate. For example, pixel 200 may include a light receiver 210 and pixel circuitry 220. The light receiver 210 includes an APD 211 mounted on a first layer (first substrate), and the pixel circuitry 220 is mounted on a second layer (second substrate). Using a hybrid bonding stacking technique, the output signal of the APD 211 on the first layer is transmitted to the pixel circuitry 220 on the second layer.

[0133] The second-layer pixel circuit 220 may include a charging circuit 221, a gating circuit 222, an in-pixel counter 223, a read latch circuit 224, a select circuit (SEL) 225, and an output bus 228.

[0134] The charging circuit 221 can charge the cathode of the APD 211 once per unit exposure time based on the charging signal (Recharge).

[0135] The gating circuit 222 can control the incrementing operation of the intra-pixel counter 223 by controlling the clock input of the intra-pixel counter 223. More specifically, the gating circuit 222 may include a NOR element, which can input the cathode node potential Vc and CHK signals and output an OUT signal.

[0136] The in-pixel counter 223 can be configured as an asynchronous counter (ripple counter) with a 3-bit counter and can be initialized to 0 via the RSTB signal. The in-pixel counter 223 performs incrementing counting (incrementing) via the rising edge of the OUT signal and detects the output (photons) of the APD 211. Any number of bits can be used in the in-pixel counter 223, as long as the in-pixel counter 223 can detect signals greater than or equal to the sensitivity channel T. i The number of values ​​can be counted. Typically, the size of the SPAD pixel 200 is limited by the size of the in-pixel counter 223, so a smaller size is better. Figure 17 The example shown uses a 3-bit configuration.

[0137] The read latch circuit 224 can maintain each bit output of the in-pixel counter 223 via a charging signal. The SEL 225 can control the reading of the bit value maintained in the read latch circuit 224 via the output bus 228 based on the SEL signal. Furthermore, the width (bit count) of the output bus 228 can preferably match the number of bits of the in-pixel counter, but they can be different. A smaller output bus width allows for easier miniaturization of the entire pixel because the layout area required for wiring can be reduced. Additionally, the most significant bit (MSB) of the in-pixel counter 223 can be maintained in the read latch circuit 224.

[0138] In this embodiment, a read latch circuit 224 may be installed to read the count value of the in-pixel counter 223 in a manner that overlaps temporally with temporal aperture imaging. Reading of all pixels can be completed within a single charging period. In this case, since the unit exposure time and readout time can overlap, no dead period for imaging due to reading will occur, and thus imaging can be performed without loss of sensitivity. Furthermore, instead of installing the read latch circuit 224, it is also feasible to perform low-speed readings during a charging period or longer by installing +1 or more extra bits in the in-pixel counter 223.

[0139] [Example of two-layer pixel circuit operation] Figure 18 A description is shown Figure 17The timing diagram for the operation of pixel circuit 220 shown is illustrated. However, to simplify the description of the counter operation, values ​​different from the actual circuit operation are used. Specifically, the value of counter 223 within the pixel can be described as decreasing when the clock arrives, but even if it were considered to be increasing, no inconsistency in the operation itself would occur.

[0140] (1) When the charging signal is at the LO level (or LO), the cathode node potential Vc of the APD 211 can be pulled up to the HI level (or HI). Therefore, the APD 211 can operate in Geiger mode and is ready to receive photons.

[0141] (2) When a photon is incident on APD 211, avalanche breakdown can occur and the cathode node potential Vc drops to the LO level.

[0142] (3) When the RSTB signal of the pixel counter 223 changes to LO, the pixel counter 223 is cleared to 0. Therefore, preparation for the counting operation can be performed.

[0143] (4) When the CHK signal changes to LO, the logic state of the cathode node potential Vc is output as an OUT signal and checked. If the cathode node potential Vc is HI, the OUT signal changes to LO through NOR logic. If the cathode node potential Vc is LO, the OUT signal changes to HI through NOR logic. When the OUT signal changes to HI, a rising edge can be generated, so the value of the in-pixel counter 223 can be incremented accordingly.

[0144] In this way, the gating circuit 222 can send the logic state of the cathode node potential Vc to the OUT signal according to the logic state of the CHK signal. Therefore, it is possible to send the logic state of the cathode node potential Vc to the OUT signal from the sensitivity channel T. i The state of the cathode node potential Vc of APD 211 (the output of APD 211) is checked at a time selected from among a plurality of different termination times. In one example, gating circuit 222 can generate a pulse signal based on the output signal of APD 211 at at least one time selected from a plurality of different termination times.

[0145] (5) Once avalanche breakdown occurs, the cathode node potential Vc increases by a count value whenever the CHK signal changes to LO in order to maintain the state. Therefore, time-aperture imaging can be performed. Figure 18 In one example, three sensitivity channels T i The circuit is set up, and during the first charging period, the first photon enters sensitivity channel T3. Therefore, three counts can be performed using the respective CHK signals of sensitivity channels T3, T2, and T1. Furthermore, the first photon enters sensitivity channel T1 during the second charging period. Therefore, one count can be performed using the CHK signal of sensitivity channel T1.

[0146] (6) When the charging signal changes to LO, the output value of the in-pixel counter 223 can be written to the read latch circuit 224 (LO - pass, HI - latch). For example, the read latch circuit 224 can allow the output value of the in-pixel counter 223 to pass when the charging signal is LO (the output value of the in-pixel counter 223 may not be maintained), and can latch (maintain) the output value of the in-pixel counter 223 when the charging signal is HI. The value written to the read latch circuit 224 can be read during a charging period. Figure 18 In one example, when the charging signal changes from LO to HI during the second charging period, the count value "3" during the first charging period can be written to the read latch circuit 224 and can be read out during the second charging period. Furthermore, subsequent incident photons within a single charging period can be ignored and not counted.

[0147] [Low power consumption of photoelectric conversion device 100] Figure 19 A circuit diagram illustrating an example equivalent circuit for calculating the power consumption at the cathode node of SPAD 211 is shown. Figure 20 It shows Figure 19 Timing diagrams of example charging and discharging operations in the equivalent circuit shown.

[0148] exist Figure 19 In this equation, the power consumption P caused by charging and discharging at the cathode node can be expressed by the following formula 7. In formula 7, V BD Indicator breakdown voltage, C tot The total capacitance at the cathode node is indicated by VEX, which is the voltage excursion amplitude (in this configuration, it is equal to the supply voltage), and f indicates the charging frequency.

[0149] [Formula 7]

[0150] Typically, to minimize pixel size, the SPAD 211 and the charging transistor or inverter circuit 229 of the charging circuit 221 can be fabricated as separate wafers and bonded using a hybrid bonding technique, one of the stacking techniques. The inverter circuit 229 can be used to shape the cathode potential into a waveform and convert it into a digital voltage. Hybrid bonding can typically require large electrode sizes, which in turn increases parasitic capacitance. For example, the parasitic capacitance C... HB It can be considered to be approximately 5 fF / ea on average. For example, the parasitic capacitance C of the charging circuit 211. d,recharge Parasitic capacitance C of inverter circuit 229 gg,invThe parasitic capacitance C of SPAD 211 can be approximately 0.1 fF. ca,spad It could be approximately 0.2fF.

[0151] Furthermore, to operate the SPAD 211 in avalanche mode with high quantum sensitivity, a high voltage (e.g., 3V) may be required as an overvoltage VEX. However, the charging / discharging power can be generated once proportional to the voltage offset amplitude of the cathode node (here, VEX) and can flow to the anode terminal, which is subjected to a large negative voltage (typically close to -20V in the case of silicon substrates), for discharge. Therefore, the charging / discharging power due to the charging operation can be considered to be more dominant than the power consumed in the subsequent in-pixel counter 223, which typically uses a low power voltage of 1V or lower. Thus, it can be said that most of the power consumed in the SPAD pixel 200 is the power consumed by the charging and discharging operations of the parasitic capacitance associated with the cathode node.

[0152] In this embodiment, multiple bits of information can be obtained in a single charging operation using a temporal aperture imaging algorithm. Therefore, compared to the conventional clock-synchronized charging method shown in Non-Patent Document 1, it is feasible to achieve a dynamic range and SNR equal to or greater than its conventional charging cycle (e.g., 1 / 2.5 times the charging cycle, e.g., as small as 40%). Thus, scenarios with high dynamic range can be achieved with significantly reduced power consumption compared to conventional clock-synchronized charging methods.

[0153] [Easy SPAD pixel 200 multi-pixel conversion] Figure 21 A schematic diagram illustrating an example variation in power consumption during imaging using a time-aperture imaging algorithm is shown. Figure 22 A schematic diagram is shown illustrating an example variation in power consumption during imaging using a conventional clock-synchronized charging method as a comparative example.

[0154] like Figure 21 As shown, in this embodiment, the SPAD pixel 200 can always perform charging operations at constant intervals regardless of the amount of incident light. Therefore, even when the incident light intensity is very strong or very weak, the power consumption variation at the cathode node can be minimized. Thus, power fluctuations in the pixel array 120 can be suppressed. Furthermore, peak current at the cathode node can also be suppressed. Therefore, the power wiring design of the pixel array 120 can be made easier, and thus multi-pixelation can be easily achieved.

[0155] In addition, such as Figure 22 As shown, in imaging using a conventional clock-synchronized charging method, the power consumption fluctuations in SPAD pixels can be large. Therefore, the power wiring design of the pixel array section 120 is difficult, and thus multi-pixelation is not easily achieved.

[0156] [Easy to miniaturize SPAD pixels to 200] Figure 23 A schematic diagram is shown illustrating an example count of the counter within a pixel per exposure time unit in an image using a temporal aperture imaging algorithm. Figure 24 A schematic diagram is shown illustrating an example count of the counter within a pixel for each cluster in an image obtained using a conventional clock-synchronized charging method as a comparative example.

[0157] In temporal aperture imaging algorithms, charging operations can always be performed at constant intervals regardless of the incident light intensity, and for very strong light, a low-sensitivity channel with a low temporal aperture ratio can be responsible for photon detection. For example, using a sensitivity channel is equivalent to capturing an image by being light-sensitive, so it is feasible to perform the counting operation by suppressing the count increment rate within a charging period. This characteristic allows the in-pixel counter 223 to be configured using a small number of bits, thus enabling pixel miniaturization.

[0158] like Figure 23 As shown, in the temporal aperture imaging algorithm of this embodiment, when sensitivity channels T1 to T7 are used, the count increases by only 7 during one exposure time unit, so a 3-bit in-pixel counter is sufficient. The size of the SPAD pixel 200 can be limited by the size required by the in-pixel counter 223, rather than the size of the SPAD 211, so pixel miniaturization can be easily achieved through a temporal aperture imaging algorithm that can capture images with a small number of bits. Furthermore, if three sensitivity channels exist, it can be implemented using a 2-bit counter, making even smaller pixels feasible.

[0159] Meanwhile, in conventional clock-synchronized charging methods, such as Figure 24 As shown, the shortest charging cycle is approximately 25 ns, and it is assumed that this is performed burstily approximately 40 times. Furthermore, within a single cluster period comprising various types of charging cycles, the maximum value can be estimated to be approximately 125 counts. Therefore, to accurately count incident photons, an in-pixel counter may need to have a large number of bits (e.g., 8 bits).

[0160] Therefore, compared with conventional methods, the temporal aperture imaging algorithm method in this embodiment can be configured with a pixel front-end circuit that suppresses the number of bits of the in-pixel counter.

[0161] [Feasibility of High Frame Rates] Unlike conventional methods, in the temporal aperture imaging algorithm, the dynamic range of the spread ratio can be determined solely by the temporal aperture ratio. The low-bit binary image can be output at a high frame rate, regardless of the extent of dynamic range spread.

[0162] According to the photoelectric conversion device 100 of this embodiment, it is feasible to suppress the increase and fluctuation of pixel power consumption while simultaneously suppressing the increase of the circuit size of pixel 200. Furthermore, according to the photoelectric conversion device 100, regardless of the extent to which the dynamic range of illumination is extended, low-bit images can be output at a high frame rate.

[0163] (Second Embodiment) In the second embodiment, the hardware of pixel 200 is shown to be formed by a three-layer substrate. Figure 25 A circuit diagram illustrating an example configuration of three layers of pixel hardware in the photoelectric conversion device according to the second embodiment is shown. To avoid repetition, detailed descriptions of configurations identical to those in the first embodiment will be omitted.

[0164] [Example of a three-layer structure] Apart from the read latch circuit 224, SEL 225, and output bus 228, the configuration of the first and second layers can be the same as the two-layer configuration in the first embodiment. For example, on the second layer, the in-pixel counter 323 (or the third-layer in-pixel counter) (the upper counter portion) can be mounted on the third layer (the third substrate). The in-pixel counter 223 on the second layer (or the second-layer in-pixel counter) and the in-pixel counter 323 on the third layer can be connected in series, and the in-pixel counter 223 and the in-pixel counter 323 can cooperate to operate as a single large in-pixel counter. For the connection between the second-layer in-pixel counter 223 and the third-layer in-pixel counter 323, connecting elements (such as nano-through-silicon vias (TSVs, or through-silicon vias)) can be used.

[0165] Then, the number of bits in the second-layer intra-pixel counter 223 and the third-layer intra-pixel counter 323 does not necessarily have to be the same. The sum of the number of bits in the intra-pixel counter 223 and the intra-pixel counter 323 can be at least CEIL(log2(sensitivity channel T)). i The number of digits) + 1 or more. However, CEIL represents the round-up function.

[0166] In the three-layer configuration, the rate at which the final bit is changed is reduced. Therefore, all pixels of the pixel array 120 can be sent during multiple charging periods, thus making the read latch circuit 224 unnecessary. Instead of the read latch circuit 224, a selector can be installed to read all bits last, and whether or not bits are read can be controlled by the BITSEL signal.

[0167] [Example of operation of a three-layer pixel circuit] Figure 26 A description is shown Figure 25 The timing diagram shows the operation of the pixel circuit 220. However, for ease of description, Figure 26 An example like this is shown: Figure 25 The second-layer in-pixel counter 223 is configured with 2 bits, the third-layer in-pixel counter 323 is configured with 1 bit, and the number of sensitivity channels is set to 3.

[0168] (1) When the pixel counter 223 on the second layer becomes the full code, the carry is transferred to the pixel counter 323 on the third layer, and the value of the pixel counter 323 on the third layer is incremented.

[0169] (2) Next, the shortest time for the second-layer intra-pixel counter 223 to become a full code is after one charging time. Therefore, by reading the most significant bit of the intra-pixel counter 323 from pixel 200 up to that point, the incrementing events continuing from the lower bits can be accurately counted outside pixel 200 without any omissions.

[0170] (3) After all the accumulation counts have ended, the remaining lower bits of the pixel counter 223 are read sequentially using the BITSEL signal. Therefore, even if not all bits are read every time the charging time is completed, the same count value can be obtained in the end.

[0171] (Third embodiment) In the third embodiment, it is described that each pixel 200 of the pixel array 120 may include a light receiver 210 and a pixel circuit 220, and the photoelectric conversion device 100 may include a data accumulator 140 as peripheral circuitry of the pixel array 120. To avoid repetition, detailed descriptions of configurations identical to those in the first embodiment will be omitted.

[0172] [Example of pixel circuitry and data accumulator in a two-layer configuration] Figure 27 A circuit diagram illustrating an example pixel 200 and an example data accumulator 140 configured in two layers according to a third embodiment is shown. In this embodiment, in addition to the plurality of pixels 200, the pixel array section 120 may also include a data accumulator 140. Each pixel 200 may each include a light receiver 210 and a pixel circuit 220. As for the configuration of the light receiver 210 and the pixel circuit 220, they are the same as those in the first embodiment, therefore a detailed description will be omitted.

[0173] The data accumulator 140 may be disposed outside the pixel array section 120 and may include a column processor 241 and a frame memory 242. The column processor 241 may accumulate (add) pixel data (e.g., the count value of the counter) read from the read latch circuit 224 or the in-pixel counter 223 during multiple unit exposure times, and may write the accumulated value (the summed value) to the frame memory 242. The column processor 241 may include, for example, an addition circuit, and one addition circuit may be installed for each column. The frame memory 242 may be a memory that holds the accumulated value by the column processor 241, and may be, for example, a static random access memory (SRAM). For example, when simply accumulating pixel data, the column processor 241 may add the data (R) read from the address corresponding to the frame memory 242 and the pixel data output from the pixel circuit 220, and may write the summed value (W) to the address corresponding to the frame memory 242. In this way, by arranging a frame memory 242 around the pixel array section 120 and digitally accumulating it, the number of bits of the counter within the pixel can be reduced, and further, pixel miniaturization can be promoted.

[0174] [Layer configuration for 200 pixels] Figure 28 A schematic block diagram illustrating an example layer configuration of pixel 200 is shown. When the substrate has a two-layer configuration (left figure), a pixel counter array 244, column processor 241, and frame memory 242, including pixel circuitry 220 and sensitivity determiner 230 within the same layer, can be arranged on the second layer. In the pixel counter array 244, multiple pixel circuits 220 can be arranged in an array shape. The column processor 241 and frame memory 242 can be divided into multiple pairs within the same layer. Figure 28 An example is shown where a pair of column processors 241 and frame memory 242 are arranged sequentially (with pixel counter array 244 in between). Figure 28 In the example shown, the components are arranged in the vertical direction of the drawing, but they can also be arranged in the horizontal direction. Furthermore, the number of pairs (divisions) of the column processor 241 and frame memory 242 is not limited to two, and can be four or more.

[0175] Furthermore, when the substrate has a three-layer configuration (right figure), the pixel counter array 244 can be placed on the second layer, and the multi-pair processor 241 and frame memory 242 can be placed on the third layer directly below the pixel counter array 244. Figure 28 An example is shown with four pairs of column processors 241 and frame memory 242 arranged. The number of pairs (divisions) of column processors 241 and frame memory 242 is not limited to four, and can be eight or more. The parasitic capacitance or resistance of the readout wiring decreases with the number of divisions, thus enabling high-speed readout operations.

[0176] (Fourth embodiment) In the fourth embodiment, a case is described where imaging is performed using a high-sensitivity channel as needed. To avoid repetition, detailed descriptions of configurations identical to those in the first embodiment will be omitted.

[0177] [Example of modifying the gating circuit] Figure 29 A circuit diagram illustrating a modified example of the gating circuit according to the fourth embodiment is shown. Figure 29 An example of a two-stage gating circuit with one logic element in each stage is shown. Gating can be performed using one gating signal at each stage of the gating circuit (i.e., a total of two gating signals at stage 2). In this embodiment, the gating signal for the first stage is the CHK signal, and the gating signal for the second stage is the EN_LS (low sensitivity) signal. Therefore, the high-sensitivity sensitivity channel (hereinafter referred to as the "high-sensitivity channel") T1 can be used to count up the in-pixel counter. The figure shows an example of using a NOR element 251 in the first stage and an AND element 252 in the second stage, but other combinations of logic elements can be used as long as the same logical operation can be achieved.

[0178] exist Figure 29 In the example shown, the cathode node potential Vc is input to one input terminal of the NOR element 251, and the CHK signal is input to the other input terminal. Additionally, the output signal (OUT signal) of the NOR element is input to one input terminal of the AND element 252, and the EN_LS signal is input to the other input terminal of the AND element 252.

[0179] Available in each sensitivity channel T i The CHK signal is enabled at the end of the exposure. When the CHK signal is enabled, the logic state that is the same as the logic state of the cathode can be output as the OUT signal.

[0180] When the EN_LS signal is LO, the logic state of the OUT signal can be output as OUT2 without using the AND element 252. When the EN_LS signal is HI, the logic state of the OUT signal can be output as the OUT2 signal. The EN_LS signal can be used to control whether to check and count the low-sensitivity channels T2 and T3. When the EN_LS signal is LO, it can be a control signal that does not count the low-sensitivity channels T2 and T3. When the EN_LS signal is HI, it can be a control signal that counts the low-sensitivity channels T2 and T3.

[0181] It is feasible to control whether to check the high-sensitivity channel T1, or whether to also check the low-sensitivity channels T2 and T3, based on the logic level of the EN_LS signal. Therefore, in scenarios where high dynamic range images are not required, the number of times the value of the intra-pixel counter 223 is incremented can be suppressed by omitting the checks on the low-sensitivity channels T2 and T3. Thus, the power consumption of the intra-pixel counter 223 can be reduced.

[0182] Then, the EN_LS signal does not need to be common to all pixels 200 in the pixel array 120. For example, the pixel array 120 can be divided into multiple pixel regions, and different EN_LS signals can be supplied to the divided pixel regions respectively. In this embodiment, it is assumed that the high-sensitivity channel is T1, and all other sensitivity channels are low-sensitivity channels. In one example, a sensitivity channel with a time length greater than a predetermined length can be a high-sensitivity channel, and a sensitivity channel with a time length less than or equal to the predetermined length can be a low-sensitivity channel.

[0183] [Operation of charging circuit 221 and gating circuit 250] Figure 30 It shows Figure 29 Timing diagram of example operation of the charging circuit 221 and the gating circuit 250 shown.

[0184] (1) When the CHK signal becomes LO and the cathode node potential Vc is LO, the logic state of the OUT signal becomes HI.

[0185] (2) For example Figure 30 As shown in the dashed circular box section, by setting the EN_LS signal to HI when checking the sensitivity channel T1, the logic state of the OUT2 signal and the logic state of the OUT signal can become the same, and this can be used as an event to increment the value of the in-pixel counter 223 in a subsequent stage.

[0186] (3) In addition, during the time of checking sensitivity channels T2 and T3, the EN_LS signal is LO and the logic state of the OUT signal is not output as the OUT2 signal, so the pixel counter 223 does not count up.

[0187] (Fifth Embodiment) In the fifth embodiment, a method for detecting the type of photons using a low-sensitivity channel based on low-light conditions will be described. Furthermore, to avoid repetition, detailed descriptions of configurations identical to those in the second embodiment will be omitted.

[0188] Figure 31 A schematic diagram is shown to illustrate the detection of photons through a low-sensitivity channel under low-light conditions. Figure 31This shows an example scenario with three channels T1 to T3 set up.

[0189] In photon detection, optical shot noise, which is theoretically impossible to remove, is involved. The standard deviation of optical shot noise can be calculated using the square root of the optical signal. For example, as illumination increases, the signal-to-noise ratio (SNR) can increase, but a significant amount of optical shot noise can also occur. When comparing artifacts occurring in temporal aperture imaging algorithms with optical shot noise, if the optical shot noise is sufficiently large, it can be considered indistinguishable from the artifacts occurring in the temporal aperture imaging algorithm. Therefore, when comparing artifacts and optical shot noise, if the optical shot noise is sufficiently large, the artifacts can be ignored.

[0190] Considering the nature of these artifacts and optical shot noise, in this embodiment, photons can be detected under low-light conditions, for example, by using either of the following two methods.

[0191] Method 1 for improving SNR in low light: Count photons through the high-sensitivity channel T1 until the optical shot noise becomes sufficiently large, and then extend the count to include the low-sensitivity channels T2 and T3.

[0192] Method 2 for improving SNR in low-light conditions: The count values ​​obtained by the high-sensitivity channel T1 and the low-sensitivity channels T2 and T3 are accumulated completely independently in the frame memory 242, and the two transfer functions corresponding to the high-sensitivity and low-sensitivity channels are connected to each other. For example, the two transfer functions are connected at the coordinate points of incident illuminance where the signal-to-noise ratio (SNR) is sufficiently higher than 30 dB (threshold). For example, the count values ​​of the high-sensitivity channel T1 are used when the illuminance has an SNR equal to or lower than 30 dB, and the count values ​​of the low-sensitivity channels T2 and T3 are used when the illuminance has an SNR higher than 30 dB.

[0193] In very low illumination ranges, the probability of photons being counted by low-sensitivity channels is very low, but not zero. Therefore, in very rare cases, pixels that are slightly brighter than their surroundings (bright spots) can occur. In such cases, the absolute value of the brightness of each pixel is small, so there is no major problem with image quality when viewing the image as a whole. However, image quality can be improved by using the methods described above.

[0194] [Example of a configuration for enabling the low-sensitivity channel of SPAD pixel 200] Figure 32The diagram illustrates specific means for Method 1 of improving low-light SNR and a circuit diagram of an example SPAD pixel 200 with a low-sensitivity channel enabled in a three-layer configuration. In this embodiment, an example is given where a high-sensitivity channel is enabled and exposure begins (low-sensitivity channel disabled). After a specific time period, if it is determined to be high-light based on the determination conditions described below, the low-sensitivity channel is enabled, and both the high-sensitivity and low-sensitivity channels continue photon counting. This time period can be, for example, until the count value of the in-pixel counter 323 in the third layer reaches a predetermined threshold. Figure 32 This illustrates an example where the specific time period is until all high-order bits of counter 323 within the third-layer pixel become HI. Furthermore, Figure 33 It shows Figure 32 The timing diagram shows an example operation of the pixel circuit 220 of the SPAD pixel 200 shown.

[0195] The pixel circuitry on the second layer may include a charging circuit 221, a first gating circuit 260, an in-pixel counter 223, a lock flag circuit 226, and a second gating circuit 227. The first gating circuit 260 may include AND and NOR elements, receive the cathode node potential Vc, the CHK_HS signal, and the OUT2 signal as inputs, and output the OUT signal to the in-pixel counter 223. The CHK_HS signal may be a signal used to check the high-sensitivity channel and may be a valid LO signal. The lock flag circuit 226 generates a lock signal to lock the low-sensitivity channel. The second gating circuit 227 may include NAND elements, receive the lock signal (LOCK) and the CHK_LS signal as inputs, and output the OUT2 signal. The CHK_LS signal may be a signal used to check the low-sensitivity channel and may be a valid HI signal.

[0196] Furthermore, the pixel circuitry on the third layer may include an in-pixel counter 323, a threshold determination circuit 324, and a SEL 325. When the value of the in-pixel counter 323 reaches a predetermined threshold, the threshold determination circuit 324 may set the overflow (OV) signal to HI.

[0197] like Figure 33As shown, in the threshold determination circuit 324, when all high bits of the third-layer pixel counter 323 become HI (from low illumination to high illumination), the OV signal can become HI. The logic state of the OV signal can be written to the lock flag DFF of the lock flag circuit 226 located on the second layer via the FBK signal, which becomes valid at the same time as the charging signal, and can be output as a lock signal. When the lock signal is LO, the lock signal indicates that the low-sensitivity channel is locked (i.e., the low-sensitivity channel is invalid), and when the lock signal is HI, the low-sensitivity channel is unlocked (i.e., the low-sensitivity channel is valid).

[0198] When the lock is released, the gating operation of the second gating circuit 227 is enabled by the HI active signal of CHK_LS. The lock flag can initially be initialized to the LO level by the RSTB signal and can be updated by the charging signal in each charging cycle. Thereafter, the gating operation of the first gating circuit 260 can be performed using all sensitivity channels from high sensitivity to low sensitivity.

[0199] When the count value of the in-pixel counter 323 reaches the predetermined threshold (in Figure 32 When all high bits in the example are HI, the OV signal can be changed to HI. This signal can be sent from the third layer to the second layer via the FBK signal and can be periodically received by the lock flag circuit 226. The lock flag, which has been written to the HI level and released once, is always kept in the unlocked state by the recovery cycle of the OR element.

[0200] For sensitivity channels T2 and T3, in addition to sensitivity channel T1, the lock flag can be initially set to the lock state via the RSTB signal. Furthermore, the HLD signal can be used to receive and maintain the output of the second layer before the latch circuitry to prevent malfunctions due to discontinuities in the input state when an OV signal is fed back from the third layer. Additionally, in Figure 33 The example shown illustrates a three-tier configuration, but it can also be applied to two-tier or other tier configurations.

[0201] Thus, the locking flag circuit 226 and the threshold determination circuit 324 can be positioned for each pixel 200, and the locking flag circuit 226 and the threshold determination circuit 324 can be used to determine (select) the sensitivity channel T for photon detection for each pixel 200. iThe sensitivity determiner 230 can disable or enable predetermined low-sensitivity channels. The sensitivity determiner 230 can continue to enable or disable the low-sensitivity channels for each pixel 200 based on the counting state of the in-pixel counter 323. Furthermore, the configuration of the threshold determination circuit 324 is not limited to two values ​​for whether or not overflow occurs, and as an application example, it is also possible to use multiple values ​​to enable multiple existing low-sensitivity channels in multiple levels.

[0202] [Example of modification to the first gating circuit] Figure 34 It shows the use of and Figure 32 The example logic circuits are the circuit diagrams of the first gating circuits of different logic circuits. Figure 34 In one example shown, with Figure 32 Unlike the example in the example, CHK_LS is the LO active signal. The first gating circuit 260 may include a first NOR element 261, a second NOR element 262, and an OR element 263. The first NOR element 261 may receive the cathode node potential Vc and the CHK_LS signal as inputs and may output the OUT_LS signal. The second NOR element 262 may receive the cathode node potential Vc and the CHK_HS signal as inputs and may output the OUT_HS signal. The OR element 263 may receive the OUT_LS signal and the OUT_HS signal as inputs and may output the OUT signal. According to the configuration of this modified example, the number of stages of the circuit configuration for the CHK signals of the high-sensitivity channel and the low-sensitivity channel can be adjusted, and the signal delay caused by the circuit configuration can be adjusted. Therefore, it is feasible to generate a time that achieves a time aperture ratio with high accuracy.

[0203] [Example of pixel circuit configuration for adaptively determining the use of low-sensitivity channels] Figure 35 The diagram illustrates another specific means of method 1 for improving low-light SNR and a circuit diagram of an example SPAD pixel 200 configured in two layers with a function for determining whether to adaptively use a low-sensitivity channel.

[0204] The pixel circuitry on the second layer may include a charging circuit 221, a first gating circuit 260, an in-pixel counter 223, an adaptive determination circuit 270, a locking flag circuit 226, a second gating circuit 227, etc. The configurations of the charging circuit 221, the first gating circuit 260, and the in-pixel counter 223 can be the same as described above, therefore their descriptions will be omitted.

[0205] The adaptive determination circuit 270 can be a circuit that adaptively determines whether to use a low-sensitivity channel based on the trend of the incident photon spacing. The adaptive determination circuit 270 may include a threshold determination circuit 271, a state memory 272, and circuitry 273 associated with the state memory.

[0206] Threshold determination circuit 271 determines whether the count value of in-pixel counter 223 matches a predetermined threshold. State memory 272 may be a memory used to maintain the state of the rate at which photons are incident on pixel 200 (also simply referred to as the "incident rate") after the start of exposure (e.g., the state of the photon count rate via in-pixel counter 223). The number of bits in state memory 272 may be appropriately set to the desired number of bits.

[0207] The adaptive determination circuit 270 can determine the incident rate of photons from the start of exposure. More specifically, the adaptive determination circuit 270 can determine that the light radiation on pixel 200 is high illumination when the incident rate of photons is higher than a certain rate (high photon rate), and can immediately enable the low sensitivity channel. In this case, optical shot noise becomes dominant, so artifacts caused by the temporal aperture imaging algorithm may not be a problem, and a good high illumination image can be obtained.

[0208] Furthermore, when the adaptive determination circuit 270 determines that the incident rate of photons is low (e.g., below or equal to a specific rate) (low photon rate), the adaptive determination circuit 270 can determine that the light radiation on pixel 200 is low illumination, and can disable the low sensitivity channel and enable the high sensitivity channel. Therefore, artifacts caused by the temporal aperture imaging algorithm can be avoided, and good low-light images can be obtained.

[0209] Furthermore, adaptive determination by adaptive determination circuit 270 can be performed at any time during the exposure period, but it is most efficient when performed in a short period immediately following the start of the exposure. Adaptive determination circuit 270 can be enabled by the EN_APD signal.

[0210] In this way, the adaptive determination circuit 270 can determine the incident rate of photons and estimate the amount of incident light on pixel 200 based on the determination of the incident rate. The adaptive determination circuit 270 can make the determination based on the counting rate of the incident photons, and it is not limited to the maximum value that can be represented by the number of bits of the in-pixel counter 223. Furthermore, the desired sensitivity channel can be selected for each pixel 200, so it is feasible even if the number of bits of the in-pixel counter 223 is small.

[0211] The locking flag circuit 226 and the adaptive determination circuit 270 can be positioned for each pixel 200, and the locking flag circuit 226 and the adaptive determination circuit 270 can be used to determine (select) the sensitivity channel T for photon detection for each pixel 200. i The sensitivity determiner 230 can disable or enable a predetermined low-sensitivity channel. The sensitivity determiner 230 can calculate the incident rate of photons based on the count value of the in-pixel counter 223, and can continue to enable or disable the low-sensitivity channel based on the calculated incident rate of photons.

[0212] <Example of determining the amount of incident light based on the incident rate of photons> Figure 36 A timing diagram is shown to describe a method for determining the amount of incident light based on the incident rate of photons.

[0213] When high-intensity light is incident on pixels 200 within the pixel array 120, the arrival time interval of photons incident on pixels 200 can be short. For example, photons can be incident on pixels 200 almost continuously without interruption. In this case, it can be predicted that the final count value of the in-pixel counter 223 will increase. When photons are incident on the pixel without interruption, the adaptive determination circuit 270 can determine that the photon rate is high. When the condition for a high photon rate is met, the adaptive determination circuit 270 can determine a high photon rate. The condition for a high photon rate can be, for example, the state in which photons are incident on the pixel without interruption (e.g., when the count value of the in-pixel counter 223 is 4 or greater three times consecutively). Figure 36 The values ​​of the in-pixel counter in the past three consecutive charges are shown to be 6, 7, and 7. When the adaptive determination circuit 270 determines a high photon rate, it can set the HRT signal to the HI level and enable the low-sensitivity channel. The HRT signal can be a signal indicating a high photon rate.

[0214] In addition, when low-illuminance light is incident on pixel 200, the intervals between the arrival times of photons can be long. For example, photons can be incident on pixel 200 intermittently. In this case, it can be expected that the final count value of the in-pixel counter 223 is very small. When the condition of a high photon rate is not satisfied, the adaptive determination circuit 270 can determine a low photon rate. In addition, the adaptive determination circuit 270 can be configured to determine a low photon rate when photons are incident on pixel 200 intermittently. The adaptive determination circuit 270 can determine a low photon rate in a state where photons are incident on pixel 200 intermittently (for example, when the count value of the in-pixel counter 223 is 0 or 1 during one charging period). When the adaptive determination circuit 270 determines a low photon rate, the adaptive determination circuit 270 can set the HRT signal to the LO level and can disable the low-sensitivity channel. In addition, these examples are not limited to the two levels of high photon rate and low photon rate, and as an application example, a multi-stage photon rate adaptive determination circuit can be installed, and multiple existing low-sensitivity channels can be enabled in multiple stages.

[0215] <Modified example of threshold determination circuit for enabling low-sensitivity channel in SPAD pixel 200> Figure 37 Another specific means of Method 1 for improving low-illuminance SNR and a circuit diagram showing a modified example of the threshold determination circuit for enabling the low-sensitivity channel in SPAD pixel 200 are shown. The data accumulator 240 can include a column processor 241, a frame memory 242, and a threshold determination circuit 243.

[0216] In order to perform threshold determination using the count value of the in-pixel counter 223, a specific number of bits of the in-pixel counter 223 may be required, so there is a possibility that the pixel size of SPAD pixel 200 will increase. As described above, the pixel data output from the pixel circuit 220 of SPAD pixel 200 can be accumulated in the frame memory 242 by the "column processor 241 placed immediately before the frame memory 242 outside the pixel array unit 120". Figure 37 A threshold determination circuit 243 for determining whether the accumulation result by the column processor 241 is greater than or equal to a threshold (for example, 256 counts) is shown. When the accumulation result by the column processor 241 is greater than or equal to the threshold (for example, 256 counts), the lock is released by clearing the lock flag of the lock flag circuit 226 via the feedback bus 291.

[0217] In this way, by including the threshold determination circuit 243 together with the column processor 241 outside the pixel array unit 120, a threshold of any large value can be used for comparison with the value after accumulation of pixel data.

[0218] The locking flag circuit 226 and the data accumulator 240 can be used to determine (select) the sensitivity channel T for photon detection for each pixel 200. i The sensitivity determiner can continue to enable or disable a predetermined low-sensitivity channel. The sensitivity determiner can enable or disable a predetermined low-sensitivity channel based on a summed value obtained by summing the count values ​​of the in-pixel counter 223 during multiple unit exposure times.

[0219] <Separating high-sensitivity and low-sensitivity count values ​​from pixel data> Figure 38 The diagram illustrates specific means of Method 2 for improving SNR in low light conditions and a block diagram showing an example configuration for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data. Figure 39 A schematic diagram illustrating a method for separating high-sensitivity count values ​​and low-sensitivity count values ​​from pixel data is shown. Figure 40 A schematic diagram illustrating an example relationship between pixel values, high-sensitivity count values, and low-sensitivity count values ​​is shown.

[0220] The count value obtained through the high-sensitivity channel T1 can be called the high-sensitivity count value, and the count values ​​obtained through the low-sensitivity channels T2 and T3 can be called the low-sensitivity count values. For example... Figure 38 As shown, the photoelectric conversion device 100 may include a data accumulator 280, a frame memory 281 for high sensitivity (high-sensitivity frame memory) and a frame memory 282 for low sensitivity (low-sensitivity frame memory), a first converter 283, a second converter 284, a first column processor 285, and a second column processor 286. The first converter 283 and the second converter 284 may correspond to the frame memory 281 for high sensitivity and the frame memory 282 for low sensitivity, respectively. In one example, the high-sensitivity frame memory 281 may be used to maintain a high-sensitivity count value, and the low-sensitivity frame memory 282 may be used to maintain a low-sensitivity count value.

[0221] As described above, the pixel circuit 220 can output data obtained by synthesizing the "count values ​​of photons detected through the high-sensitivity channel T1" and the "count values ​​of photons detected through the low-sensitivity channels T2 and T3" via a temporal aperture imaging algorithm. For high sensitivity and low sensitivity respectively, the converters can convert the image data values ​​(pixel values) into high-sensitivity count values ​​and low-sensitivity count values ​​for output. In one example, the first converter 283 can convert the pixel data output by each pixel into a high-sensitivity count value, and the second converter 284 can convert the pixel data output by each pixel into a low-sensitivity count value. The first column processor 285 and the second column processor 286 can accumulate the outputs of the first converter 283 and the second converter 284 respectively, and can store them in the high-sensitivity frame memory 281 and the low-sensitivity frame memory 282 respectively. Therefore, the pixel data compressed within a pixel is expanded by the converters.

[0222] For example, suppose the count value of the 3-bit in-pixel counter 223 is output from the pixel circuit 220. Figure 39 As shown, the high-sensitivity count value and the low-sensitivity count value can be calculated as follows.

[0223] [High-sensitivity count value].

[0224] Because the high-sensitivity channel T1 is counted when the count value of the in-pixel counter 223 is ≥1, the first converter 283 can output a high-sensitivity count value of 1. The first column processor 285 can add the high-sensitivity count value of 1 to the value read from the high-sensitivity frame memory 281 to write it into the high-sensitivity frame memory 281. That is, the value of the high-sensitivity frame memory 281 can be incremented. Furthermore, when the count value of the in-pixel counter 223 is 0, the first converter 283 can output a high-sensitivity count value of 0. The first column processor 285 can add the high-sensitivity count value of 0 to the value read from the high-sensitivity frame memory 281 to write it into the high-sensitivity frame memory 281 (i.e., no operation is performed on the high-sensitivity frame memory 281).

[0225] [Low sensitivity count value] When the count value of the in-pixel counter 223 is 1 or greater, the second converter 284 can set the "count value of the in-pixel counter 223" - "1" (the value obtained by subtracting the high-sensitivity count value of the high-sensitivity channel T1) as the low-sensitivity count value. The second column processor 286 can add the count value of the in-pixel counter 223 - 1 to the value read from the low-sensitivity frame memory 282 to write it into the low-sensitivity frame memory 282. Furthermore, when the count value of the in-pixel counter 223 is 0, the second converter 284 can output a low-sensitivity count value of 0. The second column processor 286 can add the low-sensitivity count value of 0 to the value read from the low-sensitivity frame memory 282 to write it into the low-sensitivity frame memory 282 (i.e., no operation is performed on the low-sensitivity frame memory 282).

[0226] Thus, by separating high-sensitivity and low-sensitivity count values ​​from pixel data and concatenating the transfer function obtained from each of them with an arbitrary threshold, a synthetic transfer function unaffected by artifacts caused by the temporal aperture imaging algorithm can be easily achieved. Furthermore, additional image signal processing (such as linearization) can be efficiently implemented.

[0227] As described above, a photoelectric conversion device according to embodiments of the present disclosure has been described. However, those skilled in the art can appropriately add, modify, and omit aspects of the present disclosure within the scope of its technical concept.

[0228] For example, a configuration from another embodiment may be added to the configuration in one of the first to fifth embodiments described above, or a configuration from one embodiment may be deleted.

[0229] Furthermore, in the first to fifth embodiments described above, a portion of the components arranged inside the pixel 200 may be arranged outside the pixel 200, or a portion of the components arranged outside the pixel 200 may be arranged inside the pixel 200.

Claims

1. A photoelectric conversion device, the photoelectric conversion device comprising an avalanche photodiode in each pixel of a pixel array, the avalanche photodiode comprising an anode and a cathode, the photoelectric conversion device comprising: The charger is configured to charge the anode or cathode once per unit exposure time. The gating circuit is configured to generate pulse signals based on the output of the avalanche photodiode at multiple different times within a unit exposure time; as well as The counter is configured to count pulse signals from the gating circuit.

2. The photoelectric conversion device as described in claim 1, wherein, The gating circuit is also configured to receive the output signal of the avalanche photodiode and determination signals at different times to generate a pulse signal.

3. The photoelectric conversion device as described in claim 1, further comprising: A normalization processor is configured to calculate, by summing the counts of counters over multiple unit exposure times and normalizing the sum, the number of photons contributing to the occurrence of the avalanche operation in a predetermined frame.

4. The photoelectric conversion device as described in claim 3, further comprising: The linearization processor is configured to perform linearization on a nonlinear range of the number of photons calculated by the normalization processor.

5. The photoelectric conversion device as described in claim 3, further comprising: The computing processor is configured to generate a first intermediate image by summing the count values ​​of counters during the plurality of unit exposure times, and to generate a second intermediate image by performing motion correction processing on the generated first intermediate image.

6. The photoelectric conversion device as described in claim 1, further comprising: Multiple virtual sensitivity channels have different durations within a unit exposure time, wherein the multiple virtual sensitivity channels are configured such that one of the multiple virtual sensitivity channels includes another of the multiple virtual sensitivity channels on the time axis.

7. The photoelectric conversion device as described in claim 6, wherein, The multiple different times indicate the end of the multiple virtual sensitivity channels.

8. The photoelectric conversion device as described in claim 6, wherein, The multiple virtual sensitivity channels start at the same time.

9. The photoelectric conversion device as claimed in claim 1, further comprising: The latch circuit is configured to maintain the count value of the counter within the pixel.

10. The photoelectric conversion device as claimed in claim 1, further comprising: First base; And a second substrate, stacked on top of the first substrate, The avalanche photodiode is disposed on the first substrate, and the charger, gating circuit and counter are disposed on the second substrate.

11. The photoelectric conversion device as claimed in claim 10, further comprising: A third substrate stacked on top of the second substrate. The upper counter, which is connected in series with the counter, is set on the third base.

12. The photoelectric conversion device as claimed in claim 1, further comprising: The frame memory is configured to maintain a summed value obtained by adding the count values ​​of counters over multiple unit exposure times.

13. The photoelectric conversion device as claimed in claim 12, wherein, The frame memory includes: a high-sensitivity frame memory configured to maintain a high-sensitivity count value acquired through a first virtual sensitivity channel with predetermined high sensitivity; and a low-sensitivity frame memory configured to maintain a low-sensitivity count value acquired through a second virtual sensitivity channel with predetermined low sensitivity.

14. The photoelectric conversion device as claimed in claim 13, wherein, The first virtual sensitivity channel with predetermined high sensitivity is the virtual sensitivity channel with the highest sensitivity among multiple virtual sensitivity channels.

15. The photoelectric conversion device as claimed in claim 13, further comprising: The first converter is configured to convert pixel data output from each pixel into a high-sensitivity count value; And a second converter, configured to convert pixel data output from each pixel into low-sensitivity count values.

16. The photoelectric conversion device as claimed in claim 1, wherein, The gating circuit is also configured to generate a pulse signal based on the output of the avalanche photodiode at at least one time selected from the plurality of different times.

17. The photoelectric conversion device as claimed in claim 1, wherein, The gating circuit is also configured to generate a pulse signal based on the output of the avalanche photodiode at the time corresponding to the end of the virtual sensitivity channel with the highest sensitivity among the plurality of different times.

18. The photoelectric conversion device as claimed in claim 1, further comprising: The sensitivity determiner is configured to enable or disable a predefined low-sensitivity virtual sensitivity channel for each pixel.

19. The photoelectric conversion device as claimed in claim 18, further comprising: The upper counter is connected in series with the counter. The sensitivity determiner is also configured to enable or disable a predetermined low-sensitivity virtual sensitivity channel based on the counting state of the upper counter.

20. The photoelectric conversion device as claimed in claim 18, wherein, The sensitivity determiner is also configured to: determine the incident rate of the photon based on the count value of the counter, and enable or disable a predetermined low-sensitivity virtual sensitivity channel based on the determination result of the incident rate of the photon.

21. The photoelectric conversion device as claimed in claim 18, wherein, The sensitivity determiner is also configured to enable or disable a predetermined low-sensitivity virtual sensitivity channel based on a summed value obtained by adding the count values ​​of counters over multiple unit exposure times.

Images