Image sensing device

Abstract

This invention provides an image sensing device that can detect the distance to a target object using the TOF (time-of-flight) method. [Solution] The image sensing device 100 includes a pixel 100 that generates a pulse signal based on photons reflected from a target object TO, a TDC block 110 that includes a plurality of TDCs (time-to-digital converters) each that generate a digital code corresponding to the time delay between the pulse signal and a reference pulse, and a histogram circuit 120 that generates index information and update values ​​for a memory area based on the digital code, wherein the plurality of TDCs may have different activation timings.

Description

Technical Field

[0001] The present disclosure relates to an image sensing device capable of detecting the distance to an object by the TOF (time-of-flight) method.

Background Art

[0002] In recent years, the TOF (Time of Flight) technology that has been in the spotlight irradiates pulsed light from a light source arranged inside or near the sensor, then receives the reflected light and measures the time therebetween, and extracts the distance based on the principle of the constancy of the speed of light. In order to precisely measure TOF, since a reaction must occur immediately when the light reaches the light receiving element, a very sensitive photoelectric conversion element is required. For this reason, research on single-photon avalanche diodes (SPADs) that can be fabricated by CMOS process technology has been actively conducted.

[0003] On the other hand, a lidar sensor can detect an object around the user and identify the distance between the object and the user, so that an accident that the user cannot recognize can be prevented in advance, and furthermore, autonomous driving of various electronic devices can be enabled. The TOF technology can be used to identify the distance to an object in such a lidar sensor.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Embodiments of the present invention provide an image sensing device using the TOF (time-of-flight) method that can improve the operating speed and reduce the memory capacity.

Means for Solving the Problems

[0005] An image sensing device according to an embodiment of the present invention includes a pixel that generates a pulse signal based on photons reflected from a target object, a TDC block that includes a plurality of TDCs (time-to-digital converters) each that generates a digital code corresponding to the time delay between the pulse signal and a reference pulse, and a histogram circuit that generates index information and update values ​​for a memory area based on the digital code, wherein the plurality of TDCs may have different activation timings.

[0006] An image sensing device according to another embodiment of the present invention includes: a pixel that generates a pulse signal based on photons reflected from a target object; a TDC block that generates a digital code corresponding to the time delay between the pulse signal and a reference pulse; and a histogram circuit that generates a first feature function and a second feature function having waveforms corresponding to the order of a spline function in the timestamp interval to which the photons respond, and generates information and updated values ​​for a memory area using the first feature function and the second feature function, wherein each of the first feature function and the second feature function is discretized with respect to the timestamp value and the number of detected photons, and has a stepwise structure with different peak values ​​for each step. [Effects of the Invention]

[0007] Embodiments of the present invention can provide the effect of improving the operating speed and reducing the memory capacity required for operation in an image sensing device using the TOF (time-of-flight) method.

[0008] Furthermore, the embodiments of the present invention are illustrative, and those skilled in the art will be able to make various modifications, changes, substitutions, and additions through the technical concept and scope of the appended claims, and such modifications and changes should be considered to fall within the scope of the following claims. [Brief explanation of the drawing]

[0009] [Figure 1]This figure shows a lidar system according to one embodiment of the present invention. [Figure 2] This diagram illustrates the operation of the image sensing device shown in Figure 1. [Figure 3] This figure illustrates the histogram corresponding to the photon count value in the histogram circuit shown in Figure 1. [Figure 4] This figure illustrates the feature function in the image sensing device shown in Figure 1. [Figure 5a] This figure illustrates an embodiment of histogram operation in the image sensing device shown in Figure 1. [Figure 5b] This figure illustrates an embodiment of histogram operation in the image sensing device shown in Figure 1. [Figure 6] This is a configuration diagram relating to an image sensing device according to another embodiment of the present invention. [Figure 7] This is an embodiment illustrating the histogram operation in the image sensing device shown in Figure 6. [Figure 8] This is an embodiment illustrating the histogram operation in the image sensing device shown in Figure 6. [Figure 9] This is an embodiment illustrating the histogram operation in the image sensing device shown in Figure 6. [Figure 10] This is an embodiment illustrating the histogram operation in the image sensing device shown in Figure 6. [Figure 11] This is a configuration diagram relating to an image sensing device according to another embodiment of the present invention. [Figure 12] This is an embodiment illustrating the histogram operation in the image sensing device shown in Figure 11. [Figure 13] This is an embodiment illustrating the histogram operation in the image sensing device shown in Figure 11. [Figure 14]This is an embodiment for explaining the histogram operation in the image sensing device shown in FIG. 11. [Figure 15] This is an embodiment for explaining the histogram operation in the image sensing device shown in FIG. 11. [Figure 16] This is a configuration diagram of an image sensing device according to another embodiment of the present invention. [Figure 17] This is an embodiment for explaining the histogram operation in the image sensing device shown in FIG. 16. [Figure 18] This is an embodiment for explaining the histogram operation in the image sensing device shown in FIG. 16. [Figure 19] This is an operation timing diagram of the control signal generation unit in FIG. 6. [Figure 20] This is a configuration diagram of a photographing device including an image sensing device according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure can provide various effects that can be directly or indirectly recognized by the present disclosure.

[0011] FIG. 1 is a diagram showing a lidar system according to an embodiment of the present invention. Referring to FIG. 1, the lidar system 1 can include an image sensing device 10, a communication interface 20, and a host 30.

[0012] Here, the image sensing device 10 may include a pixel 100, a time-to-digital converter (TDC) block 110, a histogramming circuit 120, and a signal processor 130. For example, the image sensing device 10 can measure the time it takes for light to reflect back from a target object using a time-correlated single-photon counting (TCSPC) method.

[0013] Pixel 100 may include a single-photon avalanche diode (SPAD) element (hereinafter referred to as "SPAD element") 101, a quenching circuit 102, and a buffer 103. Pixel 100 may include at least one SPAD element 101 and may be grouped into macro pixels. Pixel 100 may be a single macro pixel, or multiple macro pixels may be arranged in an array.

[0014] The SPAD element 101 can detect a single photon of reflected light RL reflected by an object and generate a voltage pulse corresponding to the detected single photon. As a photodiode including a photosensitive PN junction, the SPAD element 101 can generate a voltage pulse by triggering avalanche breakdown in Geiger mode, where a reverse bias voltage higher than the breakdown voltage is applied to the cathode-anode voltage. Here, Geiger mode can mean applying a reverse bias voltage greater than the breakdown voltage in order to detect a single photon with the SPAD element 101. In Geiger mode, because the electric field strength applied to the amplification layer is large, even if a small number of photons are absorbed, the avalanche current breakdown phenomenon occurs, and a large current pulse is output, making it possible to detect a single photon. Thus, the process in which avalanche breakdown is triggered by a single photon and a voltage pulse is generated will be defined as the avalanche process.

[0015] One terminal (anode) of the SPAD element 101 can be connected to the ground voltage terminal. The other terminal (cathode) of the SPAD element 101 can be connected to the sensing node SN. The SPAD element 101 can detect a single photon and output a current pulse generated to the sensing node SN.

[0016] In the embodiments of the present invention, a SPAD element 101 was given as an example of the light-receiving element (photodetector) of pixel 100, but it is not limited to the SPAD element 101. That is, in addition to the SPAD element 101, various elements that operate in Geiger mode, such as APDs (avalanche photodiodes) and SiPMs (silicon photomultiplier tubes), can be used as the light-receiving element of pixel 100.

[0017] The quenching circuit 102 can control the voltage of the SPAD element 101 and output it to the sensing node SN. After a voltage pulse is generated due to avalanche breakdown and the voltage of the sensing node SN changes, the quenching circuit 102 can perform a quenching operation that returns the voltage of the sensing node SN to Geiger mode.

[0018] The quenching circuit 102 can be connected between the sensing node SN and the ground voltage terminal. For example, if the quenching circuit 102 is implemented with active elements, it can be implemented with a transistor. Alternatively, if the quenching circuit 102 is implemented with passive elements, it can be implemented with a resistor.

[0019] Buffer 103 can generate a pulse signal based on an electrical signal generated in response to a photon incident on pixel 100 and output a pixel signal PX_OUT. Buffer 103 can generate pulse signals at a frequency corresponding to the photon reception frequency. Buffer 103 can sample analog voltage pulses generated at sensing node SN and convert them into digital pulse signals (i.e., SPAD pulses). Here, the sampling method may be a method that converts the voltage pulse level into a pulse signal having a logic level of 0 or 1 depending on whether the level of the voltage pulse is above a threshold level, but the scope of the present invention is not limited thereto.

[0020] The TDC block 110 can calculate the time delay between the pixel signal PX_OUT output from the pixel 100 and the reference pulse of the illumination light, convert it into a digital value corresponding to the time delay, and generate TDC data TDC_OUT. In this disclosure, the pulse signal of the pixel signal PX_OUT generated from the pixel 100 can be referred to as a SPAD pulse. According to the embodiment, the TDC block 110 can include a plurality of TDCs.

[0021] The histogram circuit 120 can generate a histogram based on the TDC data TDC_OUT. The histogram circuit 120 can accumulate and store timestamp data in time bins based on the TDC data TDC_OUT and determine the bin with the peak value. According to one embodiment, the histogram circuit 120 may be contained within the pixel 100. The operation of such a histogram circuit 120 will be described in more detail in the embodiments described later.

[0022] The signal processing unit 130 can determine the distance to the target object by calculating the time of flight (TOF) based on the histogram data received from the histogram circuit 120. The signal processing unit 130 can be implemented, for example, by a general-purpose digital signal processor including a combinational circuit connected to memory, a processor, or a controller. Alternatively, the signal processing unit 130 may be implemented by a custom application-specific integrated circuit (ASIC), and the type of signal processing unit is not limited.

[0023] The data regarding the time of flight of light calculated by the signal processing unit 130 can be transmitted to the host 30 via the communication interface 20. For example, the communication interface 20 may be a serial communication interface. As an example, the host 30 may provide a three-dimensional distance image of the target object via some interface, such as a display screen or a user interface, based on the data provided via the communication interface 20.

[0024] Figure 2 is a diagram illustrating the operation of the image sensing device shown in Figure 1. And Figure 3 is a diagram illustrating the histogram corresponding to the photon count value in the histogram circuit shown in Figure 1.

[0025] Referring to Figure 2, the image sensing device 10 emits pulsed light (irradiation light EL) from a light source, receives the light reflected from the target object (reflected light RL), measures the time between these two emitting lights, and extracts the distance based on the principle of the constancy of the speed of light. Here, the irradiation light EL can be projected onto the target object in the form of dots via the image sensing device 10. Depending on the distance to the target object, the position of the projected dots may change, or the intensity of the irradiation light may increase or decrease. According to this embodiment, the image sensing device 10 can operate in a photon counting mode to find the position of the dots at the beginning of its operation.

[0026] The SPAD element 101 can detect a single photon of reflected light RL reflected by the target object and generate a SPAD pulse DP corresponding to the detected single photon. The TDC block 110 can measure, in synchronization with the illumination light EL, the time from the time the light is irradiated until any one of the SPAD elements 101 within the pixel 100 responds. Here, the measurement range TR1 for detecting photons can be divided into a finite number of time bins. According to an embodiment of this disclosure, the measurement range TR1 can be divided into eight time bins.

[0027] Referring to Figure 3, the horizontal axis of the histogram can represent time bins that can show each subrange of the photon arrival time. For example, the histogram circuit 120 can contain the same number of memory regions as time bins, and each memory region can maintain an independent value. This allows each index value of a memory region to correspond to a discretized observation time.

[0028] For example, the histogram circuit 120 may have eight memory regions. The memory regions of the histogram circuit 120 can be represented by memory indices 1 to 8, corresponding to the codes of the TDC data TDC_OUT (hereinafter referred to as "TDC codes" or "digital codes").

[0029] The vertical axis of the histogram shows the photon count. The counter in the initial time bin is relatively low and can be attributed to background noise P1. Then, reflected pulses P2 with peak values ​​can be detected from some points in the time bin.

[0030] The histogram circuit 120 can store photon count values ​​accumulated in multiple time bins. The histogram circuit 120 can obtain distance information based on the time bin with the largest hit count value among the multiple time bins (for example, memory index "3").

[0031] In the measurement range TR1 where the first illuminating light EL emits light, the SPAD pulse DP1 can respond at the time corresponding to the TDC code "3". In this case, the value stored in memory area "3" can be increased by +1. Then, in the measurement range TR2 where the second illuminating light EL emits light, the SPAD pulse DP2 can respond at the time corresponding to the TDC code "5". In this case, the value stored in memory area "5" can be increased by +1. In this way, the transmission of the illuminating light EL, the detection of SPAD pulses corresponding to the reflected light RL, and the updating of the histogram are repeatedly performed, and a histogram like the one shown in Figure 3 can be generated.

[0032] In the example histogram shown in Figure 3, it can be seen that the photon count value stored in memory area "3" is the largest, and is larger than that of non-adjacent memory areas (for example, memory area "7"). From this, the signal processing unit 130 can determine that the image sensing device 10 is receiving reflected light RL from the desired target object in the time domain corresponding to memory area "3".

[0033] In the embodiments shown in Figures 2 and 3, the SPAD element 101 may not respond at the time of receiving the photon from the reflected light RL. Alternatively, the SPAD element 101 may respond at a time other than the time of receiving the photon from the reflected light RL. For example, it can be seen that in the measurement range TR2, the SPAD pulse does not respond for TDC code "3" corresponding to the reflected light RL, while the SPAD pulse DP2 responds for TDC code "5".

[0034] In other words, it is not guaranteed that photons from reflected light RL entered the SPAD element 101 within the measurement range TR2. Furthermore, even if we assume that photons entered the SPAD element 101, the probability that the SPAD element 101 will respond (photon detection efficiency, PDE) is not 100%. That is, unless the measurement is performed in a dark place, if photons of a sensitive wavelength are present, noise from a light source other than the illumination light EL (background light) may be present. Moreover, even if no photons enter the SPAD element 101, a response due to noise (Dark Count) may occur with a certain probability.

[0035] Therefore, it can be difficult to measure the desired time with a single response. Consequently, by projecting an EL light source and repeatedly measuring the time it takes for the SPAD pulse response to be detected, the difference between the projection time and the frequently occurring time range relative to the projection time can be measured as the round-trip time of the light.

[0036] The image sensing device 10 described above requires a considerable amount of memory to store a histogram for each of the 100 pixels. Therefore, in order to reduce the memory capacity, a partial histogramming method or a sketched LiDAR method can be used.

[0037] Here, the partial histogram method is a method that generates a histogram in stages by dividing the measurement into several steps, rather than generating the histogram all at once. The partial histogram method generates a coarse histogram by observing the entire measurement range at coarse time intervals, detects the peak indicating the maximum value from the coarse histogram, and generates a fine histogram with a fine time window by zooming in on the detected peak.

[0038] However, with the partial histogram method, the frame rate may decrease because the number of measurements increases. Also, because the range observed in fine measurements must be wider than the time range corresponding to the two bins of course measurements, temporal resolution may be sacrificed. Furthermore, the effect of reducing memory capacity may decrease as the number of measurements increases.

[0039] On the other hand, the sketch-drider method does not count the bin values ​​corresponding to the time when a photon is detected one by one, but rather calculates a feature function that has a time variation finer than the time interval of the bin, and then accumulates the calculated value in memory. Various functions can be applied as feature functions, but spline functions can be used. Such feature functions will be explained in more detail with reference to Figure 4 below.

[0040] Figure 4 is a diagram illustrating the feature function in the image sensing device shown in Figure 1. Referring to Figure 4, we can assume that the SPAD element 101 responded to the photon n times. In this case, the response time, i.e., the time stamp, is x. j (j=1, 2, ..., n) can be defined as follows. Here, each memory region of the histogram circuit 120 is defined by the values ​​(Z) of the spline sketch defined by [Equation 1] and [Equation 2] below. p、i ) can be maintained.

[0041]

number

[0042] In the above [Equation 1], Φ p、i (x j ) can be defined as the feature function (referred to as "FF" in the diagrams described later). Equation 1 can mean that a Discrete Fourier Transform is performed to calculate the feature function each time a time stamp is obtained, and the calculated value is stored in memory.

[0043]

number

[0044] In the above [Equation 2], Δ is obtained by dividing the total measurement time by M (where M is the number of sketches to be acquired). Here, i is i=1, ..., M, and can correspond to the observed statistical value (i.e., the number of sketches). This number of sketches M can correspond to the number of memory areas and can have the same meaning as the number of bins in the TCSPC method. p is the order of the spline function. Φp is the spline function of order p. Thus, the "feature function" can represent a function that sketches a sinusoidal waveform corresponding to the order of the spline function in the timestamp interval in which n photons respond.

[0045] For example, when the degree of the spline function is zero or first, it can be defined as shown in [Equation 3] and [Equation 4] below, respectively.

[0046]

number

[0047]

number

[0048] Figure 4(A) shows the feature function when the order p of the spline function is "0". In (A), the number of sketches (number of memory areas) M is "4", and four memory areas corresponding to feature 1 to feature 4 can accumulate these values. For example, we can assume that photon detection occurred in the time range of timestamps ξ0 and ξ1. Then, if the SPAD element 101 responds as shown by the arrow in Figure (A), the value stored in memory area "1" can be increased by +1. Therefore, when the order p of the spline function is "0", histogram operation can be performed in the same way as the operation of the TCSPC described in Figure 1.

[0049] Figure 4(B) shows the feature function when the order p of the spline function is "1". Unlike when the order p of the spline function is "0", the feature value added by a single photon detection is not two values, {0, 1}. We can assume that each memory area stores values ​​as floating-point numbers. Then, for example, if a photon is detected at the arrow point in Figure (B), the value stored in memory area 1 and the feature value stored in memory area 2 can simultaneously become +0.5 according to [Equation 1] described above.

[0050] As another example, maintaining floating-point values ​​for each memory area would increase the circuit size from a circuit implementation perspective, so it is also possible to maintain integer values. In this case, for example, the feature value on the vertical axis of diagram (B) could be increased (e.g., by 100 times), and the decimal part of the value could be rounded to obtain an integer value. In this case, if the value multiplied on the vertical axis is small, the measured value will have an error due to the effect of rounding error, so it is necessary to select an appropriate value for the allowable error.

[0051] Figures 5a and 5b illustrate embodiments of histogram operation in the image sensing device shown in Figure 1. Figure 5a shows an example of generating a histogram using the TCSPC method. For example, in the time domain corresponding to bin 4, the histogram can be generated in the same memory area regardless of when the reflected light RL photon detection occurs. Here, we can assume that the effects of noise are not considered.

[0052] In contrast, Figure 5b shows an example of generating a histogram using the spline sketch method. For example, we can assume that photon detection of reflected light RL occurs in the measurement interval TP. If photon detection of reflected light RL occurs at time T1, all feature values ​​of feature functions FF3 and FF4 can be generated. As a result, changes can be seen in the histogram not only for memory area 4 but also for the values ​​in memory area 3 adjacent to memory area 4. If photon detection of reflected light RL occurs at time T2, the feature value of feature function FF4 is generated, and changes can be seen in the histogram of memory area 4. If photon detection of reflected light RL occurs at time T3, all feature values ​​of feature functions FF4 and FF5 can be generated. As a result, changes can be seen in the histogram not only for memory area 4 but also for the values ​​in memory area 5 adjacent to memory area 4.

[0053] In other words, within the measurement interval TP, a change in the histogram is observed depending on whether the detection of reflected light RL is closer to time T1 or T3. In this case, after removing the value Cb due to background light (or dark count), the center value of the histogram can be determined for the three values ​​C3, C4, and C5 in memory areas 3, 4, and 5. This allows the position of the reflected light RL waveform to be determined at a finer level compared to the TCSPC method. Specifically, the central value Tc of the histogram can be calculated as shown in [Equation 5] below.

[0054]

number

[0055] Once the center value Tc of the reflected light RL is determined based on the time at which the irradiated light EL is generated and the response characteristics of the SPAD pulse, the time of flight (TOF) can be calculated. As shown in [Equation 5] above, the center value Tc of the histogram can have a finer resolution than Δ described in [Equation 2] above.

[0056] In other words, since the optical time of flight is averaged with adjacent memory regions after calculating the central value Tc using [Equation 5], the histogram method in Figure 5b can have finer time resolution with the same number of memory regions compared to the histogram method in Figure 5a. That is, when applying the method in Figure 5b, the number of memory regions required to obtain the desired time resolution can be reduced compared to Figure 5a.

[0057] However, the sketch-drider method requires the TDC to operate at high speed, which can lead to high power consumption. Furthermore, the required memory capacity per memory region increases, potentially increasing the total memory capacity of each memory region. Additionally, if the accuracy of the histogram is significantly limited by shot noise caused by background photons, the effectiveness of reducing the number of memory regions may also be limited. In other words, there is a limit to how much the number of memory regions can be reduced when noise increases. To improve these phenomena, the image sensing device described later can be realized.

[0058] Figure 6 is a configuration diagram relating to an image sensing device according to another embodiment of the present invention. Referring to Figure 6, the image sensing device 10_1 can include a pixel 200, a TDC block 210, a histogram circuit 220, a signal processing unit 230, and a control signal generation unit 240. In the image sensing device 10_1, redundant explanations of components similar to those in Figure 1 will be omitted, and only components that differ from those in Figure 1 will be specifically described.

[0059] Pixel 200 may include a SPAD element 201, a quenching circuit 202, and a buffer 203. The SPAD element 201 can detect a single photon of reflected light RL reflected by a target object and generate a voltage pulse corresponding to the detected single photon. The quenching circuit 202 can control the voltage of the SPAD element 201 and output it to the sensing node SN. The buffer 203 can generate a pulse signal based on the electrical signal generated in response to the photon incident on pixel 200 and output the pixel signal PX_OUT.

[0060] The TDC block 210 can calculate the time delay between the SPAD pulse output from the pixel 200 and the reference pulse of the illuminating light EL, and generate a digital code indicating the time delay, i.e., TDC data TDC_OUT.

[0061] The TDC block 210 can obtain the generation time of a reference pulse of the illuminating light EL from a timing controller (described later) that controls the light source driver, or it can consider a predetermined time (for example, a time a certain period of time before the start of the frame) as the generation time of the reference pulse. According to another embodiment, the TDC block 210 may obtain the generation time of the reference pulse from a control signal generation unit 240. According to one embodiment, the TDC block 210 may be contained inside the pixel 200.

[0062] In this disclosure, the TDC block 210 may include multiple TDCs. According to one embodiment, the number of multiple TDCs may correspond to the number of pixels 200, and is not limited thereto. The multiple TDCs may be controlled so that their operating timings differ from each other in accordance with a plurality of TDC control signals TDC_CNV1~8 supplied from the control signal generation unit 240. For example, a plurality of TDCs may be sequentially activated based on a plurality of TDC control signals TDC_CNV1~8 to generate a plurality of digital codes TDC_OUT0~7. As another example, the multiple TDCs may be controlled so that their operating timings differ from each other in accordance with a plurality of TDC control signals TDC_CNV1~4. For example, a plurality of TDCs may be sequentially activated based on a plurality of TDC control signals TDC_CNV1~4 to generate a plurality of digital codes TDC_OUT0~15. In this disclosure, the number of TDC control signals and the number of digital codes are not limited thereto.

[0063] Multiple TDC control signals TDC_CNV1~8 can have an activation level (e.g., a logic level of 1) when it is necessary to generate TDC data TDC_OUT for pixel 200, and a deactivation level (e.g., a logic level of 0) when it is not necessary to generate TDC data TDC_OUT for pixel 200. The operation of such a TDC block 210 will be described in more detail with reference to the drawings described later.

[0064] The histogram circuit 220 can generate a histogram based on the TDC data TDC_OUT. The histogram circuit 220 may include a control unit 221 and a histogram memory 222.

[0065] The control unit 221 can be activated based on the enable signal CALC_EN applied from the control signal generation unit 240. For example, if the enable signal CALC_EN is at an activation level (e.g., a logic level of 1), the control unit 221 can perform histogram operation based on the TDC data TDC_OUT. Conversely, if the enable signal CALC_EN is at an inactivation level (e.g., a logic level of 0), the control unit 221 can stop the histogram operation.

[0066] The control unit 221 can perform histogram operation by overlapping the value of the time bin corresponding to the time when a photon was detected with the value of the adjacent time bin using a spline function with fine time changes. The control unit 221 can discretize the sinusoidal feature function with respect to the time stamp value and the number of detected photons, and implement the feature function in a stepwise manner, sequentially increasing and then decreasing over multiple intervals. According to the embodiment, the feature function may be implemented in a stepwise manner, sequentially decreasing and then increasing over multiple intervals. The control unit 221 can set the time bin update (weight) value in each measurement interval in a manner in which the feature function of the adjacent measurement interval overlaps with the feature function of the said feature function.

[0067] The control unit 221 can generate information about a memory area (e.g., memory index information) and an update value for that memory area based on the TDC data TDC_OUT. For example, the control unit 221 can determine the memory area to be updated in accordance with the TDC code and generate an update value of +1 for that memory area. The memory area information and update value (calc) generated by the control unit 221 can be stored in the histogram memory 222.

[0068] The histogram memory 222 can update the value stored in the memory area by an update value (e.g., +1). According to one embodiment, the histogram memory 222 can include an integer counter. The histogram memory 222 can perform an update operation of the memory area by incrementing the integer counter once in response to the index information of the memory area and the update value +1. The value stored in the histogram memory 222 can be output to the signal processing unit 230 in fixed units (e.g., in frames).

[0069] The control signal generation unit 240 can generate multiple TDC control signals TDC_CNV1 to TDC_CNV8 to control the operation timing of the multiple TDCs. The multiple TDC control signals TDC_CNV1 to TDC_CNV8 can be activated at different timings. For example, the multiple TDC control signals TDC_CNV1 to TDC_CNV8 can be activated sequentially. The control signal generation unit 240 can also generate an enable signal CALC_EN to activate the operation of the control unit 221. For example, the control signal generation unit 240 can activate the enable signal CALC_EN when the transfer signal (TX_ON, described later) is activated at the time the illumination light EL is irradiated.

[0070] Figure 7 is a diagram illustrating the histogram operation in the image sensing device shown in Figure 6. The embodiment shown in Figure 7 illustrates an example in which the histogram circuit 220 performs histogram operation using the spline sketch method described in Figures 4 and 5 above.

[0071] Referring to Figure 7, it is shown that the order p of the spline function is "1" and the number of memory areas M is "8". The feature function can be denoted by FFi, where "i" may represent the index of the memory area. For example, there may be eight feature functions FF (FF1 to FF8), each with a different peak value set to a first-order spline function. The value shown on the vertical axis (val) represents the feature value of the feature function FF, and may be a value replaced by an 8-step stepped discrete function. As an example, the time width corresponding to one step of this stepped discrete function can be set to be smaller than the pulse width of the illuminating light EL.

[0072] The control unit 221 can be activated during the measurement range TR based on the enable signal CALC_EN. That is, the enable signal CALC_EN can transition to a logic high level (first level) when the first irradiation light EL is generated. If there is a response from the SPAD pulse, a strobe signal indicating that a response has occurred, as well as the TDC code, can be transmitted from the TDC block 210 to the control unit 221. Such a strobe signal can be included in the TDC data TDC_OUT.

[0073] Furthermore, the time resolution of the TDC data TDC_OUT can be set to 1 / 8 of the discrete function's stage, rather than one stage. That is, eight stages of each discrete function can correspond to one TDC code. The TDC block 210 can convert the response time of a SPAD pulse into a single TDC code with a time resolution corresponding to eight steps of each discrete function. The TDC block 210 can generate eight TDC codes (0 to 7) with a time resolution that divides the measurement time into eight parts.

[0074] The control signal generation unit 240 according to this disclosure can switch between eight timings (phases) P1 to P8 for each emission of the irradiating light EL in order to realize each feature function FF. For example, timing P1 can be used for the first emission, timing P2 for the second emission, and similarly thereafter timing P8 can be used for the eighth emission. Then timing P1 can be used again for the ninth emission, and timing P2 can be used for the tenth emission, and so on, allowing timings P1 to P8 to be used in a cyclical manner.

[0075] First, at the P1 timing, if the TDC control signal TDC_CNV1 reaches a logic high level (first level) before the measurement range TR starts, the first TDC is activated and the transition of the TDC code can begin. As an example, we can assume that there is a response from the SPAD pulse at time te. Then, at the P1 timing, the TDC code when this SPAD pulse response is detected can be 1. In this case, the control unit 221 can output index 1 of the memory area to be updated and the value to be added +1. Then, the value stored in memory area 1 of the histogram memory 222 can be updated by +1.

[0076] Next, at P2 timing, the TDC control signal TDC_CNV2 can be started with a delay of 1 / 8 of the time resolution of the TDC data TDC_OUT. That is, the TDC control signal TDC_CNV2 can be activated with a certain time delay compared to the TDC control signal TDC_CNV1. When the TDC control signal TDC_CNV2 reaches a logic high level (first level), the second TDC is activated, and the transition of the TDC code can be initiated.

[0077] For example, we can assume that a SPAD pulse response occurred at time te. Then, at timing P2, the TDC code when this SPAD pulse response is detected can be 1. In this case, the control unit 221 can output index 1 of the memory area to be updated and the value to be added +1. Then, the value stored in memory area 1 of the histogram memory 222 can be updated by +1. Similarly, the above histogram operation of the control unit 221 can be performed for timings P3 to P7.

[0078] Subsequently, at timing P8, the TDC control signal TDC_CNV8 can be activated with a delay of 7 / 8 of the time resolution of the TDC data TDC_OUT compared to timing P1. That is, the TDC control signal TDC_CNV8 can be activated with a certain time delay compared to the TDC control signal TDC_CNV7. When the TDC control signal TDC_CNV8 reaches a logic high level (level 1), the 8th TDC can be activated, and the transition of the TDC code can be initiated.

[0079] For example, we can assume that a SPAD pulse response occurred at time te. Then, at timing P8, the TDC code when this SPAD pulse response is detected can be 0. In this case, the control unit 221 can output index 8 of the memory area to be updated and the value to be added +1. Then, the value stored in memory area 8 of the histogram memory 222 can be updated by +1.

[0080] The control unit 221 can output +1 as the update value for the memory area to be updated. Therefore, in the section where the enable signal CALC_EN is deactivated (the section at the logic low level (second level)), the control unit 221 can choose not to update the value in the histogram memory 222 even if it receives a strobe signal of the TDC code from the TDC block 210.

[0081] We can assume that the probability of SPAD element 101 responding to a photon does not change during timings P1 to P8. Then, if a photon is incident on SPAD element 101 at time te, the value will be updated in either memory area 1 or memory area 8. We can see that the frequency of updating the value in memory area 1 is in a ratio of 7:1 to the frequency of updating the value in memory area 8.

[0082] As described above, in this embodiment, the image sensing device 10_1 does not have fixed operating timings for multiple TDCs, but rather the operating timings of multiple TDCs can be sequentially controlled according to multiple TDC control signals (e.g., TDC_CNV1~8) supplied from the control signal generation unit 240. By shifting the operating timings at which the TDC codes are generated and repeating the P1~P8 timings, the device can repeatedly perform the transmission of the illumination light EL, the detection of SPAD pulses corresponding to the reflected light RL, and the integration of histogram information (memory area index, sum value, shown as calc in the diagram). In this case, the amount of change in the operating timing of the TDC block 210 can be set more precisely than the time resolution of the TDC block 210.

[0083] Thus, because the operating timing of multiple TDCs differs for each SPAD pulse detection, measurements may occur where the code conversion time of the TDC data TDC_OUT does not coincide with the start and end of the measurement range TR. The control unit 221 can be activated during the measurement range TR based on the enable signal CALC_EN. This allows the control unit 221 to detect information that the TDC data TDC_OUT from the TDC block 210 is stroving outside the measurement range TR and prevent the value of the histogram memory 222 from being updated.

[0084] Figure 8 is another embodiment illustrating the histogram operation in the image sensing device shown in Figure 6. In the embodiment shown in Figure 7 above, there are eight feature functions FF (FF1 to FF8), and one example described is one in which the peak times are set to different linear spline functions.

[0085] However, in embodiment (A) of Figure 8, the feature function FF shown on the vertical axis is not 8 steps but a 4-step step. By controlling the operation cycle (cycle 1 to 4) of the TDC differently in each of the 4 steps of the feature function FF, the measurement operation can be repeated to generate a histogram as a time average.

[0086] Furthermore, in embodiment (B) of Figure 8, the feature function FF shown on the vertical axis is a 5-step step, and the time width of each step can be set to be different from each other. For example, after the TDC code is input in cycle 1, the feature function FF can maintain that step for two intervals and then transition to the second step in cycle 2. In each of the 5 steps of the feature function FF, the operation cycle of TDC (cycles 1 to 6) is controlled differently, but the measurement operation may be repeated in a specific step, or it may not be performed in a specific step. For example, the update value can be set to "1" in each operation cycle, but the update value "3" may be generated by repeating operation cycles 3 and 4 three times.

[0087] Figure 9 is another embodiment illustrating the histogram operation in the image sensing device shown in Figure 6. In the embodiment shown in Figure 9, the control unit 221 may have a feature function FF as a sensitivity function that is either positive (+) or negative (-). That is, it is possible to obtain feature function FF values ​​with different signs corresponding to one time bin. If the sum of the photon count values ​​is positive, the feature function FF can have a positive value, and if the sum of the photon count values ​​is negative, the feature function FF can have a negative value. Therefore, it is possible to determine information from two memory areas in one time bin.

[0088] For example, if a photon is detected at time tp1, the value of feature function FF1 can become +2, and the value of feature function FF8 can become -6. This updates the value in memory area 1 by +2 and the value in memory area 8 by -6.

[0089] Then, if a photon is detected at tp2, the value of feature function FF1 can become +4, and the value of feature function FF8 can become -4. As a result, the value in memory area 1 can be updated by +4, and the value in memory area 8 can be updated by -4.

[0090] Furthermore, if a photon is detected at tp3, the value of feature function FF1 can become +6 and the value of feature function FF2 can become +2. This allows the value in memory area 1 to be updated by +6 and the value in memory area 2 to be updated by +2.

[0091] Generally, when background light is strong, the background light is added to the count, so the memory width for each bin must be increased accordingly. However, in this disclosure, when background light is strong, the expected value of the number of incident photons of background light is equal for the interval of the discrete function with a positive sign and the interval of the discrete function with a negative sign, so the mean value of the feature function can be "0". As a result, this disclosure does not require increasing the memory width even when background light is strong.

[0092] Figure 10 is yet another embodiment illustrating the histogram operation in the image sensing device shown in Figure 6. In the image sensing device 10_1 according to the embodiment shown in Figure 10, redundant explanations of operations similar to those in Figure 7 will be omitted, and only operations that differ from those in Figure 7 will be specifically explained.

[0093] In the embodiment shown in Figure 10, the signs of the values ​​(calc) generated by the control unit 221 are represented as b1++, b1--, etc., to clearly explain their signs. Here, b1 represents the index information of the memory area, ++ represents the update value to be added, and -- represents the update value to be subtracted.

[0094] For example, we can assume that a photon is incident on the SPAD element 101 at time te. Then, the added value in memory area 1(b1) can be output as +7. And the subtracted value in memory area 8(b8) can be output as -1. In this way, the present disclosure outputs a positive update value and a negative update value for each time bin, so the bit width of the memory area can be reduced.

[0095] Figure 11 is a configuration diagram relating to an image sensing device according to another embodiment of the present invention. Referring to Figure 11, the image sensing device 10_2 can include a pixel 200, a TDC block 210_1, a histogram circuit 220_1, a signal processing unit 230, and a control signal generation unit 240_1. In the image sensing device 10_2 according to the embodiment of Figure 11, redundant explanations of components similar to those in Figure 6 will be omitted, and only components that differ from those in Figure 6 will be specifically described.

[0096] The TDC block 210_1 can include multiple TDCs. The multiple TDCs can be controlled so that their operating timings differ from each other according to multiple TDC control signals TDC_CNV1~4 supplied from the control signal generation unit 240_1. For example, multiple TDCs can be sequentially activated based on multiple TDC control signals TDC_CNV1~4, and multiple digital codes TDC_OUT0~15 can be generated. The operation of such a TDC block 210_1 will be described in more detail with reference to the drawings described later.

[0097] The histogram circuit 220_1 can generate a histogram based on the TDC data TDC_OUT. The histogram circuit 220_1 may include a first control unit 221_1, a second control unit 221_2, and a histogram memory 222_1.

[0098] If there is a response from the SPAD pulse, not only the TDC code but also a strobe signal indicating that a response has occurred can be transmitted from the TDC block 210_1 to the first control unit 221_1 and the second control unit 221_2. Such a strobe signal can be included in the TDC data TDC_OUT.

[0099] The first control unit 221_1 and the second control unit 221_2 can be activated based on the enable signal CALC_EN. For example, when the enable signal CALC_EN is at an activation level (e.g., a logic level of 1), the first control unit 221_1 and the second control unit 221_2 can perform histogram operation based on the TDC data TDC_OUT. When the enable signal CALC_EN is at an deactivation level (e.g., a logic level of 0), the first control unit 221_1 and the second control unit 221_2 can stop the histogram operation.

[0100] The first control unit 221_1 can generate information about a memory area and an updated value for that memory area (referred to as the first value "calc1") based on the TDC data TDC_OUT. The second control unit 221_2 can generate information about a memory area and an updated value for that memory area (referred to as the second value "calc2") based on the TDC data TDC_OUT. Here, the first control unit 221_1 and the second control unit 221_2 can be activated at different timings based on the enable signal CALC_EN. That is, the first value (calc1) can be output, and then the second value (calc2) can be output after a certain period of time.

[0101] The first control unit 221_1 and the second control unit 221_2 can determine the memory area corresponding to the TDC code and generate an update value of +1 for that memory area. The information and update values ​​(calc1, calc2) of the memory area generated by the first control unit 221_1 and the second control unit 221_2 can be stored in the histogram memory 222_1. The values ​​stored in the histogram memory 222_1 can be output to the signal processing unit 230 in fixed units (for example, in frames).

[0102] The control signal generation unit 240_1 can generate multiple TDC control signals TDC_CNV1~4 to control the operating timing of multiple TDCs. The multiple TDC control signals TDC_CNV1~4 can be activated at different timings. For example, the multiple TDC control signals TDC_CNV1~4 can be activated sequentially.

[0103] The control signal generation unit 240_1 can generate an enable signal CALC_EN to activate the operation of the first control unit 221_1 and the second control unit 221_2. For example, the control signal generation unit 240_1 can generate a first enable signal to activate the first control unit 221_1 and a second enable signal to activate the operation of the second control unit 221_2. In this case, the control signal generation unit 240_1 can activate the first enable signal and then activate the second enable signal after a certain period of time.

[0104] In the embodiment shown in Figure 6, only the value in one memory area can be updated during the detection of one photon. In contrast, in the embodiment shown in Figure 11, two memory areas can be updated for the detection of one photon, thus preventing a deterioration in measurement accuracy.

[0105] Figure 12 is a diagram illustrating the histogram operation in the image sensing device shown in Figure 11. In the image sensing device 10_2 according to the embodiment shown in Figure 12, redundant explanations of operations similar to those in Figure 7 will be omitted, and only operations that differ from those in Figure 7 will be specifically explained.

[0106] Referring to Figure 12, the histogram circuit 220_1 can control the update operations of two control units 221_1 and 221_2 with a single TDC code value. The TDC block 210_1 can generate TDC data TDC_OUT at twice the speed compared to the embodiment in Figure 7.

[0107] The time resolution of the TDC data TDC_OUT can be set to 2 / 8 (i.e., 1 / 4) of the discrete function, rather than one stage. That is, eight stages of the discrete function can correspond to two TDC codes. In other words, the TDC block 210_1 can convert the response time of a SPAD pulse into a single TDC code with a time resolution corresponding to four steps of each discrete function. Sixteen TDC data codes TDC_OUT0~15 can be generated within the measurement range TR. That is, the TDC block 210_1 can generate 16 TDC codes (0~15) with a time resolution that divides the measurement time into 16 parts.

[0108] The embodiment in Figure 12 requires only half the timings P1-P4 compared to the embodiment in Figure 7, because two histogram information can be obtained during a single photon detection operation. In the embodiment in Figure 12, the four timings (phases) P1-P4 can be switched with each emission of the irradiated light EL in order to realize each feature function FF. For example, timing P1 can be used for the first emission, timing P2 for the second emission, and similarly timing P4 for the fourth emission. Then, timing P1 is used again for the fifth emission, and then timing P2 is used for the sixth emission, and so on, allowing timings P1-P4 to be used in a cyclical manner.

[0109] First, at the P1 timing, before the measurement range TR begins, the TDC control signal TDC_CNV1 becomes logic high level (first level), the first TDC is activated, and the transition of the TDC code can begin. As an example, we can assume that there is a response from the SPAD pulse at time te. Then, at the P1 timing, the TDC code when this SPAD pulse response is detected can be 2. In this case, the control units 221_1 and 221_2 can output index 1 of the memory area to be updated and the value to be added +1. Then, the value stored in memory area 1 of the histogram memory 222_1 can be updated by +1.

[0110] Next, at P2 timing, the TDC control signal TDC_CNV2 can be started with a delay of 1 / 4 of the time resolution of the TDC data TDC_OUT. That is, the TDC control signal TDC_CNV2 can be activated with a certain time delay compared to the TDC control signal TDC_CNV1. When the TDC control signal TDC_CNV2 reaches a logic high level (first level), the second TDC is activated, and the transition of the TDC code can be initiated.

[0111] As an example, we can assume that a SPAD pulse response was received at time te. Then, at timing P2, the TDC code when this SPAD pulse response is detected can be 2. In this case, control units 221_1 and 221_2 can output index 1 of the memory area to be updated and the value to be added +1. Then, the value stored in memory area 1 of the histogram memory 222_1 can be updated by +1. Similarly, the above operations of control units 221_1 and 221_2 can be performed for timing P3.

[0112] Subsequently, at P4 timing, the TDC control signal TDC_CNV4 can be activated with a delay of 3 / 4 of the time resolution of the TDC data TDC_OUT compared to P1 timing. That is, the TDC control signal TDC_CNV4 can be activated with a certain time delay compared to the TDC control signal TDC_CNV3. When the TDC control signal TDC_CNV4 reaches a logic high level (first level), the fourth TDC can be activated, and the transition of the TDC code can be initiated.

[0113] As an example, we can assume that there is a response from the SPAD pulse at time te. Then, at timing P4, the TDC code when this SPAD pulse response is detected can be 1. In this case, the first control unit 221_1 can output index 1 of the memory area to be updated and the value to be added + 1. Then, the value stored in memory area 1 of the histogram memory 222_1 can be updated by + 1. In contrast, the second control unit 221_2 can output index 8 of the memory area to be updated and the value to be added + 1. Then, the value stored in memory area 8 of the histogram memory 222_1 can be updated by + 1.

[0114] Figures 13 and 14 are other embodiments illustrating the histogram operation in the image sensing device shown in Figure 11. The image sensing device 10_2 according to the embodiment shown in Figure 13 shows the case where there are two feature functions FF. By combining feature functions FF1 and FF2, it is possible to generate a combined feature function FFC that overlaps at the same TDC operating speed. When performing a histogram using the combined feature function FFC, two or more update values ​​are required.

[0115] Furthermore, the image sensing device 10_2 according to the embodiment shown in Figure 14 can set an overlapping combined feature function FFC by combining a feature function FF1 having a positive (+) value and a feature function FF2 having a negative (-) value. Here, since the combined feature function FFC is set by combining feature functions FF1 and FF2 having positive and negative values, the average value of the combined feature function FFC can be "0". As a result, the histogram counts of ambient light are canceled out, so there is no need to increase the memory width even when the background light is strong.

[0116] Figure 15 is another embodiment illustrating the histogram behavior according to the embodiment of Figure 11. In the image sensing device 10_2 according to the embodiment shown in Figure 15, redundant explanations of operations similar to those in Figure 12 will be omitted, and only operations that differ from those in Figure 12 will be specifically explained.

[0117] In the embodiment shown in Figure 15, the signs of the values ​​(calc1, calc2) generated by the first control unit 221_1 and the second control unit 221_2 are represented as b1++ and b1-- respectively to clearly explain their signs. Here, b1 represents the index information of the memory area, "++" can represent the update value to be added, and "--" can represent the update value to be subtracted.

[0118] For example, we can assume that a photon is incident on the SPAD element 101 at time te. Then, the first control unit 221_1 can output the sum of memory area 1(b1) as +7. And the second control unit 221_2 can output the subtraction of memory area 8(b8) as -1. In this way, the present disclosure outputs a positive update value and a negative update value for each time bin, thereby reducing the bit width of the memory area.

[0119] As a result, the embodiment shown in Figure 15 can suppress the degradation of the signal-to-noise ratio (SNR) by updating the values ​​in two memory areas for each photon detection.

[0120] Figure 16 is a configuration diagram relating to an image sensing device according to yet another embodiment of the present invention. Referring to Figure 16, the image sensing device 10_3 can include a pixel 200, a TDC block 210_2, a histogram circuit 220_2, a signal processing unit 230, and a control signal generation unit 240_2. In the image sensing device 10_3 according to the embodiment of Figure 16, redundant explanations of components similar to those in Figure 6 will be omitted, and only components that differ from those in Figure 6 will be specifically described.

[0121] The histogram circuit 220_2 can generate a histogram based on the TDC data TDC_OUT. The histogram circuit 220_2 can perform histogram operations by combining the spline sketch method and the partial histogram method described above. The histogram circuit 220_2 can estimate the peak position based on the histogram obtained by coarse measurement and use it to control the operation of fine measurement.

[0122] Such a histogram circuit 220_2 may include a coarse control unit 221_3, a fine control unit 221_4, and a histogram memory 222_2.

[0123] When the enable signal CALC_EN is activated, the course control unit 221_3 can generate information about a memory area and an update value (referred to as the third value "calc_c") for that memory area based on the TDC data TDC_OUT. For example, the course control unit 221_3 can determine the index of the memory area corresponding to the TDC code and generate an update value of +1 for that memory area.

[0124] The fine control unit 221_4 can generate fine measurement values ​​(referred to as the fourth value "calc_f") in the measurement interval of a specific TDC code (the zoom-in interval TZ described later). The fine control unit 221_4 can perform measurements at smaller time intervals than the course control unit 221_3 and generate fine measurement values ​​affected by noise or pulse width.

[0125] The information and updated values ​​(calc_c, calc_f) of the memory area generated by the course control unit 221_3 and the fine control unit 221_4 can be stored in the histogram memory 222_2. The values ​​stored in the histogram memory 222_2 can be output to the signal processing unit 230 in fixed units (for example, in frames). The operation of the histogram circuit 220_2 with this configuration will be explained in more detail with reference to Figures 17 and 18, which will be described later.

[0126] Figure 17 is a diagram illustrating the histogram operation in the image sensing device shown in Figure 16. In the image sensing device 10_3 according to the embodiment shown in Figure 17, redundant explanations of operations similar to those in Figure 7 will be omitted, and only operations that differ from those in Figure 7 will be specifically explained.

[0127] Referring to Figure 17, the histogram circuit 220_2 can perform histogram operation by combining a feature function FF that uses an 8-step step-like discrete function with the aforementioned partial histogram method.

[0128] Figure 17(A) shows the case where the course control unit 221_3 performs coarse measurement. In the course measurement in (A), the discrete function has 8 steps, and the number of memory areas may also be 8.

[0129] As an example, we can assume that a SPAD pulse response was received at time te. In this case, the TDC code when the SPAD pulse response is detected can be 2. In this case, the course control unit 221_3 can output index 2 of the memory area to be updated and the value to be added +1. Then, the value stored in memory area 2 of the histogram memory 222_2 can be updated by +1. The course measurement operation shown in Figure 17(A) is the same as in Figure 7, so a detailed explanation thereof is omitted.

[0130] Figure 17(B) shows the case where the fine control unit 221_4 performs fine measurement. Each fine feature function ffi (where i is the memory index) of the fine measurement can be represented in lowercase to distinguish it from each feature function FFi ​​of the course measurement. For example, there may be eight fine feature functions ff (ff1 to ff8), each set to a first-order spline function with a different peak time. The value shown on the vertical axis (val) represents the feature value of the fine feature function ff, and may be a value replaced by a two-step stepwise discrete function.

[0131] In the fine measurement in (B), the values ​​of the zoom-in interval (TZ) indicated by the arrows on both sides can be measured. One fine feature function ff1, which has two steps per step of the feature function FF, can be matched. For example, if the response pulse of reflected light RL is detected from TDC code "2", this range can be set as the zoom-in interval TZ.

[0132] The fine control unit 221_4 can use two timings (phases) p1 and p2 to realize each fine feature function ff. The fine measurement timings p1 and p2 can be represented in lowercase to distinguish them from the course measurement timings P1 to P8. For example, the fine control unit 221_4 can use the p1 timing synchronized with the first step of the fine feature function ff1, and the p2 timing synchronized with the second step of the fine feature function ff1. That is, the p2 timing can be activated after the p1 timing has started and after a certain delay.

[0133] Within the zoom-in interval TZ, eight fine TDC codes can be generated based on the p1 timing. Then, within the zoom-in interval TZ, eight fine TDC codes can be generated based on the p2 timing. The fine measurement value (calc_f) generated by the fine control unit 221_4 can be transmitted to the histogram memory 222_2. The value stored in memory area 2 of the histogram memory 222_2 can be updated only by the fine measurement value (calc_f).

[0134] The discrete function used in fine measurement has two steps, and the number of memory areas may be eight. In this case, the accuracy required for course measurement is not the accuracy of the final measurement, but rather a level at which no signal loss occurs in fine measurement is sufficient. Because the time during which background light noise can affect each feature function in fine measurement is short, it is possible to achieve the same accuracy as TCSPC, which includes partial histograms.

[0135] Figure 18 is another embodiment illustrating the histogram operation in the image sensing device shown in Figure 16. In the image sensing device 10_2 according to the embodiment shown in Figure 18, redundant explanations of operations similar to those in Figure 17 will be omitted, and only operations that differ from those in Figure 17 will be specifically explained.

[0136] In the embodiment shown in Figure 18, the value (val) shown on the vertical axis represents the fine feature function ff. While the feature function in Figure 17 is shown as a two-step stepped discrete function, the embodiment in Figure 18 can have a one-shot pulse-like feature function. The pulse-like fine feature function ff can correspond to one step of the feature function FF.

[0137] The fine feature functions ff1 to ff8 can be sequentially activated within the zoom-in interval TZ, corresponding to each step of the feature functions FF1 to FF8. For example, at time tp, when the peak value is highest, the pulse of the first fine feature function ff1 can be activated, corresponding to the feature function FF1. Subsequently, the pulse of the fine feature function ff2 can be activated, corresponding to the feature function FF2, which has the second highest peak value. Similarly, pulses of the fine feature functions ff3 to ff8, which are matched to the feature functions FF3 to FF8, can be sequentially activated.

[0138] When the fine feature function ff is activated, the fine control unit 221_4 can output more refined information (updated values ​​refined by TDC code "2" (referred to as "calc_f")) based on the TDC code.

[0139] Figure 19 is an operation timing diagram for the control signal generation unit in Figure 6. The embodiment shown in Figure 19 can be applied not only to the embodiment in Figure 6, but also to the embodiments in Figures 11 and 16. However, in this disclosure, the embodiment applicable to Figure 6 will be described as an example.

[0140] Referring to Figure 19, the control signal generation unit 240 can be reset at time T1 based on the reset signal RST. Here, the reset signal RST may be a signal generated by a timing controller (described later).

[0141] After a certain period of time has elapsed since the reset of the control signal generation unit 240, the TDC control signal TDC_CNV1 is activated at time T2, and the transition of the TDC code can begin. After a certain period of time has elapsed since time T2, the TDC control signal TDC_CNV2 is activated at time T3, and the TDC code can transition. Similarly, after a certain period of time has elapsed since time T3, the TDC control signals TDC_CNV3 to T4 are sequentially activated, and the TDC code can transition.

[0142] Subsequently, when the transfer signal TX_ON is activated at T5, the illumination light EL can be irradiated onto the target object. Here, the transfer signal TX_ON may be a signal generated from a timing controller (described later). The control signal generation unit 240 can activate the enable signal CALC_EN at T5. Then, the histogram operation of the histogram circuit 220 can be performed during the interval in which the enable signal CALC_EN is activated.

[0143] Thus, this disclosure makes it possible to generate a more detailed histogram by sequentially activating the TDC control signals TDC_CNV1~8 at different timings P1~P8, rather than fixing them. Such operation can be achieved, for example, by applying a clock (CLK), DLL (Delay Locked Loop), or counter with a frequency higher than the time resolution of the TDC to the control signal generation unit 240.

[0144] Figure 20 is a configuration diagram of an imaging apparatus including an image sensing device according to an embodiment of the present invention. Referring to Figure 20, the imaging device CD may refer to a device such as a digital still camera for taking still images or a digital video camera for taking moving images. For example, the imaging device CD can be implemented in a digital single-lens reflex (DSLR), mirrorless camera, or mobile phone (especially a smartphone), but is not limited to these. The imaging device CD may also be a concept that includes a lens and an image sensor, and is capable of photographing a subject and generating an image. According to one embodiment, the imaging device CD may be a lidar sensor.

[0145] The imaging device CD may include an image sensing device 10_4 and an image signal processor (ISP).

[0146] The image sensing device 10_4 may be a CIS (Complementary Metal Oxide Semiconductor Image Sensor) that converts incident light into an electrical signal. The image sensing device 10_4 can measure distance using the TOF (Time of Flight) principle. The image sensing device 10_4 in the embodiment of Figure 20 can represent the image sensing devices 10, 10_1, 10_2, and 10_3 described above.

[0147] The image sensing device 10_4 may include a light source LS, a lens module LM, a pixel array 300, a pixel driver 320, a readout circuit 330, a timing controller 340, and a light source driver 350. In the embodiment shown in Figure 20, the pixel 310 included in the pixel array 300 may represent the pixel 200 in the aforementioned drawing. The timing controller 340 may include the control signal generation units 240, 240_1, and 240_2 in the aforementioned drawing.

[0148] The light source LS can irradiate the target object TO with light in response to the clock signal MLS from the light source driver 350. The light source LS may be a monochromatic illumination source that combines a laser diode (LD), light-emitting diode (LED), near-infrared laser (NIR), point light source LS, white lamp, and monochromator, or a combination of other laser light sources LS that emit light in a specific wavelength band (e.g., infrared or visible light).

[0149] For example, the light source LS may be light in the infrared wavelength band having a wavelength of 800 nm to 1000 nm (hereinafter referred to as "infrared light"), and this disclosure will be explained assuming that the light source LS emits infrared light. On the other hand, the light emitted from the light source LS may be pulsed light having a predetermined period, amplitude, and pulse width. Although only one light source LS is shown in Figure 20 for the sake of explanation, multiple light source LS may be arranged around the lens module LM.

[0150] The lens module LM can collect light reflected from the target object TO and focus it on pixels 310 of the pixel array 300. For example, the lens module LM may include a focusing lens made of glass or plastic surface or other cylindrical optical element. The lens module LM may include multiple lenses aligned around the optical axis.

[0151] The pixel array 300 may include a plurality of pixels 310 arranged continuously in a two-dimensional matrix structure. That is, the plurality of pixels 310 can be arranged continuously along the column direction and the row direction. Each pixel 310 can, under the control of the pixel driver 320, photoelectrically convert incident light received via the lens module LM and generate and output a pixel signal, which is an electrical signal corresponding to the incident light. In this case, the pixel signal may not be a signal indicating the color of the target object TO, but rather a signal indicating information corresponding to the distance to the target object TO.

[0152] Each pixel 310 may be an infrared light pixel that detects incident light, including reflected light (RL), which is incident light when emitted light (EL) from a light source LS is reflected off the target object TO, and generates a pixel signal. In this disclosure, the infrared light pixel may also be a depth pixel for calculating the distance to the target object TO.

[0153] A pixel array 300, which has multiple pixels 310 arranged in a row, can detect the distance to a target object TO using a direct TOF method. For reference, the direct TOF method is a method that calculates the distance to a target object TO by directly measuring the round-trip time from the time pulsed light is shone on the target object TO to the time the pulsed light is reflected from the target object TO and becomes incident, and then calculating the round-trip time and the speed of light.

[0154] The pixel driver 320 can drive the pixel array 300 in response to the control of the timing controller 340. For example, the pixel driver 320 can generate a control signal that allows for the selection and control of pixels 310 included in at least one row line among the multiple row lines of the pixel array 300. The pixel driver 320 can also generate a recharging signal that controls a recharging operation in which charge is injected into the sensing node connected to the SPAD element 201 of the pixel 310.

[0155] The readout circuit 330 is located on one side of the pixel array 300 and can calculate the time delay between the pulse signal output from each pixel 310 and a reference pulse, and generate and store digital data corresponding to the time delay. Here, the reference pulse may be a pulse of the clock signal MLS. The readout circuit 330 may include digital logic that calculates the time delay between the pulse signal of each pixel 310 and the reference pulse to generate digital data, and an output buffer that stores the generated digital data. Such digital logic and output buffer may be referred to as the TDC blocks 200, 210, 210_1, and 210_2 described above. The readout circuit 330 can transfer the stored digital data to the image signal processor ISP in accordance with the control of the timing controller 340.

[0156] The readout circuit 330 processes the pixel signal PX_OUT output from the pixel array 300 in accordance with the control of the timing controller 340, and can generate and store depth data for detecting the distance to the target object TO. Specifically, the readout circuit 330 can calculate the time of flight corresponding to the SPAD pulse generated by each pixel 310 upon detecting incident light including reflected light RL, and can store the time of flight corresponding to the SPAD pulse. The readout circuit 330 can transfer the stored time of flight to the image signal processor ISP in accordance with the control of the timing controller 340.

[0157] The timing controller 340 can control the overall operation of the image sensing device 10_4. Specifically, the timing controller 340 can generate clock and timing signals to control the operation of the pixel driver 320 and the light source driver 350. According to one embodiment, the timing controller 340 can generate clock and timing signals in response to data transmitted from the readout circuit 330 or to the request of the image signal processor ISP. The timing controller 340 can control the optical power of the illuminating light EL by controlling the light source driver 350 in response to the control signal applied from the readout circuit 330.

[0158] Furthermore, the timing controller 340 can control the activation or deactivation of the readout circuits 330 and control the simultaneous or sequential transfer of the digital data stored in each readout circuit 330 to the image signal processor ISP. According to one embodiment, the timing controller 340 may include a logic control circuit, a phase lock loop (PLL) circuit, a timing control circuit, and a communication interface circuit.

[0159] The light source driver 350 can generate a clock signal MLS that can drive the light source LS in accordance with the control of the timing controller 340. The light source driver 350 can control the waveform (period, amplitude, and pulse width, etc.) of the illumination light EL output from the light source LS.

[0160] The image signal processor ISP can control the operation of the image sensing device 10_4. The image signal processor ISP can analyze the digital data input from the image sensing device 10_4 to determine the mode of the image sensing device 10_4, and can control the image sensing device 10_4 to operate in the determined mode.

[0161] The image signal processor ISP can perform image signal processing on the image data IDATA received from the image sensing device 10_4 and generate processed image data. The image data IDATA may include the time of flight as described above. The image signal processor ISP can perform image signal processing on the image data to reduce noise and improve image quality, such as interpolation and lens distortion correction.

[0162] The image signal processor ISP can transfer the processed image data to a host device (e.g., the host 30 mentioned above). The host device may be a processor that processes the image-processed image data received from the image signal processor ISP (e.g., an application processor), a memory that stores the image data (e.g., non-volatile memory), or a display device that visually outputs the image data (e.g., an LCD (liquid crystal display)). [Explanation of Symbols]

[0163] 1. Rider System 10 Image Sensing Device 20 Communication Interfaces 30 hosts 100 pixels 101 SPAD Element 102 Quenching circuit 103 buffer 110 TDC Block 120 Histogram Circuit 130 Signal Processor 200 TDC Block 200 pixels 201 SPAD element 202 Quenching Circuit 203 buffer 210 TDC Block 220 Histogram Circuit 221 Control Unit 222 Histogram Memory 230 Signal Processing Unit 240 Control signal generation unit 300 pixel array 310 pixels 320 Pixel Driver 330 circuits 340 Timing Controller 350 Light source driver

Claims

1. A pixel that generates a pulsed signal based on photons reflected from the target object, A TDC block includes a plurality of TDCs (time-to-digital converters) each that generate a digital code corresponding to the time delay between the pulse signal and the reference pulse, A histogram circuit that generates index information and update values ​​for a memory area based on the aforementioned digital code, Includes, An image sensing device in which the aforementioned multiple TDCs have different activation timings.

2. The aforementioned pixel is A SPAD (single photon avalanche diode) element that generates voltage pulses via a sensing node, A quenching circuit that controls the reverse bias voltage applied to the SPAD element to perform a quenching operation, The image sensing apparatus according to claim 1, further comprising a buffer that samples the voltage pulse generated at the sensing node and converts it into a pulse signal.

3. The aforementioned histogram circuit is The image sensing apparatus according to claim 1, wherein the value of the time bin corresponding to the time when the photon was detected in the timestamp interval is overlapped with the value of the adjacent time bin, and the index information of the memory area and the updated value are generated based on the digital code at the time the pulse signal responds.

4. The aforementioned histogram circuit is A control unit that generates the updated value using a feature function in an interval obtained by dividing the timestamp interval by the number of memory areas, The image sensing apparatus according to claim 1, further comprising a histogram memory that updates the value of the memory area by the update value.

5. The aforementioned feature function is, The image sensing device according to claim 4, which is a function that sketches a sinusoidal waveform corresponding to the order of the spline function in the aforementioned timestamp interval.

6. The control unit, The feature function is discretized with respect to the timestamp value and the number of detected photons, and the peak value of each step is output in a stepwise manner. The image sensing device according to claim 4, wherein the update value is set in accordance with each of the steps of the stepped arrangement.

7. The aforementioned feature function has a form in which it increases sequentially over multiple measurement intervals and then decreases. The image sensing device according to claim 6, wherein the update value is set to a positive value in accordance with each step of the feature function.

8. The aforementioned feature function has a form in which it sequentially decreases and then increases over multiple intervals. The image sensing device according to claim 6, wherein the update value is set to a negative value in accordance with each step of the feature function.

9. The aforementioned TDC block is The image sensing device according to claim 6, which converts the digital code so that a plurality of steps of the feature function correspond to a single digital code.

10. The aforementioned histogram circuit is A first control unit that generates information about a first memory area and a first update value based on the digital code, A second control unit that generates information regarding a second memory area and a second updated value based on the aforementioned digital code, The image sensing apparatus according to claim 1, comprising: a histogram memory that updates the value of the first memory area by the first update value and updates the value of the second memory area by the second update value.

11. The image sensing apparatus according to claim 10, wherein the first control unit and the second control unit are activated at different timings.

12. The aforementioned histogram circuit is A course control unit that generates index information and updated values ​​for the memory area based on the digital code during the measurement interval, A fine control unit that generates fine measurement values ​​for the memory area in a zoom-in section corresponding to a specific digital code in which the photon is detected, The image sensing apparatus according to claim 1, further comprising a histogram memory that updates the value of the memory area in accordance with the output of the course control unit and the fine control unit.

13. The course control unit, The feature function is discretized with respect to the timestamp value and the number of detected photons, and the peak value of each step is output in a stepwise manner with different values. The image sensing device according to claim 12, wherein the update value is set in accordance with each of the steps of the stepped arrangement.

14. The fine control unit is The image sensing device according to claim 13, wherein a plurality of fine feature functions corresponding to each step of the feature function are generated in the zoom-in section, and the plurality of fine feature functions are activated at different timings from each other.

15. Each of the aforementioned multiple fine feature functions is, The image sensing device according to claim 14, wherein each step of the feature function is discretized and has a step-like structure with different peak values.

16. Each of the aforementioned multiple fine feature functions is, The image sensing device according to claim 14, having a one-shot pulse shape corresponding to each step of the feature function.

17. The image sensing apparatus according to claim 1, further comprising a control signal generation unit that sequentially activates a plurality of TDC control signals for controlling the plurality of TDCs after a reset operation, and generates an enable signal for activating the histogram circuit when the target object is irradiated with light.

18. A pixel that generates a pulsed signal based on photons reflected from the target object, A TDC block that generates a digital code in response to the time delay between the pulse signal and the reference pulse, A histogram circuit that generates a first feature function and a second feature function having waveforms corresponding to the order of the spline function in the timestamp interval to which the photon responds, and generates information and updated values ​​for the memory area using the first feature function and the second feature function, Includes, The first feature function and the second feature function are, An image sensing device having discretized values ​​with respect to the timestamp value and the number of detected photons, with each step having a different peak value.

19. The aforementioned TDC block is A first TDC generates multiple first digital codes based on a first TDC control signal at a first timing, The image sensing apparatus according to claim 18, comprising a second TDC that generates a plurality of second digital codes based on a second TDC control signal at a second timing after the first timing.

20. The first feature function and the second feature function are, The image sensing apparatus according to claim 19, wherein time bin intervals corresponding to the time when the photon is detected overlap, and the updated values ​​for at least two or more memory areas are generated at the time the pulse signal responds.

Images