Image sensing device

Abstract

Disclosed is an image sensing apparatus capable of detecting a distance to a target object according to a time of flight (TOF) method. The image sensing apparatus includes: a pixel configured to generate a pulse signal based on photons reflected from a target object; a plurality of time-to-digital converters (TDCs) configured to generate a digital code corresponding to a time delay between the pulse signal and the reference pulse; and a histogram circuit configured to generate index information of the memory region and an updated value of the memory region based on the digital code, in which the plurality of TDCs have different enable timing points.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to an image sensing device. Background Technology

[0002] Recently, the much-discussed Time-of-Flight (TOF) technology involves emitting pulsed light from a light source located within or near a sensor towards a target object, receiving the light reflected from the target object, and using both the emitted and reflected light to calculate the round-trip time. Based on the principle of the constancy of the speed of light, the calculated round-trip time is used to measure the distance to the target object. For accurate TOF measurements, since the light must react as soon as it reaches the light-receiving element, TOF technology requires highly sensitive photoelectric conversion elements. Therefore, single-photon avalanche diodes (SPADs), which can be fabricated using CMOS technology, have been actively researched.

[0003] LiDAR sensors can detect objects around a user and determine the distance between the object and the user. This allows LiDAR sensors to prevent accidents that the user might not have detected beforehand and to enable autonomous driving of various electronic devices. Time-of-Flight (TOF) technology can be used with LiDAR sensors to determine the distance to a target object. Summary of the Invention

[0004] Various embodiments of this disclosure provide a TOF-based image sensing device that can reduce memory capacity while improving its operating speed.

[0005] According to embodiments of the present disclosure, an image sensing device may include: a pixel configured to generate a pulse signal based on photons reflected from a target object; a plurality of time-to-number converters (TDCs) configured to generate digital codes corresponding to a time delay between the pulse signal and a reference pulse; and a histogram circuit configured to generate index information of a memory region and an update value of the memory region based on the digital codes, wherein the plurality of TDCs have different enable timing points.

[0006] According to embodiments of the present disclosure, an image sensing device may include: a pixel configured to generate a pulse signal based on photons reflected from a target object; a time-to-number converter (TDC) block configured to generate a digital code corresponding to a time delay between the pulse signal and a reference pulse; and a histogram circuit configured to generate a first feature function and a second feature function, each of the first feature function and the second feature function having a waveform corresponding to the order of a spline function within a timestamp segment in which the photon responds, and generating information about a memory region and an updated value of the memory region based on the first feature function and the second feature function, wherein each of the first feature function and the second feature function is discretized relative to the number of photon detection counts and the timestamp value, and has a stepped shape with each step having a different peak value.

[0007] According to embodiments of this disclosure, a method for operating an image sensing device may include the following steps: generating a pulse signal based on photons reflected from a target object; generating a digital code corresponding to a time delay between the pulse signal and a reference pulse at different timing points; and generating index information of a memory region and an update value of the memory region based on the digital code.

[0008] It will be understood that the foregoing general description and the following detailed description of the embodiments of this disclosure are both illustrative and descriptive, and are intended to provide a further description of the claimed embodiments. Attached Figure Description

[0009] The above and other features and advantages of embodiments of this disclosure will become readily apparent when considered in conjunction with the accompanying drawings, with reference to the following detailed description.

[0010] Figure 1 This is a diagram illustrating a lidar system according to an embodiment of the present disclosure.

[0011] Figure 2 This illustrates an embodiment according to the present disclosure. Figure 2 A diagram illustrating the operation of the image sensing device.

[0012] Figure 3 This illustrates an embodiment of the present disclosure for use with Figure 1 The diagram shown is a histogram of the histogram circuit based on the photon count values.

[0013] Figure 4 This illustrates an embodiment of the present disclosure for use with Figure 1 The graph shows the characteristic functions of the image sensing device.

[0014] Figure 5A and Figure 5B This illustrates an embodiment of the present disclosure for use with Figure 1The diagram shows the histogram operation of the image sensing device.

[0015] Figure 6 This is a diagram illustrating another image sensing device according to an embodiment of the present disclosure.

[0016] Figures 7 to 10 This illustrates an embodiment of the present disclosure for use with Figure 6 The diagram shows the histogram operation of the image sensing device.

[0017] Figure 11 This is a diagram illustrating another image sensing device according to an embodiment of the present disclosure.

[0018] Figures 12 to 15 This illustrates an embodiment of the present disclosure for use with Figure 11 The diagram shows the histogram operation of the image sensing device.

[0019] Figure 16 This is a schematic diagram illustrating another image sensing device according to yet another embodiment of the present disclosure.

[0020] Figure 17 and Figure 18 This illustrates an embodiment of the present disclosure for use with Figure 16 The diagram shows the histogram operation of the image sensing device.

[0021] Figure 19 It is shown Figure 6 The timing diagram shown illustrates the operation of the control signal generator.

[0022] Figure 20 This is a block diagram illustrating an imaging apparatus including an image sensing device according to an embodiment of the present disclosure. Detailed Implementation

[0023] This disclosure provides embodiments and examples of an image sensing apparatus capable of detecting the distance to a target object using a time-of-flight (TOF) method, which can be used in a configuration that substantially solves one or more technical or engineering problems and mitigates limitations or disadvantages encountered in some image sensing apparatuses in the art. Some embodiments of this disclosure relate to a TOF-based image sensing apparatus capable of reducing memory capacity while improving its operating speed. Recognizing the aforementioned problems, the TOF-based image sensing apparatus according to this disclosure can reduce the memory capacity required for system operation while improving its operating speed.

[0024] Reference will now be made in detail to some embodiments of this disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to denote the same or similar parts. Although this disclosure is readily available in various modifications and alternatives, specific embodiments are shown in the drawings as examples. However, this disclosure should not be construed as limiting itself to the embodiments set forth herein.

[0025] Various embodiments will now be described with reference to the accompanying drawings. However, it should be understood that this disclosure is not limited to the specific embodiments, but includes various modifications, equivalents, and / or substitutions of the embodiments. The embodiments of this disclosure can provide various advantageous effects that can be directly or indirectly identified.

[0026] Figure 1 This is a circuit diagram showing a lidar system 1 according to an embodiment of the present disclosure.

[0027] Reference Figure 1 The lidar system 1 may include an image sensing device 10, a communication interface 20, and a host 30.

[0028] Image sensing device 10 may include pixels 100, a time-to-number converter (hereinafter referred to as "TDC") block 110, histogram circuitry 120, and signal processor 130. For example, image sensing device 10 may measure the time until light is reflected from and returns from a target object by using a time-correlated single-photon counting (hereinafter referred to as "TCSPC") method.

[0029] Pixel 100 includes a single-photon avalanche diode (SPAD) element (hereinafter referred to as "SPAD element") 101, a quenching circuit 102, and a buffer 103. Pixel 100 may be grouped into macropixel units including at least one SPAD element 101. Pixel 100 may include a single macropixel or may include multiple macropixels arranged in an array.

[0030] SPAD element 101 can detect a single photon of reflected light (RL) reflected from a target object and can generate a voltage pulse corresponding to the detected single photon. SPAD element 101 can operate as a photodiode including a photosensitive PN junction. SPAD element 101 can generate a voltage pulse because a single incident photon triggers avalanche breakdown in Geiger mode, where a reverse bias voltage caused by a cathode-anode voltage higher than the breakdown voltage occurs. Geiger mode can be defined as applying a reverse bias voltage greater than the breakdown voltage to SPAD element 101 to detect a single photon in SPAD element 101. In Geiger mode, due to the large intensity of the electric field applied to the amplification layer, even if a small number of photons are absorbed, avalanche current breakdown occurs and a large current is output, thus enabling the detection of a single photon. Therefore, the process of triggering avalanche breakdown by a single photon and generating a voltage pulse will be defined hereinafter as the avalanche process.

[0031] SPAD element 101 can be connected to a ground voltage terminal via one of its terminals (i.e., the anode). The other terminal of SPAD element 101 (i.e., the cathode) can be connected to a sensing node (SN). SPAD element 101 can generate a current pulse by detecting a single photon and can output the generated current pulse to the sensing node (SN).

[0032] Although embodiments of this disclosure show SPAD element 101 as a light receiving element (i.e., light detection element) of pixel 100, the scope of the embodiments is not limited to SPAD element 101. That is, in addition to SPAD element 101, various elements operating in Geiger mode, such as avalanche photodiodes (APDs), silicon photomultiplier tubes (SiPMs), etc., can be used as light receiving elements of pixel 110.

[0033] The quenching circuit 102 can control the voltage of the SPAD element 101 and output the resulting voltage to the sensing node (SN). After a voltage pulse is generated due to avalanche breakdown and the voltage of the sensing node (SN) changes, the quenching circuit 102 can perform a quenching operation to return the voltage of the sensing node (SN) to Geiger mode.

[0034] The quenching circuit 102 can be connected between the sensing node (SN) and the ground voltage terminal. For example, if the quenching circuit 102 is implemented as an active device, it can be implemented as a transistor. In another embodiment, if the quenching circuit 102 is implemented as a passive device, it can be implemented as a resistor.

[0035] Buffer 103 can generate a pulse signal based on an electrical signal generated according to photons incident on pixel 100, and can output a pixel signal (PX_OUT). Buffer 103 can generate the pulse signal at a frequency based on the frequency of the received photons. Buffer 103 can sample analog voltage pulses generated from the sensing node (SN), and can convert the sampled analog voltage pulses into digital pulse signals (i.e., SPAD pulses). The sampling method may be a method of converting voltage pulses into pulse signals with logic levels 0 or 1 based on whether the voltage pulse level is equal to or higher than a threshold level, but embodiments of this disclosure are not limited thereto.

[0036] TDC block 110 can calculate the time delay between the pixel signal (PX_OUT) output from pixel 100 and the reference pulse of the emitted light (EL) and can convert the time delay into a digital value, and can generate TDC data (TDC_OUT). The pulse signal of the pixel signal (PX_OUT) generated from pixel 100 may be referred to as a SPAD pulse. In some embodiments, TDC block 110 may include multiple TDCs.

[0037] Histogram circuit 120 can generate a histogram based on TDC data (TDC_OUT). Histogram circuit 120 can accumulate and store timestamp data in time bins based on TDC data (TDC_OUT), and can determine bins with peak values. In some embodiments, histogram circuit 120 may also be included in pixel 100. Detailed operation of histogram circuit 120 will be described in more detail with reference to the following embodiments.

[0038] Signal processor 130 can determine the distance to a target object by calculating the time of flight (TOF) of light (hereinafter referred to as "optical TOF") based on histogram data received from histogram circuit 120. For example, signal processor 130 can be implemented as a general-purpose digital signal processor, processor, or controller, which includes combinational circuitry connected to a memory. Alternatively, signal processor 130 can also be implemented as a custom application-specific integrated circuit (ASIC), and the type of signal processor is not limited thereto.

[0039] Data on the optical Time-of-Flight (TOF) calculated by the signal processor 130 can be transmitted to the host 30 via the communication interface 20. For example, the communication interface 20 may be a serial communication interface. For example, the host 30 may provide a three-dimensional (3D) distance image of the target object through an interface such as a display screen or a user interface (UI) based on the data received through the communication interface 20.

[0040] Figure 2 This illustrates an embodiment according to the present disclosure. Figure 2 A diagram showing the operation of the image sensing device 10. Figure 3 It is shown that it is used for Figure 1 The histogram circuit 120 shown is a histogram based on the photon count values.

[0041] Reference Figure 2 The image sensing device 10 can emit pulsed light (i.e., emitted light EL) from a light source, receive light reflected from a target object (i.e., reflected light RL), use the emitted light (EL) and reflected light (RL) to calculate the round-trip time, and use the calculated round-trip time to measure the distance to the target object based on the principle of the constancy of the speed of light.

[0042] The emitted light (EL) can be projected onto the target object in the form of a point by the image sensing device 10. Depending on the distance to the target object, the position of the projected point can change, or the intensity of the emitted light (EL) can be increased or decreased. In some embodiments, the image sensing device 10 may operate in a photon counting mode for locating the point at the start of its operation.

[0043] SPAD element 101 can detect individual photons of reflected light (RL) reflected by a target object and can generate a SPAD pulse (DP) corresponding to the detected individual photon. TDC block 110 can measure the time from the start time of light emission synchronized with the emitted light (EL) to the response time of any SPAD 101 included in pixel 100 in response to the emitted light (EL). The measurement range (TR1) used for photon detection can be divided into a finite number of time compartments. In some embodiments, the measurement range (TR1) can be divided into eight time compartments.

[0044] Reference Figure 3 The horizontal axis of the histogram can represent time bins, which can each represent a subrange of photon arrival time. For example, the histogram circuit 120 can include a number of memory regions equal to the number of time bins, and each memory region can maintain independent values. Therefore, each index value of the memory region can correspond to a discretized observation time.

[0045] For example, the number of memory regions of histogram circuit 120 can be set to 8. The memory regions of histogram circuit 120 can be represented by memory indices 1 to 8 corresponding to the code of TDC data (TDC_OUT) (hereinafter referred to as "TDC code" or "digital code").

[0046] The vertical axis of the histogram represents the photon count value. The count value in the initial time chamber is relatively low and corresponds to background noise (P1). Reflected pulses with peak values ​​at some time points in the time chambers can be detected (P2).

[0047] Histogram circuit 120 can accumulate and store photon count values ​​in multiple time slots. Histogram circuit 120 can obtain distance information based on the time slot with the maximum hit count value among the multiple time slots (e.g., memory index "3").

[0048] A SPAD pulse (DP1) can respond at the time corresponding to TDC code "3" within the measurement range (TR1) of emitting the first emitted light (EL). In this case, the value stored in memory region "3" can be incremented by "+1". Then, a SPAD pulse (DP2) can respond at the time corresponding to TDC code "5" within the measurement range (TR2) of emitting the second emitted light (EL). In this case, the value stored in memory region "5" can be incremented by "+1". Thus, the operations of transmitting emitted light (EL), detecting SPAD pulses corresponding to reflected light (RL), and updating the histogram are repeatedly performed, resulting in the formation of... Figure 3 The histogram shown.

[0049] exist Figure 3 As shown in the histogram, it can be seen that the photon count value stored in memory region "3" represents the maximum value and has a larger value than that of non-adjacent memory regions (e.g., memory region "7"). As a result, signal processor 130 can determine that image sensing device 10 is receiving reflected light (RL) from the desired target object in the time domain corresponding to memory region "3".

[0050] exist Figure 2 and Figure 3 In one embodiment, the SPAD element 101 may not respond at the time of receiving photons caused by the reflected light (RL). Alternatively, there are cases where the SPAD element 101 responds at a time other than the time of receiving photons caused by the reflected light (RL). For example, it can be seen that the SPAD pulse does not respond at TDC code "3" corresponding to the reflected light (RL) within the measurement range (TR2), and the SPAD pulse (DP2) responds at TDC code "5" within the measurement range (TR2).

[0051] That is, it is impossible to limit the number of photons caused by reflected light (RL) incident on the SPAD element 101 within the measurement range (TR2). Furthermore, even if photons are incident on the SPAD element 101, the probability that the SPAD element 101 will respond (i.e., the photon detection efficiency PDE) may be less than 100%. In other words, in the absence of measurement processing in a dark environment, if photons with sensitive wavelengths are present, noise from a light source other than the emitted light (EL) (e.g., ambient light) may exist. Additionally, even if no photons are incident on the SPAD element 101, a response due to noise (e.g., dark counting) may occur with a specific probability.

[0052] As a result, it may be difficult to measure the desired time using only a single response. Therefore, the measurement of the time for repeatedly performing the transmission of emitted light (EL) and detecting the response of the SPAD pulse is performed, and the difference between the time range (with a high frequency relative to the light transmission time) and the light transmission time can be measured as the round-trip time (RTT) of the light.

[0053] The image sensing device 10 may require a considerable amount of memory to store the histograms of each pixel 100. Therefore, a partial histogram method or a sketch LiDAR method can be used to reduce the memory requirement.

[0054] According to the partial histogram method, the histogram is not generated all at once, but rather in stages by performing measurements multiple times. The partial histogram method creates a coarse histogram by observing the entire measurement range at coarse time intervals, detects peaks indicating the maximum value of the coarse histogram, and amplifies the detected peaks, thus enabling the creation of a fine histogram using fine time windows.

[0055] However, according to the partial histogram method, the frame rate may degrade as the number of measurements increases. Additionally, since the range observed through fine measurements must be wider than the time range corresponding to the two bins in the coarse measurements, temporal resolution may be sacrificed (reduced). Furthermore, the effectiveness of reducing memory capacity may deteriorate as the number of measurements increases.

[0056] On the other hand, the sketch lidar method does not count the values ​​of the bins corresponding to the time of the detected photon one by one. Instead, it calculates a feature function whose time variation is finer than the time interval of the bins, and then stores and accumulates the calculated values ​​in memory. Various functions can be applied to the feature function, but spline functions can be used. This feature function will be described below. Figure 4 To describe in more detail.

[0057] Figure 4 It is shown that it is used for Figure 1 A graph showing the characteristic functions of the image sensing device 10.

[0058] exist Figure 4 In this context, SPAD element 101 responds to photons N times. In this case, each response time (i.e., timestamp) can be defined as “x”. j (where j = 1, 2, ..., n)". Each memory region of the histogram circuit 120 can maintain the values ​​of the spline sketch (Z) defined by Equations 1 and 2 below. p,i ).

[0059] Formula 1

[0060]

[0061] In Equation 1, Φ p,i (x j ) can be defined as a characteristic function (referred to as "FF" in the figure described below).

[0062] Equation 1 can mean that whenever a timestamp is obtained, a discrete Fourier transform is performed to calculate the characteristic function, and the calculated value is stored in the memory area.

[0063] Formula 2

[0064]

[0065] In Equation 2, Δ is the total measurement time divided by M, where M is the number of sketches to be obtained. Here, "i" is denoted by "i = 1, ..., M" and can correspond to the observed statistics (i.e., the number of sketches). The number of sketches (M) can correspond to the number of memory regions and can have the same meaning as the number of bins used in the TCSPC method. "p" is the order of the spline function. "Φ" p "It is a (p) order spline function. Thus, the "characteristic function" can be represented as a function that depicts a sinusoidal waveform corresponding to the order of the spline function within the time stamp interval of N occurrences of the photon response.

[0066] For example, when the order of the spline function is 0, the spline function can be defined as Equation 3; when the order of the spline function is 1, the spline function can be defined as Equation 4.

[0067] Formula 3

[0068]

[0069] Formula 4

[0070]

[0071] exist Figure 4 In this context, (A) can represent the feature function when the order (p) of the spline function is set to zero "0". (A) can also represent an example case where the number of sketches (M) (i.e., the number of memory regions) is set to "4", and the four memory regions corresponding to the first to fourth features (feature1 to feature4) can store the corresponding accumulated values. For example, photons are detected within the time range of timestamps (ξ0, ξ1). Then, when the SPAD element 101 as... Figure 4 When the response is indicated by the arrow direction shown in (A), the value stored in memory region "1" can be increased by "+1". Therefore, when the order (p) of the spline function is "0", it can be performed according to the... Figure 1 The histogram operation is performed in the same way as the TCSPC operation described in the document.

[0072] exist Figure 4 In the diagram, (B) represents the eigenfunction when the order (p) of the spline function is set to "1". Unlike the case where the order (p) of the spline function is set to "0", the eigenvalues ​​added by a single photon detection are not represented by the two values ​​{0,1}.

[0073] Each memory region stores the value as a floating-point number. Then, for example, when in Figure 4 When a photon is detected at the arrow point shown in (B), according to Equation 1 above, the value stored in memory region "1" and the feature value stored in memory region "2" can simultaneously reach "+0.5".

[0074] In another example, from a circuit implementation perspective, maintaining values ​​stored in various memory regions as floating-point numbers may lead to an increase in circuit size so that values ​​can also be maintained as integers. In this case, for example, Figure 4 The eigenvalues ​​of the vertical axis of (B) can be increased (e.g., increased by a factor of 100), and the values ​​after the decimal point can be rounded to obtain integer values. In this case, if the value to be multiplied by the value on the vertical axis is small, the measurement will have an error due to the rounding error, so that an appropriate allowable error needs to be selected.

[0075] Figure 5A and Figure 5B It is shown that it is used for Figure 1 A diagram showing the histogram operation of the image sensing device 10.

[0076] Figure 5A An example of generating a histogram using the TCSPC method is shown. For instance, a histogram can be generated in the same memory region in the time domain corresponding to bin4, regardless of the timing of photon detection of the reflected light (RL). Furthermore, the effects of noise are not considered.

[0077] on the other hand, Figure 5B This illustrates an example of generating a histogram using a spline sketching method. For instance, photon detection of reflected light (RL) occurs during the measurement period (TP). If photon detection of reflected light (RL) occurs at a timing point (T1), all eigenvalues ​​of characteristic functions FF3 and FF4 can be generated.

[0078] Therefore, the histogram can change not only in the values ​​of memory region "4" but also in the values ​​of memory region "3" adjacent to memory region "4". If the photon detection of the reflected light (RL) occurs at timing point (T2), eigenvalues ​​of the characteristic function FF4 can be generated, and changes may occur in the histogram of memory region "4". If the photon detection of the reflected light (RL) occurs at timing point (T3), all eigenvalues ​​of characteristic functions FF4 and FF5 can be generated. Therefore, changes in the histogram can occur not only in the values ​​of memory region "4" but also in the values ​​of memory region "5" adjacent to memory region "4".

[0079] That is, the histogram can be changed depending on whether the time of photon detection of reflected light (RL) in the measurement period (TP) is closer to the timing point T1 or T3.

[0080] In this case, the center value of the histogram can be obtained for the three values ​​(C3, C4, C5) of memory regions (3, 4, 5) after excluding the value (Cb) caused by ambient light (or dark count). Therefore, the position of the reflected light (RL) waveform can be obtained in more detail than the TCSPC method.

[0081] Specifically, the center value (Tc) of the histogram can be calculated as shown in Equation 5 below.

[0082] Formula 5

[0083]

[0084] If the center value (Tc) of the reflected light (RL) is obtained based on the timing of the generated emitted light (EL) and the response characteristics of the SPAD pulse, then the optical TOF (time of flight) can be calculated. As shown in Equation 5, the center value (Tc) of the histogram can have a finer resolution than the “Δ” described in Equation 2.

[0085] That is, since the optical TOF is calculated by the center value (Tc) calculated using Equation 5, and then added to the values ​​of adjacent memory regions and averaged, the method used is similar to... Figure 5A In the histogram method, the same number of memory regions, Figure 5B The histogram method can have a finer temporal resolution. That is, when applying... Figure 5B When using this method, the number of memory regions required to obtain the desired time resolution can be less than [amount missing]. Figure 5A The situation.

[0086] However, the sketch lidar method requires high-speed operation of the TDC, thus increasing power consumption. Additionally, the memory capacity required per memory region can be increased, allowing for an increase in the memory capacity corresponding to one memory region. The effectiveness of reducing the number of memory regions may also be limited when the accuracy of the histogram is greatly restricted by short-range noise caused by photons in ambient light. There are limitations in reducing the number of memory regions as noise increases. To improve this, an image sensing device, described later, can be implemented.

[0087] Figure 6 This is a schematic diagram illustrating an image sensing device 10_1 according to another embodiment of the present disclosure.

[0088] Reference Figure 6 The image sensing device 10_1 may include a pixel 200, a TDC block 210, a histogram circuit 220, a signal processor 230, and a control signal generator 240. For simplicity, the components related to the image sensing device 10_1 will be omitted herein. Figure 1 For components with the same repeated descriptions, only refer to the descriptions of the same components. Figure 6 Detailed description and Figure 1 Different components.

[0089] Pixel 200 may include a SPAD element 201, a quenching circuit 202, and a buffer 203. The SPAD element 201 can detect individual photons of reflected light (RL) reflected by a target object and can generate a voltage pulse corresponding to the detected individual photon. The quenching circuit 202 can control the voltage of the SPAD element 201 and can output the resulting voltage to a sensing node (SN). The buffer 203 can generate a pulse signal based on an electrical signal generated according to the photons incident on pixel 200 and can output a pixel signal (PX_OUT).

[0090] TDC block 210 can calculate the time delay between the SPAD pulse output from pixel 200 and the reference pulse of the emitted light (EL), and can generate a digital code representing the time delay, namely, TDC data (TDC_OUT).

[0091] The TDC block 210 may obtain the timing point for generating the reference pulse of emitted light (EL) from a timing controller (described later) that controls the light source driver, or a predetermined time (e.g., a time prior to the start time of the frame) may be considered as the timing point for generating the reference pulse. According to another embodiment, the TDC block 210 may also obtain the generation time of the reference pulse from the control signal generator 240. According to one embodiment, the TDC block 210 may be included in pixel 200.

[0092] In this disclosure, TDC block 210 may include multiple TDCs. According to one embodiment, the number of multiple TDCs may correspond to the number of pixels 200, but the number of TDCs is not limited thereto. The operating timing points of the multiple TDCs can be controlled differently based on multiple TDC control signals (TDC_CNV1 to TDC_CNV8) received from the control signal generator 240. For example, the multiple TDCs may be sequentially activated based on the multiple TDC control signals (TDC_CNV1 to TDC_CNV8) to generate multiple digital codes (TDC_OUT0 to TDC_OUT7). In another example, the operating timing points of the multiple TDCs can be controlled differently based on the multiple TDC control signals (TDC_CNV1 to TDC_CNV4).

[0093] For example, multiple TDCs can be sequentially activated based on multiple TDC control signals (TDC_CNV1 to TDC_CNV4) to generate multiple digital codes (TDC_OUT0 to TDC_OUT15). In this disclosure, the number of TDC control signals and the number of digital codes are not limited thereto.

[0094] When TDC data (TDC_OUT) needs to be generated for pixel 200, each of the plurality of TDC control signals (TDC_CNV1 to TDC_CNV8) may have an enable level (e.g., logic level 1), and when TDC data (TDC_OUT) does not need to be generated for pixel 200, it may have a disable level (e.g., logic level 0). The detailed operation of TDC block 210 will be described in more detail with reference to the accompanying drawings described below.

[0095] Histogram circuit 220 can generate a histogram based on TDC data (TDC_OUT). Histogram circuit 220 may include controller 221 and histogram memory 222.

[0096] The controller 221 can be enabled based on an enable signal (CALC_EN) received from the control signal generator 240. For example, when the enable signal (CALC_EN) is at an enabled level (e.g., logic level 1), the controller 221 can perform histogram operations based on TDC data (TDC_OUT). On the other hand, when the enable signal (CALC_EN) is at a disabled level (e.g., logic level 0), the controller 221 can stop histogram operations.

[0097] Controller 221 performs histogram operations by using a spline function with fine time variations to overlap the values ​​of time bins corresponding to the time of photon detection with those of adjacent time bins. Controller 221 can discretize the sinusoidal feature function relative to the timestamp value and the number of photon detection counts, sequentially increasing and then decreasing the feature function across multiple time intervals. Thus, controller 221 can implement the sinusoidal feature function as a step-shaped function. According to an embodiment, the feature function can be implemented in a step shape that sequentially decreases and then increases across multiple time intervals. Controller 221 can determine the update (weight) value of the time bin based on the specific shape of the overlap between the corresponding feature function in each measurement period and the feature functions of neighboring measurement periods adjacent to that measurement period.

[0098] Controller 221 can generate information about memory regions (e.g., memory index information) and corresponding update values ​​for those memory regions based on TDC data (TDC_OUT). For example, controller 221 can determine the memory region to be updated in response to a TDC code and can generate an update value of "+1" for the corresponding memory region. The memory region information and update value (calc) generated by controller 221 can be stored in histogram memory 222.

[0099] The histogram memory 222 can update the values ​​stored in the corresponding memory regions by updating the value (e.g., +1). According to an embodiment, the histogram memory 222 may include an integer counter. The histogram memory 222 can perform an update operation on the corresponding memory region by incrementing the integer counter once in response to the index information of the memory region and the update value "+1". The values ​​stored in the histogram memory 222 can be output to the signal processor 230 in specific units (e.g., frame units).

[0100] The control signal generator 240 can generate multiple TDC control signals (TDC_CNV1 to TDC_CNV8) for controlling the operating timing points of multiple TDCs. These multiple TDC control signals (TDC_CNV1 to TDC_CNV8) can be enabled at different timing points. For example, the multiple TDC control signals (TDC_CNV1 to TDC_CNV8) can be enabled sequentially. The control signal generator 240 can also generate an enable signal (CALC_EN) for enabling the controller 221. For example, when the transmission signal (TX_ON, described later) is enabled during the emission of light (EL), the control signal generator 240 can enable the enable signal (CALC_EN).

[0101] Figure 7 It is shown that it is used for Figure 6 The diagram shows the histogram operation of the image sensing device.

[0102] Figure 7The implementation shows the histogram circuit 220 using Figure 4 and Figure 5B The example operation of performing histogram operations using the spline sketch method described in [the document] is shown.

[0103] Reference Figure 7 The order (p) of the spline function is set to "1" and the number of memory regions (M) is set to "8". The characteristic function can be represented as "FFi", where "i" can refer to the index of the memory region. For example, the characteristic function (FF) can be set to 8 (FF1 to FF8), and each characteristic function can be set as a first-order spline function with a different peak value. The value (val) shown on the vertical axis can represent the characteristic value of the characteristic function (FF) and can be a value substituted into a discrete function with an 8-step stepped shape. For example, the time width corresponding to one step of this stepped discrete function can be set to be less than the pulse width of the emitted light (EL).

[0104] Controller 221 can be enabled based on an enable signal (CALC_EN) during the measurement range (TR). That is, the enable signal (CALC_EN) can transition to a logic high level (e.g., a first level) at the timing point of the first emitted light (EL). When a response to a SPAD pulse is present, not only a TDC code but also a strobe signal indicating the occurrence of the response can be sent from TDC block 210 to controller 221. This strobe signal can be included in the TDC data (TDC_OUT).

[0105] The time resolution of the TDC data (TDC_OUT) can be set to 1 / 8 of the discrete function instead of 1 stage. That is, 8 stages of each discrete function can correspond to one TDC code. TDC block 210 can convert the response time of the SPAD pulse into a TDC code using the time resolution corresponding to the 8 stages of each discrete function. TDC block 210 can generate 8 TDC codes (0-7) using the time resolution of dividing the measurement time into 8 time segments.

[0106] To achieve the various characteristic functions (FF), the control signal generator 240 according to this disclosure can switch between eight timing points (i.e., eight stages) P1 to P8 for each emission of the emitted light (EL). For example, timing point P1 can be used during the first emission of the emitted light (EL), and timing point P2 can be used during the second emission of the emitted light (EL). Thus, timing point P8 can be used during the eighth emission of the emitted light (EL). In the ninth emission of the emitted light (EL), timing point P1 can be used again, and in the tenth emission of the emitted light (EL), timing point P2 can be used. Thus, timing points P1 to P8 can be used in a cyclical manner as described above.

[0107] First, at time point P1, if the TDC control signal (TDC_CNV1) is at a logic high level (e.g., the first level) before the start of the measurement range (TR), the first TDC can be enabled and the transition of the TDC code can begin. For example, when at time point (t e When a response to a SPAD pulse is detected, the TDC code at time point P1 can be "1". In this case, controller 221 can output the index "1" of the memory region to be updated and the value to be added "+1". Then, the value stored in memory region "1" of histogram memory 222 can be updated with "+1".

[0108] Subsequently, at time point P2, the TDC control signal (TDC_CNV2) can be activated with a time delay of 1 / 8 of the time resolution of the TDC data (TDC_OUT). That is, the TDC control signal (TDC_CNV2) can be activated with a specific time delay compared to the TDC control signal (TDC_CNV1). When the TDC control signal (TDC_CNV2) reaches a logic high level (e.g., a first level), the second TDC can be activated and the TDC code transition can begin.

[0109] For example, when at a time point (t) e When a response to a SPAD pulse is detected, the TDC code at time point P2 can be "1". In this case, controller 221 can output the index "1" of the memory region to be updated and the value to be added "+1". As a result, the value in memory region "1" of histogram memory 222 can be updated by "+1". Similarly, the histogram operation of controller 221 as described above can be performed at each of the time points P3 to P7.

[0110] Subsequently, at time point P8, the TDC control signal (TDC_CNV8) can be activated with a time delay of 7 / 8 of the time resolution of the TDC data (TDC_OUT) compared to time point P1. That is, the TDC control signal (TDC_CNV8) can be activated with a specific time delay compared to the TDC control signal (TDC_CNV7). When the TDC control signal (TDC_CNV8) reaches a logic high level (e.g., a first level), the eighth TDC can be activated and the TDC code transition can begin.

[0111] For example, when at a time point (t) eWhen a response to a SPAD pulse exists, the TDC code at the time point P8 when the response to the SPAD pulse is detected can be "0". In this case, the controller 221 can output the index "8" of the memory region to be updated and the value to be added "+1". As a result, the value in memory region "8" of the histogram memory 222 can be updated by "+1".

[0112] The controller 221 can output "+1" as the update value of the memory region to be updated. Therefore, even if the enable signal (CALC_EN) is deactivated during the time period (i.e., the time period when the enable signal (CALC_EN) is at a logic low level (second level), the controller 221 may not update the value of the histogram memory 222.

[0113] The probability of SPAD element 101 responding to a photon remains unchanged at timing points P1 to P8. Then, when the photon is at timing point (t... e When incident on SPAD element 101, it can be seen that the value of memory region 1 or memory region 8 is updated. It can be seen that the update frequency of the value of memory region 1 is 7:1 relative to the update frequency of the value of memory region 8.

[0114] As described above, the image sensing device 10_1 according to this embodiment does not need to fix the operation timing points of the multiple TDCs, and can sequentially control the operation timing points of the multiple TDCs according to the multiple TDC control signals (e.g., TDC_CNV1 to TDC_CNV8) received from the control signal generator 240.

[0115] By shifting the timing point for generating the TDC code and repeating the timing points (P1 to P8), the transmission and detection of the emitted light (EL), the operation of the SPAD pulse corresponding to the reflected light (RL), and the accumulation of histogram information can be repeatedly performed (e.g., Figure 7 The operation (calc), memory region index, and summation value) can be performed. At this time, the change in the timing of the operation of the TDC block 210 can be set more precisely than the time resolution of the TDC block 210.

[0116] As described above, since the timing of the operation of multiple TDCs differs for each detection of the SPAD pulse, measurements may occur where the code transition time of the TDC data (TDC_OUT) is inconsistent with the start and end of the measurement range (TR). The controller 221 can be enabled based on an enable signal (CALC_EN) during the measurement range (TR). Therefore, the controller 221 can detect not only the measurement range (TR) but also the specific information required to gating the TDC data (TDC_OUT) from the TDC block 210, so that the controller 221 can prevent the value of the histogram memory 222 from being updated.

[0117] Figure 8 It is shown that it is used for Figure 6 The diagram shows the histogram operation of the image sensing device.

[0118] In the above Figure 7 In one implementation, as an example, the characteristic function (FF) is described as 8 (FF1 to FF8) and each characteristic function is set as a first-order spline function with a different peak time.

[0119] However, Figure 8 The implementation of (A) shows that the characteristic function (FF) shown on the vertical axis is formed as a stepped shape with 4 steps, rather than a stepped shape with 8 steps. In each of the 4 steps of the characteristic function (FF), the operation cycle (cycle1 to cycle4) of the TDC can be controlled differently, and the measurement operation can be repeated to generate a histogram as a time average.

[0120] In addition, Figure 8 In the implementation of (B), the characteristic function (FF) shown on the vertical axis can be illustrated as a five-step staircase shape, and the time width of each step can be set differently. For example, after inputting the TDC code in cycle 1, the characteristic function (FF) can maintain the steps without changing for two segments, and then change to the second step in cycle 2. In each of the five steps of the characteristic function (FF), the operation cycle (cycle 1 to cycle 6) of the TDC can be controlled differently, but the measurement operation can be repeated in a specific step, or the measurement operation can be not performed in a specific step. For example, although the update value can be set to "1" in each operation cycle, the update value "3" can be generated by repeating each of operation cycles 3 and 4 three times.

[0121] Figure 9 It is shown that it is used for Figure 6 The diagram shows the histogram operation of the image sensing device.

[0122] Reference Figure 9 The controller 221 may have a characteristic function (FF) as a sensitivity function, which may have a positive (+) value or a negative (-) value. That is, the controller 221 may obtain a value of the characteristic function (FF) with different signs in response to a time slot. If the accumulated value of the photon count is positive (+), the characteristic function (FF) may have a positive (+) value; if the accumulated value of the photon count is negative (-), the characteristic function (FF) may have a negative (-) value. Therefore, information about two memory regions can be determined in one time slot.

[0123] For example, if at a time point (t) p1If photon detection occurs, the value of characteristic function (FF1) can be "+2" and the value of characteristic function (FF8) can be "-6". Therefore, the value of memory region "1" can be updated to "+2" and the value of memory region "8" can be updated (decreased) to "-6".

[0124] If at the time point (t) p2 If a photon is detected, the value of the characteristic function (FF1) can be changed to "+4", and the value of the characteristic function FF8 can be changed to "-4". Therefore, the value of memory region "1" can be updated to "+4", and the value of memory region "8" can be updated to "-4".

[0125] Additionally, at the time point (t) p3 When a photon is detected, the value of the characteristic function (FF1) can be changed to "+6", and the value of the characteristic function (FF2) can be changed to "+2". Therefore, the value of memory region "1" can be updated to "+6", and the value of memory region "2" can be updated to "+2".

[0126] Typically, when ambient light (also known as "background light") is strong, ambient light is added to the count, thus ensuring the memory width of each compartment.

[0127] However, according to this disclosure, when the ambient light is strong, the expected number of incident photons of ambient light in the time interval with a positive (+) sign on the discrete function is equal to the expected number of incident photons of ambient light in the time interval with a negative (-) sign on the discrete function, thus the average value of the characteristic function can be zero. According to the image sensing device of this disclosure, even when the ambient light is strong, there is no need to increase the memory width.

[0128] Figure 10 This illustrates another embodiment of the present disclosure for use with Figure 6 The diagram shows the histogram operation of the image sensing device.

[0129] According to Figure 10 In the image sensing device 10_1 of the embodiment, for the sake of simplicity, the following will be omitted: Figure 7 For components with the same repeated descriptions, only refer to the descriptions of the same components. Figure 10 Detailed description and Figure 7 Different components.

[0130] exist Figure 10 In the implementation, in order to clearly describe the sign of the value (calc) generated by the controller 221, in Figure 10 The symbols “b1++” and “b1--” will be used. Here, “b1” can represent the index information of the memory region, “++” can represent the update value to be added, and “--” can represent the update value to be subtracted.

[0131] For example, when at a time point (t) e When a photon is incident on SPAD element 101, the sum (b1) of memory region "1" can be output as "+7". The subtraction (b8) of memory region "8" can be output as "-1". As described above, this disclosure can reduce the bit width of the memory region by outputting positive and negative update values ​​in each time slot.

[0132] Figure 11 This is a schematic diagram illustrating an image sensing device 10-2 according to another embodiment of the present disclosure.

[0133] Reference Figure 11 The image sensing device 10_2 may include a pixel 200, a TDC block 210_1, a histogram circuit 220_1, a signal processor 230, and a control signal generator 240_1. For simplicity, the components related to the image sensing device 10_2 will be omitted in this document. Figure 6 For components with the same repeated descriptions, only refer to the descriptions of the same components. Figure 11 Detailed description and Figure 6 Different components.

[0134] TDC block 210_1 may include multiple TDCs. The operating timing of the multiple TDCs can be controlled differently based on multiple TDC control signals (TDC_CNV1 to TDC_CNV4) received from control signal generator 240_1. For example, based on the multiple TDC control signals (TDC_CNV1 to TDC_CNV4), the multiple TDCs can be sequentially activated to generate multiple digital codes (TDC_OUT0 to TDC_OUT15). The operation of TDC block 210_1 will be described in more detail with reference to the accompanying drawings described below.

[0135] Histogram circuit 220_1 can generate a histogram based on TDC data (TDC_OUT). Histogram circuit 220_1 may include a first controller 221_1, a second controller 221_2, and a histogram memory 222_1.

[0136] When a response to a SPAD pulse is present, a strobe signal indicating that not only a TDC code has appeared but a response has also appeared can be transmitted from TDC block 210_1 to first controller 221_1 and second controller 221_2. This strobe signal can be included in the TDC data (TDC_OUT).

[0137] The first controller 221_1 and the second controller 221_2 can be enabled based on an enable signal (CALC_EN). For example, when the enable signal (CALC_EN) is at an enabled level (e.g., logic level 1), the first controller 221_1 and the second controller 221_2 can perform histogram operations based on TDC data (TDC_OUT). When the enable signal (CALC_EN) is at a disabled level (e.g., logic level 0), the first controller 221_1 and the second controller 221_2 can stop histogram operations.

[0138] The first controller 221_1 can generate information about the memory region and the corresponding updated value of the memory region (referred to as the first value "calc1") based on TDC data (TDC_OUT). The second controller 221_2 can generate information about the memory region and the corresponding updated value of the memory region (referred to as the second value "calc2") based on TDC data (TDC_OUT). The first controller 221_1 and the second controller 221_2 can be enabled at different timing points based on an enable signal (CALC_EN). That is, the first value (calc1) can be output, and the second value (calc2) can be output after a certain time has elapsed since the output time of the first value (calc1).

[0139] The first controller 221_1 and the second controller 221_2 can determine the memory region corresponding to the TDC code, and can generate an update value of "+1" for the corresponding memory region. The update values ​​(calc1, calc2) generated by the first controller 221_1 and the second controller 221_2 and the information of the memory region can be stored in the histogram memory 222_1. The values ​​stored in the histogram memory 222_1 can be output to the signal processor 230 in specific units (e.g., in frames).

[0140] Additionally, the control signal generator 240_1 can generate multiple TDC control signals (TDC_CNV1 to TDC_CNV4) for controlling the operating timing points of multiple TDCs. These multiple TDC control signals (TDC_CNV1 to TDC_CNV4) can be activated at different timing points. For example, the multiple TDC control signals (TDC_CNV1 to TDC_CNV4) can be activated sequentially.

[0141] Control signal generator 240_1 can generate an enable signal (CALC_EN) to enable the first controller 221_1 and the second controller 221_2. For example, control signal generator 240_1 can generate a first enable signal to enable the first controller 221_1 and a second enable signal to enable the second controller 221_2. In this case, control signal generator 240_1 can enable the first enable signal and then enable the second enable signal after a certain period of time has elapsed from the enable time of the first enable signal.

[0142] exist Figure 6 In this implementation, when a photon is detected, the value of only one memory region can be updated. In contrast, in... Figure 11 In this implementation, two memory regions can be updated for a single photon detection operation, thereby preventing degradation of measurement accuracy.

[0143] Figure 12 It is shown that it is used for Figure 11 The diagram shows the histogram operation of the image sensing device.

[0144] In the image sensing device 10_2, for the sake of simplicity, the terms "and" will be omitted in this document. Figure 7 For components with the same repeated descriptions, only refer to the descriptions of the same components. Figure 12 Detailed description and Figure 7 Different components.

[0145] Reference Figure 12 The histogram circuit 220_1 can control the update operations of two controllers (221_1, 221_2) using a single TDC code value. The TDC block 210_1 can be twice as powerful as... Figure 7 The speed of the implementation method is used to generate TDC data (TDC_OUT).

[0146] The time resolution of the TDC data (TDC_OUT) can be set to 2 / 8 (i.e., 1 / 4), instead of one step (one stage) of the discrete function. That is, eight steps of the discrete function can correspond to two TDC codes. TDC block 210_1 can convert the response time of the SPAD pulse into a TDC code with a time resolution corresponding to four steps of each discrete function. The 16 codes (TDC_OUT0 to TDC_OUT15) of the TDC data can be generated within the measurement range (TR). That is, TDC block 210_1 can generate 16 TDC codes (0 to 15) with a time resolution that divides the measurement time into 16 segments.

[0147] because Figure 12 The implementation method can obtain two sets of histogram information during a single photon detection operation, thus... Figure 7 Compared to the previous implementation, only half the number of timing points (P1 to P4) are required. Figure 12 In this implementation, four timing points (four stages) P1 to P4 can be switched for each emission of the emitted light (EL) to achieve the various characteristic functions (FF). For example, timing point P1 can be used for the first emission, and timing point P2 can be used for the second emission. Then, timing point P4 can be used for the fourth emission in the same manner as described above. Then, timing point P1 can be used again for the fifth emission, and then timing point P2 can be used for the sixth emission. Thus, timing points (P1 to P4) can be used in a cyclical manner as described above.

[0148] First, at time point P1, before the measurement range (TR) is initiated, the TDC control signal (TDC_CNV1) reaches a logic high level (first level) to enable the first TDC and allow the TDC code transition to begin. For example, when at time point (t... e When a response to a SPAD pulse is detected, the TDC code at time point P1 can be "2". In this case, the controllers (221_1, 221_2) can output the index "1" of the memory region to be updated and the value to be added "+1". Then, the value in memory region "1" of the histogram memory 222_1 can be updated with "+1".

[0149] Subsequently, at time point P2, the TDC control signal (TDC_CNV2) can be activated with a time delay of 1 / 4 of the time resolution of the TDC data (TDC_OUT). That is, the TDC control signal (TDC_CNV2) can be activated with a specific time delay compared to the TDC control signal (TDC_CNV1). When the TDC control signal (TDC_CNV2) reaches a logic high level (first level), the second TDC can be activated and the TDC code transition can begin.

[0150] For example, when at a time point (t) e When a response to a SPAD pulse is detected, the TDC code at time point P2 can be "2". In this case, the controllers (221_1, 221_2) can output the index 1 of the memory region to be updated and the value to be added "+1". Then, the value in memory region "1" of the histogram memory 222_1 can be updated with "+1". Similarly, the operation of the controllers (221_1, 221_2) as described above can be performed at time point P3.

[0151] Subsequently, at time point P4, the TDC control signal (TDC_CNV4) can be activated with a time delay of 3 / 4 of the time resolution of the TDC data (TDC_OUT) compared to time point P1. That is, the TDC control signal (TDC_CNV4) can be activated with a time delay of a specific time compared to the TDC control signal (TDC_CNV3). When the TDC control signal (TDC_CNV4) reaches a logic high level (first level), the fourth TDC can be activated and the TDC code transition can begin.

[0152] For example, when at a time point (t) e When a response to a SPAD pulse is detected, the TDC code at the time point (P4) when the response to the SPAD pulse is detected can be "1". In this case, the first controller 221_1 can output the index "1" of the memory region to be updated and the value to be added "+1". Then, the value in memory region "1" of the histogram memory 222_1 can be updated with "+1". On the other hand, the second controller 221_2 can output the index "8" of the memory region to be updated and the value to be added "+1". Then, the value in memory region "8" of the histogram memory 222_1 can be updated with "+1".

[0153] Figure 13 and Figure 14 This illustrates another embodiment of the present disclosure for use with Figure 11 The diagram shows the histogram operation of the image sensing device.

[0154] according to Figure 13 The image sensing device 10_2 of the embodiment can represent an example case where two feature functions (FF) exist. Feature functions (FF1) and (FF2) can be combined to generate a combined feature function (FFC) that overlaps at the same operating rate (TDC). When performing histogram operations using the combined feature function (FFC), two or more update values ​​may be required.

[0155] In addition, according to Figure 14 The image sensing device 10_2 of the embodiment determines the overlapping combined feature function (FFC) by combining feature functions (FF1) with positive (+) values ​​and feature functions (FF2) with negative (-) values. Since the feature functions (FF1) with positive values ​​and feature functions (FF2) with negative values ​​are combined to set the combined feature function (FFC), the average value of the combined feature function (FFC) can be zero. Then, since the number of histogram counts of ambient light is canceled out, it is not necessary to increase the memory width even when the ambient light is strong.

[0156] Figure 15 It shows the basis Figure 11A diagram illustrating histogram operations in the implementation method.

[0157] exist Figure 15 In the image sensing device 10_2 shown, for the sake of simplicity, the components will be omitted in this document. Figure 12 Repeated descriptions of the same operation will only be referred to... Figure 15 Detailed description and Figure 12 Different operations are performed.

[0158] exist Figure 15 In the implementation, in order to clearly describe the signs of the values ​​(calc1, calc2) generated by the first controller 221_1 and the second controller 221_2, in Figure 15 The symbols “b1++” and “b1--” will be used. Here, “b1” can represent the index information of the memory region, “++” can represent the update value to be added, and “--” can represent the update value to be subtracted.

[0159] For example, when at a time point (t) e When a photon is incident on SPAD element 101, the sum (b1) of memory region "1" can be output as "+7". The subtraction (b8) of memory region "8" can be output as "-1". As described above, this disclosure can reduce the bit width of the memory region by outputting positive and negative update values ​​in each time slot.

[0160] therefore, Figure 15 The implementation method can suppress signal-to-noise ratio (SNR) degradation by updating the values ​​of two memory regions for a single photon detection.

[0161] Figure 16 This is a schematic diagram illustrating an image sensing device according to another embodiment of the present disclosure.

[0162] Reference Figure 16 The image sensing device 10_3 may include a pixel 200, a TDC block 210_2, a histogram circuit 220_2, a signal processor 230, and a control signal generator 240_2. For simplicity, the components related to the image sensing device 10_3 will be omitted in this document. Figure 6 For components with the same repeated descriptions, only refer to the descriptions of the same components. Figure 16 Detailed description and Figure 6 Different components.

[0163] Histogram circuit 220_2 can generate a histogram based on TDC data (TDC_OUT). Histogram circuit 220_2 can perform histogram operations by combining spline sketching methods and partial histogram methods. Histogram circuit 220_2 can estimate the peak position based on the histogram obtained through coarse measurement, and can use the estimated peak position to control the operation of fine measurement.

[0164] The histogram circuit 220_2 may include a coarse controller 221_3, a fine controller 221_4, and a histogram memory 222_2.

[0165] When the enable signal (CALC_EN) is enabled, the coarse controller 221_3 can generate not only information about the memory region based on the TDC data (TDC_OUT), but also an update value (referred to as the third value "calc_c") for the corresponding memory region. For example, the coarse controller 221_3 can determine the index of the memory region corresponding to the TDC code and can generate the update value of the corresponding memory region as "+1".

[0166] The fine controller 221_4 can generate a fine measurement value (referred to as the fourth value "calc_f") during the measurement period of a specific TDC code (i.e., the amplified segment TZ described later). The fine controller 221_4 can perform measurements at smaller time intervals than the coarse controller 221_3 and can generate fine measurement values ​​due to the effects of noise or pulse width.

[0167] The updated values ​​(calc_c, calc_f) generated by the coarse controller 221_3 and the fine controller 221_4, along with information from the memory region, can be stored in the histogram memory 222_2. The values ​​stored in the histogram memory 222_2 can be output to the signal processor 230 in specific units (e.g., in frames).

[0168] The operation of the histogram circuit 220_2 with the above configuration will be described below. Figure 17 and Figure 18 To describe in more detail.

[0169] Figure 17 It is shown that it is used for Figure 16 The diagram shows the histogram operation of the image sensing device.

[0170] In the image sensing device 10_3, for the sake of simplicity, the terms "and" will be omitted in this document. Figure 7 Repeated descriptions of the same operation will only be referred to... Figure 17 Detailed description and Figure 7 Different operations are performed.

[0171] Reference Figure 17The histogram circuit 220_2 can perform histogram operations by combining the characteristic function (FF) of a discrete function using an 8-step stepped shape with a partial histogram method.

[0172] exist Figure 17 Example of a coarse measurement being performed by coarse controller 221_3 is shown in (A). Figure 17 In a rough measurement of (A), the number of steps of the discrete function can be 8, and the number of memory regions can be 8.

[0173] For example, when at a time point (t) e When a response to a SPAD pulse is detected, the TDC code for detecting the response to the SPAD pulse can be 2. In this case, the coarse controller 221_3 can output the index 2 of the memory region to be updated and the value to be added "+1". Then, the value in memory region "2" of the histogram memory 222_2 can be updated "+1". Figure 17 The coarse measurement operation shown in (A) is similar to Figure 7 Since they are the same, their detailed description will be omitted.

[0174] exist Figure 17 Example of a fine measurement performed by the fine controller 221_4 is shown in (B). The fine characteristic functions (ffi) of the fine measurement (where “i” is the memory index) can be written in lowercase to distinguish the fine characteristic functions (ffi) from the characteristic functions (FFi) of the coarse measurement.

[0175] For example, the number of fine feature functions (ff) can be set to 8 (ff1 to ff8), and each fine feature function (ff) can be set as a first-order spline function with a different peak time. The value (val) shown on the vertical axis can represent the eigenvalue of the fine feature function (ff), and can be the value substituted into a discrete function with a step shape of two steps.

[0176] exist Figure 17 In the fine measurement of (B), the value of the magnified segment (TZ) indicated by the bidirectional arrow can be measured. Each step of the matching characteristic function (FF) has a fine characteristic function (ff1) with 2 steps. For example, this range can be set as the magnified segment (TZ) when a response pulse of reflected light (RL) is detected at TDC code "2".

[0177] The fine controller 221_4 can use two timing points (two stages) (p1, p2) to implement each fine characteristic function (ff).

[0178] The timing points (p1, p2) for fine measurements can be written in lowercase letters to distinguish them from the timing points (P1 to P8) for coarse measurements. For example, the fine controller 221_4 can use timing point P1 synchronously with one step of the fine characteristic function (ff1), and can use timing point P2 synchronously with two steps of the fine characteristic function (ff1). That is, timing point P2 can be activated after starting at timing point P1 and delaying for a specific time period.

[0179] Eight fine TDC codes can be generated based on timing point P1 within the amplification zone (TZ). Eight fine TDC codes can also be generated based on timing point P2 within the amplification zone (TZ). The fine measurement value (calc_f) generated by the fine controller 221_4 can be transferred to the histogram memory 222_2. The value stored in memory area "2" of the histogram memory 222_2 can be updated according to the fine measurement value (calc_f).

[0180] The number of steps in the discrete function used in fine measurement can be 2, and the number of memory regions can be 8. In this case, it may be sufficient for the coarse measurement to achieve a level of accuracy that does not result in signal loss due to fine measurement (rather than the accuracy of the final measurement). Because the effect of ambient light noise on each characteristic function is short-lived during fine measurement, fine measurement can achieve the same accuracy as TCSPC including a partial histogram.

[0181] Figure 18 It is shown that it is used for Figure 16 The diagram shows the histogram operation of the image sensing device.

[0182] In the image sensing device 10_2, for the sake of simplicity, the terms "and" will be omitted in this document. Figure 17 Repeated descriptions of the same operation will only be referred to... Figure 18 Detailed description and Figure 17 Different operations are performed.

[0183] exist Figure 18 In this implementation, the value (val) shown on the vertical axis can represent the fine characteristic function (ff). Similar to... Figure 17 Compared to the characteristic function shown as a discrete function formed by a two-step staircase shape, in Figure 18 In this implementation, the characteristic function may have a single-pulse shape. The fine characteristic function (ff) of the pulse shape may correspond to a step of the characteristic function (FF).

[0184] The fine-grained characteristic functions (ff1–ff8) can be sequentially activated at each step of the characteristic functions (FF1–FF8) within the amplification range (TZ). For example, at the timing point (t) with the highest peak value... pThe pulse of the first fine feature function (ff1) can be activated in response to feature function (FF1). Subsequently, the pulse of the fine feature function (ff2) can be activated in response to feature function (FF2) having a second peak value. Similarly, the pulses of fine feature functions (ff3 to ff8) respectively matched with feature functions (FF3 to FF8) can be activated sequentially in response to feature functions (FF3 to FF8).

[0185] When the fine feature function (ff) is enabled, the fine controller 221_4 can output more detailed information based on the TDC code (i.e., the detailed update value (calc_f) from the TDC code "2").

[0186] Figure 19 It is shown Figure 6 The timing diagram shown illustrates the operation of the control signal generator.

[0187] Figure 19 The implementation method can also be applied not only to Figure 6 The implementation method, and applied to Figure 11 and Figure 16 The implementation methods are as described herein, but are not limited thereto. For better understanding and description of this disclosure, Figure 19 The implementation method is applied to Figure 6 The implementation method.

[0188] Reference Figure 19 The operation of the control signal generator 240 can be reset at a timing point (T1) based on a reset signal (RST). The reset signal (RST) can be a signal generated by the timing controller (described later).

[0189] After a predetermined time has elapsed since the reset time of the control signal generator 240, the TDC control signal (TDC_CNV1) can be enabled at time point (T2), thereby initiating the TDC code transition. After a predetermined time has elapsed since time point (T2), the TDC control signal (TDC_CNV2) can be enabled at time point (T3), thereby allowing the TDC code transition. Similarly, after a predetermined time has elapsed since time point (T3), the TDC control signals (TDC_CNV3 to TDC_CNV8) are sequentially enabled until time point (T4) is reached, thereby allowing the TDC code transition.

[0190] Subsequently, when the transmission signal (TX_ON) is enabled at the timing point (T5), the target object can be illuminated using emitted light (EL). The transmission signal (TX_ON) can be a signal generated by the timing controller (described later). The control signal generator 240 can enable the enable signal (CALC_EN) at the timing point (T5). Then, the histogram operation of the histogram circuit 220 can be performed during the time period when the enable signal (CALC_EN) is enabled.

[0191] As described above, the image sensing apparatus according to this disclosure can generate histograms more accurately by sequentially activating the TDC control signals (TDC_CNV1 to TDC_CNV8) at different timing points (P1 to P8) (instead of using fixed TDC control signals (TDC_CNV1 to TDC_CNV8)). This operation can be achieved by applying, for example, a clock (CLK), a DLL (delay-locked loop), or a counter with a time resolution higher than TDC to the control signal generator 240.

[0192] Figure 20 This is a schematic diagram illustrating an imaging apparatus (CD) including an image sensing device according to an embodiment of the present disclosure.

[0193] Reference Figure 20 An imaging device (CD) can refer to, for example, a digital still camera for capturing still images or a digital video camera for capturing moving images. For example, an imaging device (CD) can be implemented as a digital SLR (DSLR) camera, a mirrorless camera, or a smartphone. An imaging device (CD) can include a device having both a lens and an image pickup element, enabling the device to capture (or photograph) a target object, thus creating an image of the target object. In some embodiments, the imaging device (CD) can be implemented as a lidar sensor.

[0194] The imaging device (CD) may include an image sensing device 10_4 and an image signal processor (ISP).

[0195] The image sensing device 10_4 can be a complementary metal-oxide-semiconductor image sensor (CIS) for converting optical signals into electrical signals, or include a complementary metal-oxide-semiconductor image sensor (CIS) for converting optical signals into electrical signals. The image sensing device 10_4 can use a time-of-flight (TOF) method to measure the distance to a target object. Figure 20 In the embodiments, the image sensing device 10_4 may represent the above-mentioned image sensing devices (10, 10_1, 10_2, 10_3).

[0196] The image sensing device 10_4 may include a light source (LS), a lens module (LM), a pixel array 300, a pixel driver 320, a readout circuit 330, a timing controller 340, and a light source driver 350. (See reference...) Figure 20 Pixel 310, included in pixel array 300, may represent pixel 200. Timing controller 340 may include control signal generators (240, 240_1, 240_2).

[0197] The light source (LS) can emit light toward the target object (TO) upon receiving a clock signal (MLS) from the light source driver 350. The light source (LS) can be a laser diode (LD) or light-emitting diode (LED) that emits light with a specific wavelength (e.g., infrared (IR) light or visible light), or it can be any combination of near-infrared laser (NIR), point light source, monochromatic light source combined with a white lamp or monochromator, and other laser sources.

[0198] For example, a light source (LS) may emit infrared (IR) light with a wavelength of 800 nm to 1000 nm. In some embodiments, the following description will be based on the emission of infrared light from the light source (LS). On the other hand, the light emitted from the light source (LS) may be pulsed light with a predetermined period, amplitude, and pulse width. Although for the sake of description, Figure 20 Only one light source (LS) is shown, but the implementation is not limited to this; multiple light sources (LS) may also be arranged near the lens module (LM).

[0199] The lens module (LM) collects light reflected from the target object (TO) and allows the collected light to be focused onto pixels 310 of the pixel array 300. For example, the lens module (LM) may include a focusing lens having a surface formed of glass or plastic, or another cylindrical optical element having a surface formed of glass or plastic. The lens module (LM) may include multiple lenses arranged around an optical axis.

[0200] The pixel array 300 may include a plurality of pixels 310 arranged continuously in a two-dimensional (2D) matrix structure, wherein the pixels 310 are arranged continuously in the column direction and in the row direction perpendicular to the column direction. Each pixel 310 can convert incident light received by the lens module (LM) into an electrical signal corresponding to the amount of incident light, and thus can use the electrical signal to output a pixel signal. In this case, the pixel signal may not indicate the color of the target object (TO), but may be a signal indicating the distance to the target object (TO).

[0201] Each unit pixel 310 may be an infrared pixel that generates a pixel signal by detecting incident light, including reflected light (RL) generated when emitted light (EL) from a light source (LS) is reflected from a target object (TO) and incident on the unit pixel 310. In some embodiments, the infrared pixel may be a depth pixel used to calculate the distance to the target object (TO).

[0202] A pixel array 300 with multiple pixels 310 can use a direct time-of-flight (TOF) method to detect the distance to a target object (TO). For reference, the direct TOF method can directly measure the round-trip time from the first time a pulse of light is emitted toward the target object (TO) to the second time the pulse of light reflected from the target object (TO) is incident. Therefore, the distance to the target object (TO) can be calculated by calculating the round-trip time and the speed of light.

[0203] The pixel driver 320 can drive the pixel array 300 under the control of the timing controller 340. For example, the pixel driver 320 can generate a control signal that can select and control the pixels 310 included in at least one of the multiple row lines of the pixel array 300. In addition, the pixel driver 320 can generate a recharge signal for controlling a recharge operation that injects charge into the sensing node of the SPAD element connected to the pixel 310.

[0204] A readout circuit 330 may be disposed on one side of the pixel array 300. It calculates the time delay between the pulse signals output from each pixel 310 and a reference pulse, and generates digital data corresponding to the time delay. The reference pulse may be a clock signal (MLS) pulse. The readout circuit 330 may include: digital logic circuitry configured to generate digital data by calculating the time delay between the pulse signals of each pixel 310 and the reference pulse; and an output buffer configured to store the generated digital data. The digital logic circuitry and the output buffer are collectively referred to below as the time-counting circuit (TDC) (200, 210, 210_1, 210_2). The readout circuit 330 may send the stored digital data to the image signal processor (ISP) under the control of the timing controller 340.

[0205] The readout circuit 330, under the control of the timing controller 340, processes the pixel signal (PX_OUT) output from the pixel array 300, and generates and stores depth data for detecting the distance to a target object (TO). Specifically, the readout circuit 330 calculates and stores the time of flight (TOF) corresponding to the SPAD pulses generated when each pixel 310 senses incident light including reflected light (RL). Under the control of the timing controller 340, the readout circuit 330 sends the stored TOF to the image signal processor (ISP).

[0206] The timing controller 340 controls the overall operation of the image sensing device 10_4. Specifically, the timing controller 340 generates clock and timing signals to control the operation of the pixel driver 320 and the light source driver 350. According to one embodiment, the timing controller 340 generates the clock and timing signals in response to a request from the image signal processor (ISP) or data received from the readout circuit 330. The timing controller 340 can control the optical power of the emitted light (EL) by controlling the light source driver 350 in response to a control signal received from the readout circuit 330.

[0207] Additionally, the timing controller 340 can control the activation or deactivation of the readout circuit 330, and can control the simultaneous or sequential transmission of digital data stored in the readout circuit 330 to the image signal processor (ISP). According to one embodiment, the timing controller 340 may include logic control circuitry, phase-locked loop (PLL) circuitry, timing control circuitry, communication interface circuitry, etc.

[0208] The light source driver 350 can generate a clock signal (MLS) that can drive the light source (LS) under the control of the timing controller 340. The light source driver 350 can control the waveform (e.g., period, amplitude, pulse width, etc.) of the emitted light (EL) output from the light source (LS).

[0209] The image signal processor (ISP) can control the operation of the image sensing device 10_4. In particular, the image signal processor (ISP) can determine the operating mode of the image sensing device 10_4 by analyzing the digital data received from the image sensing device 10_4, and can control the image sensing device 10_4 to operate in the determined mode.

[0210] The image signal processor (ISP) can perform image signal processing on the image data (IDATA) received from the image sensing device 10_4, and can generate processed image data. The image data (IDATA) may include the aforementioned time-of-flight (TOF). The image signal processor (ISP) can reduce noise in the image data and can perform various types of image signal processing (e.g., interpolation, lens distortion correction, etc.) to improve the image quality of the image data.

[0211] The image signal processor (ISP) can send the processed image data to a host device (not shown). The host device (not shown) can be a processor (e.g., an application processor) for processing the ISP image data received from the image signal processor (ISP), a memory (e.g., a non-volatile memory) for storing the ISP image data, or a display device (e.g., a liquid crystal display (LCD)) for visually displaying the ISP image data.

[0212] It is evident from the above description that the image sensing device based on TOF technology according to the embodiments of this disclosure can reduce the memory capacity required for system operation while improving its operating speed.

[0213] The embodiments disclosed herein can provide a variety of beneficial effects that can be identified directly or indirectly.

[0214] Although several exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments can be conceived based on what is described and / or shown in this disclosure. Furthermore, these embodiments can be combined to form additional embodiments.

[0215] Cross-reference to related applications

[0216] This patent application claims priority and benefit to Korean Patent Application No. 10-2024-0140231, filed on October 15, 2024, the entirety of which is incorporated herein by reference.

Claims

1. An image sensing device, the image sensing device comprising: A pixel that generates a pulse signal based on photons reflected from a target object; Multiple time-to-number converters (TDCs) generate digital codes corresponding to the time delay between the pulse signal and the reference pulse; as well as A histogram circuit that generates index information for memory regions and update values ​​for those memory regions based on the digital code. The multiple TDCs have different activation timing points.

2. The image sensing device according to claim 1, wherein The pixels include: A single-photon avalanche diode (SPAD) element that generates voltage pulses through a sensing node; A quenching circuit that controls the reverse bias voltage applied to the SPAD element to perform a quenching operation; and A buffer that samples the voltage pulses generated by the sensing node and converts the sampled voltage pulses into the pulse signal.

3. The image sensing device according to claim 1, wherein The histogram circuit: The value of the time slot corresponding to the time when the photon was detected in the timestamp segment overlaps with the value of the adjacent time slot; and When the pulse signal is responded to, the index information of the memory region and the update value of the memory region are generated based on the digital code.

4. The image sensing device according to claim 1, wherein The histogram circuit includes: A controller that generates the updated value of the memory region based on a feature function in a time segment obtained by dividing the timestamp segment by the number of memory regions; and A histogram memory that updates values ​​stored in the memory region using the updated values ​​in the memory region.

5. The image sensing device according to claim 4, wherein The characteristic function is a function that depicts a sinusoidal waveform corresponding to the order of the spline function within the timestamp segment.

6. The image sensing device according to claim 4, wherein The controller: The feature function is discretized relative to the number of photon detection counts and timestamp values ​​to output a discretized feature function with a stepped shape having different peak values ​​at each step; and Determine the update value of the memory region corresponding to each step formed in the stepped shape.

7. The image sensing device according to claim 6, wherein, The characteristic function has a form that increases sequentially and then decreases over multiple measurement periods; and The update value of the memory region is set to a positive value corresponding to each step of the characteristic function.

8. The image sensing device according to claim 6, wherein, The characteristic function has a form that decreases and then increases sequentially over multiple time intervals; and The update value of the memory region is set to a negative value corresponding to each step of the characteristic function.

9. The image sensing device according to claim 6, wherein The plurality of TDCs perform the conversion of the digital code such that the plurality of steps of the characteristic function correspond to a digital code.

10. The image sensing device according to claim 1, wherein The histogram circuit includes: A first controller generates information about a first memory region and a first update value for the first memory region based on the digital code; A second controller generates information about a second memory region and a second updated value for the second memory region based on the digital code; and A histogram memory that updates a value stored in the first memory region using the first update value of the first memory region and updates a value stored in the second memory region using the second update value of the second memory region.

11. The image sensing device according to claim 10, wherein, The first controller and the second controller are activated at different timing points.

12. The image sensing device according to claim 1, wherein The histogram circuit includes: A coarse controller that generates the index information of the memory region and the updated value of the memory region based on the digital code during the measurement period; A fine controller generates fine measurement values ​​for the memory region in an amplified segment corresponding to a specific digital code in the digital code that detects the photon; and A histogram memory updates the values ​​stored in the memory region in response to the output signals of the coarse controller and the fine controller.

13. The image sensing device according to claim 12, wherein The coarse controller: The feature function is discretized relative to the number of photon detection counts and timestamp values ​​to output a discretized feature function with a stepped shape having different peak values ​​at each step; and Determine the update value of the memory region corresponding to each step formed in the stepped shape.

14. The image sensing device according to claim 13, wherein, The precision controller: In the amplified section, multiple refined feature functions corresponding to each step of the feature function are generated. The multiple fine feature functions are activated at different timing points.

15. The image sensing device according to claim 14, wherein Each of the plurality of fine feature functions is formed according to a step shape with different peaks obtained when the various steps of the feature function are discretized.

16. The image sensing device according to claim 14, wherein Each of the plurality of fine feature functions has a single pulse shape corresponding to each step of the feature function.

17. The image sensing device according to claim 1, further comprising: A control signal generator sequentially enables multiple TDC control signals for controlling the multiple TDCs after a reset operation, and generates an enable signal for enabling the histogram circuit when emitted light shines on the target object.

18. An image sensing device, the image sensing device comprising: A pixel that generates a pulse signal based on photons reflected from the target object; A time-to-digital converter (TDC) block that generates a digital code corresponding to the time delay between the pulse signal and the reference pulse; as well as A histogram circuit generates a first characteristic function and a second characteristic function, each having a waveform corresponding to the order of the spline function within a timestamp segment of the photon response, and generates information about the memory region and an updated value for the memory region based on the first characteristic function and the second characteristic function. In this process, each of the first and second feature functions is discretized relative to the number of detection counts and timestamp values ​​of the photons, and each step has a stepped shape with different peak values.

19. The image sensing device according to claim 18, wherein The TDC block includes: A first TDC, which generates a plurality of first digital codes based on a first TDC control signal at a first timing point; and The second TDC generates multiple second digital codes based on the second TDC control signal at a second timing point after the first timing point.

20. The image sensing device according to claim 19, wherein The first feature function and the second feature function cause at least two or more memory region update values ​​to be generated when the pulse signal responds after the time bins corresponding to the time when the photon is detected overlap.

Images