imaging device

Abstract

An imaging apparatus is disclosed, comprising a sensor capable of detecting the distance to a target object based on a time-of-flight (TOF) method. In one embodiment, the imaging apparatus includes: an image signal processor, which may include: an analyzer configured to generate an intensity image of the object to be imaged by an image sensor based on a count value representing the number of detection events obtained by sensing reflected light from the object and detecting photons of the reflected light; analyze changes in the intensity value of the intensity image over a first time period; and analyze changes in temperature over the first time period; and an intensity attenuation determiner configured to determine the occurrence of intensity attenuation over time in the intensity image based on the analysis results of the analyzer.

Description

Technical Field

[0001] The technology and implementation disclosed in this patent document generally relate to an imaging device that includes a sensor capable of detecting the distance to a target object based on a time-of-flight (TOF) method. Background Technology

[0002] Image sensing devices capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With advancements in industries such as automotive, medical, computer, and communications, the demand for high-performance image sensing devices is growing across various fields, including smartphones, digital cameras, gaming consoles, the Internet of Things (IoT), robotics, surveillance cameras, and medical miniature cameras.

[0003] Image sensing devices can be used to acquire color or depth images of a scene. Recently, the Time-of-Flight (TOF) method has been employed to capture depth images of a scene by measuring the time it takes for light emitted from a light source (e.g., infrared light) to reflect off a target object and return to the sensor. This measurement allows for the calculation of the distance to the target object. Summary of the Invention

[0004] Various embodiments of the disclosed technology relate to an image signal processor capable of determining whether intensity attenuation occurs or not in an intensity image acquired by reflected light.

[0005] In one embodiment of the disclosed technology, an imaging apparatus includes: an image signal processor, which may include: an analyzer configured to generate an intensity image of an object to be imaged by an image sensor based on a count value representing the number of detection events obtained by sensing reflected light from an object and detecting photons of the reflected light, analyze changes in the intensity value of the intensity image over a first time period, and analyze changes in temperature over the first time period; and an intensity attenuation determiner configured to determine the occurrence of intensity attenuation of the intensity image over time based on the analysis results of the analyzer.

[0006] In another embodiment of the disclosed technology, an imaging device may include: a light source module configured to emit modulated light toward a target object; a time-of-flight (TOF) sensor including pixels configured to generate pixel data in response to detecting reflected light reflected from the target object; a temperature sensor configured to generate a temperature value corresponding to the temperature of the time-of-flight (TOF) sensor; and an image signal processor configured to generate an intensity image based on a count value representing the number of detection events in which the pixel detects photons of the reflected light, analyze changes in the intensity value of the intensity image over a first time period, analyze changes in the temperature value over the first time period, and determine the occurrence of intensity decay of the pixel.

[0007] It should be understood that the above general description of the disclosed technology and the following detailed description are illustrative and descriptive, and are intended to provide a further description of the implementation of the claimed disclosed technology. Attached Figure Description

[0008] The above and other features and advantages of the disclosed technology will become apparent from the following detailed description when considered in conjunction with the accompanying drawings.

[0009] Figure 1 This is a block diagram illustrating an example of an imaging apparatus according to some embodiments of the disclosed technology.

[0010] Figure 2 This illustrates some implementations based on the disclosed technology. Figure 1 The block diagram shown is for a time-of-flight (TOF) sensor.

[0011] Figure 3 This illustrates some implementations based on the disclosed technology. Figure 2 A schematic diagram illustrating an example of macropixels included in a time-of-flight (TOF) sensor.

[0012] Figure 4 This illustrates some implementations based on the disclosed technology. Figure 3 The circuit diagram for the example pixel shown.

[0013] Figure 5 This illustrates some implementations based on the disclosed technology. Figure 1 A block diagram of an example image signal processor is shown.

[0014] Figure 6 This illustrates some implementations based on the disclosed technology. Figure 5 The flowchart shown is an example operation flowchart of an image signal processor.

[0015] Figure 7A and Figure 7B This is a graph illustrating example curves showing the changes in intensity values ​​and temperature based on some embodiments of the disclosed technology.

[0016] Figure 8A and Figure 8B This is a graph illustrating example curves showing the changes in intensity values ​​and temperature based on other embodiments of the disclosed technology.

[0017] Figure 9 This illustrates some implementations based on the disclosed technology. Figure 1 A block diagram of an example computing device corresponding to an image signal processor. Detailed Implementation

[0018] This patent document provides an implementation and example of an imaging device that can detect the distance to a target object based on a time-of-flight (TOF) method. The TOF method can be used in a configuration to substantially solve one or more technical or engineering problems and mitigate limitations or drawbacks encountered in some imaging devices in the art. Some implementations of the disclosed technology involve an image signal processor that can determine whether intensity attenuation occurs in an intensity image acquired from reflected light. In view of the above problems, an image signal processor based on some embodiments of the disclosed technology and an imaging device including such an image signal processor can improve the reliability of depth images by filtering out chips where intensity attenuation has occurred.

[0019] Reference will now be made in detail to some embodiments of the disclosed technology, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. While the disclosed technology is susceptible to various modifications and alternatives, specific embodiments thereof are shown in the drawings by way of example. However, the disclosed technology should not be construed as being limited to the embodiments described herein.

[0020] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to specific embodiments, but includes various modifications, equivalents, and / or substitutions of the embodiments. Embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized through the disclosed technology.

[0021] In time-of-flight (TOF) sensors, such as LiDAR (Light Detection and Ranging) sensors, a histogram represents the distribution of time or signal strength relative to distance information. The sensor emits multiple light signals (e.g., lasers) over a period of time, and the arrival time of each reflected light signal is divided into time bins. The histogram count represents the frequency of arrival of the reflected signal within each time bin. As the histogram count gradually decreases, an intensity attenuation (IA) problem arises in such TOF sensors over time. This leads to a decrease in the corresponding intensity code value of the generated intensity image (e.g., a 2D image generated based on the histogram count), resulting in reduced sensing reliability.

[0022] To address this issue, the disclosed techniques can be implemented in some embodiments to provide an algorithm for detecting intra-avalanche diode (SPAD) irradiation (IA) in a Time-to-Flight (TOF) sensor using single-photon avalanche diode (SPAD) pixels and time-to-digital converter (TDC) circuitry. In some embodiments, such an algorithm may include identifying chips (e.g., image sensor chips) exhibiting IA by: (1) tracking intensity over a specific time period; (2) tracking temperature over a specific time period; (3) performing intensity fluctuation checks on a time-by-time basis; and (4) performing correlation analysis between temperature and intensity. In some implementations, to track intensity, the processor monitors the temporal variation of representative intensity values ​​of macropixels that can fully receive laser signals (e.g., vertical-cavity surface-emitting lasers, VCSELs) under normal conditions. In some implementations, to track temperature, the processor tracks the internal temperature of the TOF sensor over the same time period. In some implementations, intensity fluctuation checks may be performed as follows: Based on the average intensity value, the algorithm calculates the maximum rate of change of intensity per unit time. If the absolute value of the maximum rate of change of intensity exceeds a predefined threshold, a potential IA problem is indicated. In some implementations, the correlation analysis between temperature and intensity may be performed as follows. The algorithm calculates the correlation coefficient between temperature and average intensity values. In some implementations, a chip can be identified as having an IA problem if the following two conditions are met: (1) the correlation coefficient between temperature and intensity is below a threshold; and (2) the maximum rate of change of intensity is above a threshold. Figure 1 This is a block diagram illustrating an example of an imaging device 10 based on some embodiments of the disclosed technology.

[0023] Reference Figure 1The imaging device 10 can be, for example, a digital still camera for capturing still images or a digital video camera for capturing moving images. For example, the imaging device 10 can be implemented as a digital SLR (DSLR) camera, a mirrorless camera, a smartphone, etc. The imaging device 10 may include both a lens and an image pickup element, enabling the device to capture a target object and generate an image of the target object.

[0024] The imaging device 10 may include an image signal processor 100, a light source module 200, a distance sensor such as a time-of-flight (TOF) sensor 300, and a temperature sensor 400.

[0025] Imaging device 10 can use the TOF (Time of Flight) principle to measure the distance to target object 20. The TOF principle is based on the time it takes for light emitted by imaging device 10 to be reflected from target object 20 and then return to imaging device 10 to calculate the distance to target object 20.

[0026] The image signal processor 100 can process the generated image signal to perform noise cancellation and improve image quality. For example, the image signal processor 100 can process a depth image generated using the TOF sensor 300 to perform noise cancellation and improve image quality.

[0027] The depth image output from the image signal processor 100 may be stored in the internal or external memory of the imaging device 10 or a device equipped with the imaging device 10, either upon user request or through automation, or may be displayed on a display upon user request or through automation. Alternatively, the depth image output from the image signal processor 100 may be used to control the operation of the imaging device 10 or a device equipped with the imaging device 10.

[0028] Furthermore, the image signal processor 100 can collect pixel data (e.g., per-pixel data) received from the TOF sensor 300 for each pixel, and can thus generate at least one of a depth image or an intensity image indicating the distance to the target object 20.

[0029] The image signal processor 100 based on the disclosed technology can determine whether intensity decay has occurred by analyzing changes in intensity value over time and changes in temperature over time. (Refer to...) Figures 2 to 9 The components used to analyze intensity images and temperature changes are described in detail, as well as the process for determining whether intensity decay occurs or not based on the analysis results.

[0030] The light source module 200 can emit modulated light (ML) toward the target object 20 upon receiving a control signal from the image signal processor 100. The light source module 200 can be a laser diode (LD) or light-emitting diode (LED) for emitting light with a specific wavelength band (e.g., infrared (IR) light or visible light), or it can be any combination of a near-infrared laser (NIR), a point light source, a monochromatic light source combined with a white light lamp or monochromator, and other laser sources. For example, the light source module 200 can emit infrared (IR) light with a wavelength of 800 nm to 1000 nm. In some embodiments, the following description will be given assuming that the light source module 200 emits infrared (IR) light. The modulated light (ML) can be pulsed light modulated with predetermined modulation characteristics (e.g., waveform, wavelength, period, amplitude, frequency, phase, duty cycle, etc.). In one example, the light source module 200 can include a vertical-cavity surface-emitting laser (VCSEL) source. A VCSEL source can correspond to a semiconductor laser diode that emits laser light in a direction perpendicular to the surface.

[0031] The TOF sensor 300 may include one or more light sensors to receive and detect reflected light from the target object 20 that reflects modulated light (ML) from the light source module 200, such as... Figure 1 As shown. This detection of reflected light from the target object 20 by the TOF sensor 300 can be used to measure the distance to the target object 20 by measuring the time it takes for light to travel from the light source module 200 to the target object 20 and back to the TOF sensor 300. This allows the distance to the target object 20 to be calculated or determined under the control of the image signal processor 100. In some embodiments, the TOF method can be implemented as a direct TOF method. The direct TOF method directly measures the round-trip time from a first time when modulated light (ML) modulated using predetermined modulation characteristics is emitted to the target object 20 to a second time when reflected light (ML_R) reflected from the target object 20 is incident and detected. Therefore, the distance to the target object 20 can be calculated by calculating the round-trip time and the speed of light.

[0032] The TOF sensor 300 can be operated by receiving control signals from the image signal processor 100. In some embodiments, the TOF sensor 300 may include a plurality of photosensitive sensors arranged in pixels for sensing reflected light from the target object 20. Adjacent pixels in the TOF sensor 300 may be grouped to form macropixels (MPs) for TOF distance sensing. The TOF sensor 300 may provide the image signal processor 100 with pixel data generated during the calculation of the distance to the target object 20. Furthermore, the TOF sensor 300 may send the result of measuring the distance to the target object 20 to the image signal processor 100. For example, the TOF sensor 300 may send the round-trip time of the modulated light (ML) and the reflected light (ML_R) to the image signal processor 100.

[0033] Temperature sensor 400 can detect the temperature of TOF sensor 300 and provide the detected temperature data (TDATA) to image signal processor 100. For example, temperature sensor 400 can be located near a light sensor such as a macropixel (hereinafter referred to as "MP") of TOF sensor 300, which receives and detects reflected light (ML_R) from target object 20. However, the installation location of temperature sensor 400 within image signal processor 100 is not limited to this example and can be adjusted as needed.

[0034] Figure 2 This illustrates some implementations based on the disclosed technology. Figure 1 The block diagram shown is for a time-of-flight (TOF) sensor.

[0035] Reference Figure 2 The TOF sensor 300 may include macropixels (MPs), a time-to-digital converter (TDC) 310, and a histogram generation circuit 320. For example, the TOF sensor 300 may use a time-correlated single-photon counting (TCSPC) method to measure the time it takes for light to reflect from a target object and return to the sensor.

[0036] A macropixel (MP) can include multiple pixels (PX) arranged in both row and column directions to generate a pixel signal (PXOUT). In some implementations, the pulse signal of the pixel signal (PXOUT) generated by the macropixel (MP) can be referred to as a single-photon avalanche diode (SPAD) pulse. See below for further details. Figure 3 and Figure 4 Describes the detailed configuration and operation of macropixels (MPs).

[0037] The TDC 310 can calculate the time delay between the pulse of the pixel signal (PXOUT) output from the macropixel (MP) and the reference pulse of the modulated light (ML), and can convert the obtained pulse into a digital value corresponding to the time delay to generate TDC data (TDC_OUT).

[0038] For example, TDC 310 may include: digital logic configured to calculate the time delay between a pulse of a pixel signal (PXOUT) and a reference pulse to generate digital data; and an output buffer configured to store the generated digital data. TDC 310 may be collectively referred to as digital logic and the output buffer. TDC 310 may be a converter configured to convert information about the arrival times of photons received by multiple SPAD elements (described later) into digital signals. In some implementations, TDC 310 may be a TDC block comprising multiple TDCs.

[0039] Histogram generation circuit 320 can collect photoelectric conversion data over time based on TDC data (TDC_OUT) and can generate histograms and image data (IDATA). Here, the histogram can correspond to accumulated pixel data or the photoelectric conversion of a single pixel over time to display a graph of the resulting data (e.g., the result distribution). Histogram generation circuit 320 can accumulate and store timestamp data in a time interval based on TDC data (TDC_OUT) and can identify intervals with peaks. Although this example describes histogram generation circuit 320 and TDC 310 as separate components, other implementations are possible. For example, histogram generation circuit 320 can be integrated into TDC 310 as needed.

[0040] Figure 3 This illustrates some implementations based on the disclosed technology. Figure 2 A schematic diagram illustrating an example of macropixels included in a time-of-flight (TOF) sensor.

[0041] Reference Figure 3 A macro pixel (MP) can include multiple pixels (PX).

[0042] For example, (N×N) pixels (where “N” is a natural number greater than or equal to 2) can be grouped to form unit pixels (UPs). A macro pixel (MP) can include multiple unit pixels (UPs) arranged in a matrix configuration.

[0043] For example, a macro pixel (MP) can correspond to a group of pixels (PX) arranged in a (24×24) or (12×12) matrix. Although for ease of description, this example describes a macro pixel (MP) as including (24×24) or (12×12) pixels (PX) in both the row and column directions, other implementations are possible, and the number of pixels constituting a macro pixel is not limited to this.

[0044] Multiple pixels (PX) can detect incident reflected light (ML_R) and generate a pixel signal (PXOUT). Pixels (PX) can be grouped into macropixel units. Here, a pixel (PX) can be a single macropixel, or multiple macropixels can be arranged in an array structure.

[0045] The following description is provided under the premise that each pixel (PX) is a single-photon avalanche diode (SPAD) pixel used for detecting the distance of target object 20 based on a direct time-of-flight (TOF) method. However, the scope of the disclosed technology is not limited thereto. Reference will be made later. Figure 4 Describes the detailed configuration and operation of pixels (PX).

[0046] Figure 4 This illustrates some implementations based on the disclosed technology. Figure 3 The circuit diagram for the example pixel shown.

[0047] Reference Figure 4 A pixel (PX) may include a single-photon avalanche diode (SPAD), a quenching circuit (QC), a digital buffer (DB), and a recharge circuit (RC).

[0048] The SPAD can detect single photons of reflected light (RL) from a target object 20 and generate current pulses corresponding to the detected single photons. The SPAD can operate as a photodiode including a photosensitive PN junction. Since avalanche breakdown is triggered by single photons incident in Geiger mode with a reverse bias caused by a cathode-anode voltage exceeding the breakdown voltage, the SPAD can generate current pulses.

[0049] Geiger mode allows the SPAD to be biased with a reverse voltage greater than its breakdown voltage, enabling the detection of single photons within the SPAD. In Geiger mode, the electric field applied to the amplification layer is sufficiently strong that even the absorption of a small number of photons triggers avalanche current breakdown, resulting in a large output current. This allows for the detection of single photons. In the following text, the process by which avalanche breakdown is triggered by a single photon and generates a voltage pulse will be referred to as the avalanche process.

[0050] One terminal of the SPAD can receive a first bias voltage (VDDPX) used to apply a reverse bias voltage (hereinafter referred to as the "operating voltage") higher than the breakdown voltage to the SPAD. For example, the first bias voltage (VDDPX) can be a positive (+) voltage whose absolute value is lower than the absolute value of the breakdown voltage. The other terminal of the SPAD can be connected to a sensing node (Ns), and the SPAD can output a current pulse generated by detecting a single photon to the sensing node (Ns).

[0051] Although the disclosed embodiments have shown SPADs as light-receiving elements (i.e., light-detecting elements) for pixels (PXs), the scope of these embodiments is not limited to SPADs. That is, in addition to SPADs, various elements operating in Geiger mode, such as avalanche photodiodes (APDs), silicon photomultiplier tubes (SiPMs), etc., can be used as light-receiving elements for pixels (PXs).

[0052] The quenching circuit (QC) controls the voltage of the SPAD. After a voltage pulse is generated due to avalanche breakdown and the voltage of the sensing node (Ns) changes, the quenching circuit (QC) can perform a quenching operation to restore the voltage of the sensing node (Ns) to Geiger mode.

[0053] After the time when the avalanche process might have occurred has passed, the quenching transistor (QX) of the quenching circuit (QC) is turned on based on the quenching control signal (QCS) to electrically connect the sensing node (Ns) to the ground voltage. Therefore, the reverse bias voltage applied to the SPAD drops below the breakdown voltage, and the avalanche process may be quenched (or stopped).

[0054] A digital buffer (DB) can sample analog current pulses to be applied to a sensing node (Ns) and convert the sampled analog current pulses into digital pulse signals. The DB can generate the pulse signal at a frequency based on the frequency of the received photons. Here, the sampling method can be a method of converting the current pulse into a pulse signal with a logic level of 0 or 1 based on whether the level of the current pulse is equal to or higher than a threshold level, but the scope of the disclosed technology is not limited thereto. Therefore, the pulse signal output from the digital buffer (DB) can be transmitted to the TDC310 as a direct pixel signal (PXOUT).

[0055] The recharge circuit (RC) can inject charge into the sensing node (Ns) so that the SPAD can re-enter the Geiger mode that can trigger avalanche breakdown after being quenched by the quenching circuit (QC) during the avalanche process.

[0056] For example, the recharge circuit (RC) may include a switch (e.g., a transistor) that can selectively connect a second bias voltage to the sensing node (Ns) based on a recharge control signal. When the switch is turned on, the voltage at the sensing node (Ns) can reach the second bias voltage.

[0057] For example, the sum of the absolute values ​​of the second bias voltage and the first bias voltage (VDDPX) can be greater than the absolute value of the breakdown voltage, and the second bias voltage can be a negative (-) voltage. Therefore, the SPAD can enter Geiger mode, and the avalanche process can continue when a single photon is received at the next time point.

[0058] Although this embodiment has disclosed an example of implementing the quenching circuit (QC) and recharge circuit (RC) as active devices for ease of description, it should be noted that the quenching circuit (QC) and recharge circuit (RC) can also be implemented as passive devices as needed. For example, the quenching transistor (QX) of the quenching circuit (QC) can also be replaced by a resistor.

[0059] Figure 5 This illustrates some implementations based on the disclosed technology. Figure 1 A block diagram of an example image signal processor is shown.

[0060] Reference Figure 5 The image signal processor 100 may include an analyzer 110, an intensity attenuation determiner 120, and a pixel processor 130.

[0061] Here, the analyzer 110 can track changes in intensity values ​​over time based on image data (IDATA) generated by collecting pixel data of each pixel received from the TOF sensor 300. Furthermore, the analyzer 110 can receive temperature data (TDATA) from the temperature sensor 400 and track changes in temperature values ​​over time. The analyzer 110 can track changes in both intensity and temperature values ​​over time and generate analysis data (ADATA).

[0062] In one implementation, the analyzer 110 can track qualitative information related to intensity in the image data (IDATA). Here, the qualitative information can be the result of a qualitative assessment of intensity, or it can be information indicating whether intensity decay is reflected over time in the image captured by the TOF sensor 300. For example, the qualitative information can indicate the degree to which the intensity value is reflected in the captured image.

[0063] In another embodiment, the analyzer 110 can track quantitative information related to the intensity in the image data (IDATA). Here, the quantitative information can be the result of a quantitative assessment of the intensity, and can be information obtained by quantifying the intensity values ​​in the image captured by the TOF sensor 300 that decay over time. For example, the quantitative information can be information obtained by quantifying the degree of decay of the intensity code reflected in the captured image.

[0064] The analyzer 110 can generate an intensity image based on image data (IDATA) generated when each macropixel (MP) included in the TOF sensor 300 receives reflected light (ML_R). Here, "intensity image" can be an image indicating the number of counts obtained when each macropixel (MP) included in the TOF sensor 300 receives reflected light (ML_R) and a single-photon avalanche diode (SPAD) responds to the reflected light (ML_R). The term "intensity" can indicate the number of photons generated when the SPAD of a macropixel included in the target kernel receives and responds to the reflected light (ML_R).

[0065] For example, for a specific time period when the light source is in the ON (on) state, the analyzer 110 can accumulate and store representative intensity values ​​based on the number of laser irradiations identified by the SPAD included in the macropixel (MP), and can analyze the stored representative intensity values.

[0066] In some implementations, analyzer 110 can analyze the degradation order of macropixels (MPs) in the intensity image. For example, analyzer 110 can determine the region of interest (ROI) from the intensity image and analyze the change over time in the intensity representative value of the laser projection region where the ROI is set and the laser is strongly projected. Here, the intensity representative value can represent the average intensity in the ROI.

[0067] The reflected light (ML_R) from the target object 20 can be projected in the form of points by the TOF sensor 300 included in the imaging device 10. Here, the aforementioned "laser projection area" can refer to the area projected onto the pixel in the form of points by the light source. In addition, the region of interest (ROI) can refer to the local area where the TDC 310 is enabled or disabled. Each ROI used to detect part or all of the projection points can be flexibly adjusted.

[0068] When a specific usage period has elapsed, depending on defects in the SPAD structure or the TDC 310 (or histogram generation circuit 320), the TOF sensor 300 can continuously reduce the histogram count value (i.e., the number of histogram counts). Furthermore, even if the power is restarted, the histogram generation operation can begin from the reduced count value, and the degree of reduction in the count value can increase proportionally to the intensity of the light source. Since the intensity image is generated by adding the histogram count values, the intensity value also decreases as the histogram count value decreases. As described above, the phenomenon of SPAD or TDC 310 degradation and a decrease in intensity value will be defined hereinafter as "intensity decay".

[0069] Therefore, the analyzer 110 can set the average intensity value (hereinafter referred to as the "average intensity value") and the temperature value as factors for determining whether the intensity has decayed, and can analyze changes in the intensity value (i.e., the trend of intensity value changes). This will be discussed later. Figure 6 , Figure 7A , Figure 7B , Figure 8A and Figure 8B The operation of analyzer 110 in tracking and analyzing intensity values ​​over time is described in more detail.

[0070] The intensity attenuation determiner 120 can determine whether the intensity of a macropixel (MP) has attenuated based on the analysis data (ADATA) received from the analyzer 110, and can generate attenuation data (IA) that includes information indicating that intensity attenuation has occurred.

[0071] The intensity attenuation determiner 120 can compare analytical data (ADATA) with preset thresholds and generate attenuation data (IA) based on the comparison results. For example, the intensity attenuation determiner 120 can use a correlation coefficient (first condition) and the rate of change of intensity value per unit time (second condition) to determine whether intensity attenuation occurs or not. When both the first and second conditions are met, the intensity attenuation determiner 120 can determine that intensity attenuation has occurred.

[0072] The intensity attenuation determiner 120 can generate attenuation data (IA) through correlation analysis. The intensity attenuation determiner 120 can use correlation coefficients to determine the linear correlation between the average temperature value and the average intensity value. For example, in some implementations, the Pearson correlation coefficient can be used. However, the Kendall or Spearman correlation coefficients can also be used, and the range of correlation coefficients used to analyze this correlation is not limited to these.

[0073] For example, the intensity attenuation determiner 120 can calculate the covariance between the temperature value and the average intensity value, and can calculate the correlation coefficient based on the calculated covariance. Here, the covariance can be defined as the value obtained by numerically representing the common part of the distribution of the average intensity value and the distribution of the temperature value. That is, the intensity attenuation determiner 120 can obtain the correlation coefficient by dividing the covariance value by the standard deviation of the temperature value and the standard deviation of the average intensity value.

[0074] When the correlation coefficient is less than a preset first threshold, the intensity attenuation determiner 120 can determine that intensity attenuation has occurred. Here, the first threshold can be determined in advance through experiments.

[0075] The intensity attenuation determiner 120 can calculate the absolute value of the rate of change of the intensity value per unit time (hereinafter referred to as the first value), and can calculate the second value by dividing the first value by the maximum value of the intensity value. Here, the maximum value can refer to the value with the highest intensity code value in the region of interest (ROI) used to determine the intensity attenuation. When the second value is higher than a preset second threshold, the intensity attenuation determiner 120 can determine that intensity attenuation has occurred. Here, the second threshold can be determined in advance experimentally.

[0076] Furthermore, the pixel processor 130 can deactivate target macropixels where intensity attenuation has occurred based on attenuation data (IA) received from the intensity attenuation determiner 120. Here, the target macropixel may correspond to one of the macropixels included in the TOF sensor 300 that serves as a target for determining whether a macropixel is an attenuating pixel (where intensity attenuation has occurred).

[0077] For example, if the target macropixel (MP) is determined to be an intensity-attenuated pixel (where intensity attenuation has occurred), the imaging apparatus 10 may not use the pixel data of the target macropixel. Here, the operation of deactivating the target macropixel can be performed by a pixel corrector (not shown). The pixel corrector (not shown) may be included within the pixel processor 130, but is not limited thereto.

[0078] In another embodiment, the image signal processor 100 can perform operations at the module level to analyze intensity images, determine whether intensity decay has occurred or not, and / or process macropixels where intensity decay has occurred. Therefore, during the module testing phase, among multiple modules that include macropixels, modules that include macropixels expected to have intensity decay after a specific time period (i.e., dynamic dead zone macropixels, DDMPs) can be identified in advance. This allows appropriate action to be taken against this defective module.

[0079] In another embodiment, the image signal processor 100 can also be used during the wafer inspection stage to analyze the intensity image, determine whether intensity attenuation has occurred or not, or process macropixels in which intensity attenuation has occurred.

[0080] As described above, in addition to detecting macropixels that experience intensity decay on each chip including the imaging device 10, defect prevention (anti-defect) measures can be taken based on the analysis results of intensity decay to reduce the likelihood of defective macropixels occurring in the future.

[0081] Figure 6 This illustrates some implementations based on the disclosed technology. Figure 5 The flowchart shown is an example operation flowchart of an image signal processor. Figure 7A and Figure 7B This is a graph illustrating example curves showing the changes in intensity values ​​and temperature based on some embodiments of the disclosed technology. Figure 8A and Figure 8B This is a graph illustrating example curves showing the changes in intensity values ​​and temperature based on other embodiments of the disclosed technology.

[0082] Reference Figure 6 The analyzer 110 can receive the reflected light (ML_R) from each macropixel (MP) included in the TOF sensor 300 and can generate an intensity image based on the count obtained through the SPAD reaction. The analyzer 110 can capture an intensity image for a predetermined time while the light source is on, and can simultaneously track changing intensity values. The analyzer 110 can analyze temperature data (IDATA) received from the temperature sensor 400 and can track changes in temperature values ​​over time (operation S1).

[0083] Analyzer 110 generates intensity data by analyzing the changing trends of intensity values ​​of selected pixels in an intensity image captured at each time point (e.g., at specific time intervals). For example, analyzer 110 can extract the average intensity values ​​of selected pixels (e.g., pixels with the top few percentage intensity values) to analyze the changing trends of the intensity values ​​of selected pixels over time within the intensity image. Here, "selected pixels" can refer to those pixels with the highest intensity codes (e.g., the top few percentages) throughout the entire intensity image.

[0084] Analyzer 110 can analyze the changing patterns of average intensity values ​​and temperature values ​​over time. For example... Figure 7A As shown in the graph, the average intensity value in the ROI (region of interest) where the laser is strongly projected generally tends to increase over time. Figure 7B As shown in the curve, when illuminated by a light source, the temperature value also shows an overall upward trend over time.

[0085] Similarly, as Figure 8A As shown in the graph, the average intensity value in the ROI (region of interest) where the laser is strongly projected exhibits an overall decreasing pattern over time. Figure 8B As shown in the graph, when illuminated by the light source, the temperature value also exhibits a pattern of overall decrease over time. For example, Figure 8A and Figure 8B This is a graph depicting the process of first turning off the power supply and then restarting it with the power supply being switched on.

[0086] That is, refer to Figure 7A and Figure 7B The curve shows that the average intensity value changes over time in a pattern similar to the temperature value changes over time. (Refer to...) Figure 8A and Figure 8B Even when the power is restarted, the patterns of change of the average intensity value over time and the patterns of change of the temperature value over time can be similar to each other.

[0087] For ease of description, Figures 7A to 8B The graph shown is merely an example, and Figures 7A to 8B The waveform, time, quantity, etc. of the curves shown can vary significantly based on the analysis results of the analyzer 110, but are not limited thereto.

[0088] Therefore, the intensity attenuation determiner 120 can calculate the correlation coefficient between the average intensity value and the temperature value, and can determine the calculated correlation coefficient as a first condition for determining whether intensity attenuation occurs or not (operation S2). The intensity attenuation determiner 120 can determine whether intensity attenuation occurs or not by comparing the correlation coefficient with a preset first threshold (operation S3).

[0089] For example, when the correlation coefficient is less than a first threshold, the intensity attenuation determiner 120 can determine that intensity attenuation has occurred. On the other hand, when the correlation coefficient is equal to or higher than the first threshold, the intensity attenuation determiner 120 can determine the corresponding pixel as a normal pixel without intensity attenuation.

[0090] The intensity attenuation determiner 120 can calculate the rate of change of the intensity value per unit time and can determine the calculated rate of change as a second condition for determining whether intensity attenuation occurs or not (operation S4). That is, the intensity attenuation determiner 120 can convert the change of the intensity value per unit time into an absolute proportion of the maximum intensity value. The intensity attenuation determiner 120 can compare the rate of change of the intensity value with a preset second threshold and can determine whether intensity attenuation occurs or not based on the comparison result (operation S5).

[0091] For example, when the rate of change of the intensity value is greater than a preset second threshold, the intensity attenuation determiner 120 can determine that intensity attenuation has occurred.

[0092] That is, when there is a significant difference between the intensity value of a normal macropixel and the intensity value of a macropixel that has experienced intensity attenuation, the intensity attenuation determiner 120 can determine that intensity attenuation has occurred. On the other hand, when the rate of change of the intensity value is less than a second threshold, the intensity attenuation determiner 120 can determine that the pixel is a normal pixel that has not experienced intensity attenuation.

[0093] The intensity attenuation determiner 120 can determine whether intensity attenuation has occurred based on a first condition, or based on a second condition.

[0094] In some embodiments of the disclosed technology, the intensity attenuation determiner 120 can determine that intensity attenuation has occurred when both the first and second conditions are met (operation S6). For example, after the same amount of time has elapsed, the intensity attenuation determiner 120 can determine that the intensity attenuation is maximum when the average intensity value increases most sharply. In other words, it can be seen that the chip with the maximum intensity attenuation exhibits more severe degradation in reliability.

[0095] Figure 9 It is shown that... Figure 1 A block diagram of an example of a computing device 1000 corresponding to the image signal processor 100.

[0096] Reference Figure 9 The computing device 1000 can represent a device for performing... Figure 1 An implementation of the hardware configuration for the operation of the image signal processor 100.

[0097] The computing device 1000 can be mounted on a separate chip from the chip on which the time-of-flight (TOF) sensor is mounted. In one embodiment, the chip on which the time-of-flight (TOF) sensor is mounted and the chip on which the computing device 1000 is mounted can be implemented in a single package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0098] The computing device 1000 may include a processor 1010, a memory 1020, an input / output (I / O) interface 1030, and a communication interface 1040.

[0099] Processor 1010 can handle execution Figure 1 The data and / or instructions required for the operation of the components (110, 120, 130) of the image signal processor 100 described herein. That is, processor 1010 may refer to image signal processor 100, but the scope of the disclosed technology is not limited thereto.

[0100] The memory 1020 may store data and / or instructions required to perform the operations of the components (110, 120, 130) of the image signal processor 100, and may be accessed by the processor 1010. For example, the memory 1020 may be volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) or non-volatile memory (e.g., programmable read-only memory (PROM), erasable PROM (EPROM), EEPROM (electrically erasable PROM), flash memory, etc.).

[0101] That is, the computer program for performing the operation of the image signal processor 100 disclosed in this document is recorded in the memory 1020 and executed and processed by the processor 1010, thereby realizing the operation of the image signal processor 110.

[0102] Input / output (I / O) interface 1030 is an interface that connects external input devices (e.g., keyboard, mouse, touch panel, etc.) and / or external output devices (e.g., display) to processor 1010 to allow sending and receiving data.

[0103] The communication interface 1040 is a component that can send and receive various data with external devices (e.g., application processors, external memory, etc.) and can be a device that supports wired or wireless communication.

[0104] As can be clearly seen from the above description, image signal processors and imaging devices including image signal processors based on some embodiments of the disclosed technology can improve the reliability of depth images by filtering out chips with intensity attenuation.

[0105] The implementation of the disclosed technology can provide various effects that can be directly or indirectly identified through the aforementioned patent documents.

[0106] While several exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments can be designed based on the description and / or illustration in this patent document. Therefore, the scope of the disclosed technology should not be limited to the above-described embodiments, but should include their equivalents.

[0107] Cross-reference to related applications

[0108] This patent document claims priority and benefit to Korean Patent Application No. 10-2025-0038974, filed on March 26, 2025, the disclosure of which is incorporated herein by reference in its entirety as part of the disclosure of this patent document.

Claims

1. An imaging device, the imaging device comprising: An image signal processor, comprising: An analyzer generates an intensity image of the object to be imaged by an image sensor based on a count value representing the number of detection events obtained by sensing reflected light from the object and detecting photons of the reflected light; analyzes changes in the intensity value of the intensity image over a first time period; and analyzes changes in temperature over the first time period; and An intensity attenuation determiner determines the occurrence of intensity attenuation of the intensity image over time based on the analysis results of the analyzer.

2. The imaging apparatus according to claim 1, wherein the imaging apparatus further comprises: At least one macro pixel, wherein the at least one macro pixel includes a single-photon avalanche diode (SPAD) element for sensing reflected light from the object and detecting photons of the reflected light.

3. The imaging apparatus according to claim 1, wherein the imaging apparatus further comprises: At least one macro pixel, wherein the at least one macro pixel comprises a plurality of pixels arranged in a matrix configuration for sensing the reflected light. in, The plurality of pixels are grouped into a predetermined number of pixel groups to form a unit pixel.

4. The imaging device according to claim 1, wherein, The analyzer: Determine the region of interest (ROI) in the intensity image; and Analyze the average intensity values ​​above a predetermined threshold in the areas within the ROI where light is projected.

5. The imaging apparatus according to claim 1, wherein, The analyzer: Capture the intensity image within a predetermined time period; and Analyze the trend of the average intensity value of the selected pixels in the intensity image captured at each time point over time.

6. The imaging apparatus according to claim 5, wherein, The selected pixels are pixels with intensity values ​​corresponding to a predetermined number of the highest intensity codes in the intensity image.

7. The imaging apparatus according to claim 1, wherein, The intensity attenuation determiner: The correlation coefficient between temperature values ​​and average intensity values ​​is used to determine the occurrence of intensity decay.

8. The imaging apparatus according to claim 7, wherein, The intensity attenuation determiner: Calculate the covariance between the average intensity value and the temperature value; and The correlation coefficient is calculated by dividing the calculated covariance by the standard deviation of the temperature value and the average intensity value.

9. The imaging apparatus according to claim 7, wherein, The intensity attenuation determiner: The occurrence of intensity decay is determined in response to the correlation coefficient being less than a preset first threshold.

10. The imaging apparatus according to claim 1, wherein, The intensity attenuation determiner: The conversion is performed by transforming the change in intensity value per unit time into an absolute proportion of the maximum intensity value to determine the occurrence of intensity decay.

11. The imaging apparatus according to claim 10, wherein, The conversion includes: Calculate the absolute value of the rate of change of the intensity value per unit time; and The first value is determined by dividing the absolute value by the maximum intensity value.

12. The imaging apparatus according to claim 11, wherein, The intensity attenuation determiner: The occurrence of intensity decay is determined in response to the first value being higher than a preset second threshold.

13. The imaging apparatus according to claim 12, wherein the imaging apparatus further comprises: A pixel processor that disables macropixels identified by the intensity attenuation determiner as having experienced the intensity attenuation.

14. An imaging apparatus, the imaging apparatus comprising: A light source module that emits modulated light toward the target object; A time-of-flight (TOF) sensor, comprising pixels that generate pixel data in response to detecting reflected light from the target object; A temperature sensor that generates a temperature value corresponding to the temperature of the Time-of-Flight (TOF) sensor; as well as An image signal processor that generates an intensity image based on a count value representing the number of detection events in which the pixel detects photons of the reflected light; Analyze the changes in the intensity values ​​of the intensity image during the first time period; Analyze the changes in temperature values ​​during the first time period; And determine the occurrence of intensity decay of the pixel.

15. The imaging apparatus according to claim 14, wherein, The time-of-flight (TOF) sensor includes: A time-to-digital converter (TDC) converts information about the time it takes for the reflected light to reach the time-of-flight (TOF) sensor into a digital signal; and A histogram generation circuit that generates a histogram based on the digital signal.

16. The imaging apparatus according to claim 14, wherein, The pixels in the TOF sensor include single-photon avalanche diode (SPAD) elements.

17. The imaging apparatus according to claim 14, wherein, The image signal processor: The correlation coefficient between the temperature value and the average intensity value is used to determine the occurrence of intensity decay.

18. The imaging apparatus according to claim 17, wherein, The image signal processor: The occurrence of intensity decay is determined in response to the correlation coefficient being less than a preset first threshold.

19. The imaging apparatus according to claim 14, wherein, The image signal processor: The first value is determined by converting the change in the intensity value per unit time into an absolute proportion of the maximum intensity value, in order to determine the occurrence of intensity decay.

20. The imaging apparatus according to claim 19, wherein, The image signal processor: The occurrence of intensity decay is determined in response to the first value being higher than a preset second threshold.

Images