Depth sensor and image signal processor changing binning mode

The depth sensor and image signal processor adapt binning mode based on external light intensity to improve measurement accuracy by minimizing noise and maintaining resolution.

KR103045018B1Active Publication Date: 2026-11-10SAMSUNG ELECTRONICS CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020210097963
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-04-21
Filing Date
2021-07-26
Publication Date
2026-11-10
Estimated Expiration
2041-07-26

AI Technical Summary

Technical Problem

Existing depth sensors face a trade-off between resolution and noise due to binning methods, with analog and digital binning having different noise sources, and there is a need to adapt binning mode based on external light conditions.

Method used

A depth sensor and image signal processor that dynamically adjust the binning mode based on the intensity of external light, using a controller to determine whether to perform analog or digital binning based on ambient light conditions, thereby minimizing noise and improving distance measurement accuracy.

Benefits of technology

The solution enhances the accuracy of distance measurements by adapting the binning mode to external light conditions, reducing noise and maintaining resolution effectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112021086237799-PAT00008_ABST
    Figure 112021086237799-PAT00008_ABST
Patent Text Reader

Abstract

A depth sensor and an image signal processor for changing a binning mode are disclosed. A method of operation for a depth sensor for measuring distance to an object according to one aspect of the technical concept of the present disclosure is characterized by comprising the steps of: outputting a pixel signal from at least one depth pixel included in a pixel array; generating external light information based on the intensity of external light outside the depth sensor measured using the pixel signal; and setting the binning mode of the depth sensor to an analog binning mode or a digital binning mode based on the external light parameter value.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The technical concept of the present disclosure relates to a depth sensor, and more specifically, to a depth sensor and an image signal processor that change a binning mode. Background Technology

[0002] 3D camera or LADAR (Laser Detection And Ranging) technology may be used to acquire distance information of objects. For example, a time of flight (TOF) method may be used to generate a depth image by measuring the distance between an imaging device and a subject by measuring the round-trip time of light. The TOF method includes processing to project light of a specific wavelength onto a subject and to extract a depth image by measuring or capturing the light reflected from the subject with a photodiode or camera.

[0003] At this time, binning technology that combines multiple pixel information into one can be used to increase the measurement distance, but there is a problem that the resolution is reduced by binning. In addition, there are analog binning and digital binning methods, and the main noise source may differ depending on each method. The problem to be solved

[0004] The problem that the technical concept of the present disclosure aims to solve is to provide a depth sensor and an image signal processor that change the binning mode based on the intensity of external light. means of solving the problem

[0005] To achieve the above objectives, a method of operation for a depth sensor for measuring distance to an object according to one aspect of the technical concept of the present disclosure comprises: a step of outputting a pixel signal from at least one depth pixel included in a pixel array; a step of generating external light information based on the intensity of external light outside the depth sensor measured using the pixel signal; and a step of setting the binning mode of the depth sensor to an analog binning mode or a digital binning mode based on the external light parameter value.

[0006] To achieve the above objectives, a depth sensor according to one aspect of the technical concept of the present disclosure comprises a transmitter that transmits an incident signal to an object and a receiver that receives a reflected signal reflected from the object to measure the distance to the object, wherein the receiver comprises a pixel array including a plurality of depth pixels that receive the reflected signal, a readout circuit that receives a pixel signal generated from the pixel array, and a controller configured to calculate the intensity of external light based on the pixel signal and determine an analog binning mode or a digital binning mode of the readout circuit based on the intensity of the external light, wherein the readout circuit operates in the analog binning mode or the digital binning mode to generate a digital pixel signal based on the pixel signal.

[0007] To achieve the above objectives, a depth sensor for measuring distance to an object according to one aspect of the technical concept of the present disclosure is characterized by including a controller configured to calculate the intensity of external light based on a pixel signal generated for a reflected signal reflected from the object, determine whether the operating environment of the depth sensor is indoors or outdoors based on the intensity of the external light, and perform analog binning or digital binning on the pixel signal based on the determined operating environment.

[0008] To achieve the above objectives, a method of operation of an image signal processor (ISP) according to one aspect of the technical concept of the present disclosure is characterized by comprising the steps of: receiving a pixel signal generated for a reflected signal reflected from an object from a depth sensor that measures the distance to an object; calculating the intensity of external light based on the pixel signal; and determining whether to perform an analog binning operation or a digital binning operation of the depth sensor based on the intensity of the external light.

[0009] To achieve the above objectives, an image signal processor (ISP) according to one aspect of the technical concept of the present disclosure is characterized by comprising: a parameter calculation circuit configured to receive a pixel signal generated for a reflected signal reflected from an object from a depth sensor that measures the distance to an object, and to calculate an ambient light parameter value reflecting the intensity of ambient light based on the pixel signal; and a binning mode determination circuit configured to determine that the depth sensor performs an analog binning operation if the ambient light parameter value exceeds a threshold value, and to determine that the depth sensor performs a digital binning operation if the ambient light parameter value is less than or equal to the threshold value. Effects of the invention

[0010] According to the depth sensor and image signal processor for changing the binning mode of the technical concept of the present disclosure, by varying the binning mode according to the intensity of external light and minimizing noise in the pixel signal, the accuracy of the measured distance or generated depth data can be improved. Brief explanation of the drawing

[0011] FIG. 1 is a block diagram showing an image processing device according to an exemplary embodiment of the present disclosure. FIG. 2 is a timing diagram showing incident and reflected signals according to an exemplary embodiment of the present disclosure. FIG. 3 is an exemplary diagram showing the structure of a depth pixel according to an exemplary embodiment of the present disclosure. FIG. 4 is an exemplary diagram showing the structure of a pixel array according to an exemplary embodiment of the present disclosure. FIG. 5 is a circuit diagram schematically showing a portion of a depth pixel according to an exemplary embodiment of the present disclosure. FIG. 6 is a timing diagram showing a photogate signal according to an exemplary embodiment of the present disclosure. FIG. 7 is an exemplary diagram showing the structure of a pixel array according to an exemplary embodiment of the present disclosure. FIG. 8 is a block diagram showing a controller according to an exemplary embodiment of the present disclosure. FIG. 9 is a graph showing the relationship between external light parameters and the intensity of external light according to an exemplary embodiment of the present disclosure. FIG. 10 is a graph showing external light parameters according to the operating environment of a depth sensor according to an exemplary embodiment of the present disclosure. FIG. 11 is a graph showing noise according to the binning mode of a depth sensor according to an exemplary embodiment of the present disclosure. FIG. 12 is a block diagram showing a readout circuit according to an exemplary embodiment of the present disclosure. FIG. 13 is a flowchart illustrating the operation of a depth sensor according to an exemplary embodiment of the present disclosure. FIG. 14 is a flowchart illustrating a method for setting a binning mode of a depth sensor according to an exemplary embodiment of the present disclosure. FIG. 15 is a graph showing a method for setting a binning mode of a depth sensor according to an exemplary embodiment of the present disclosure. FIG. 16 is a block diagram showing an image processing device according to an exemplary embodiment of the present disclosure. FIG. 17 is a block diagram showing an electronic device including a depth sensor according to an exemplary embodiment of the present disclosure. Specific details for implementing the invention

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0013] FIG. 1 is a block diagram showing an image processing device according to an exemplary embodiment of the present disclosure, and FIG. 2 is a timing diagram showing an incident signal and a reflected signal according to an exemplary embodiment of the present disclosure.

[0014] Referring to FIGS. 1 and 2, the image processing device (10) may include a depth sensor (100) and an image signal processor (ISP) (200). The image processing device (10) can generate depth data by measuring the distance from an object (1) located outside.

[0015] The depth sensor (100) can operate in a time of flight (TOF) manner and can measure the distance to an object (1) based on the phase difference between an incident signal (TX) and a reflected signal (RX) as shown in FIG. 2. To measure the distance to an object (1), the depth sensor (100) may include a transmitter (110) that transmits an incident signal (TX) to the object (1) and a receiver (120) that receives a reflected signal (RX) reflected from the object (1) and generates a digital pixel signal (DSIG_PX) based on the reflected signal (RX). The transmitter (110) may include a driving circuit (111), a light source (112), and an optical system (113), and the receiver (120) may include a controller (121), a pixel array (122), and a readout circuit (123).

[0016] The driving circuit (111) can generate a clock signal to drive the light source (112) based on the control of the controller (121). The light source (112) can generate a modulated optical signal in response to the clock signal and radiate it to an object (1) through the optical system (1130). The clock signal or the modulated optical signal may be a sinusoidal wave or a square wave.

[0017] The light source (112) may be composed of an LED (light emitting diode), an OLED (organic light emitting diode), an AMOLED (active-matrix organic light emitting diode), or a laser diode. In one embodiment, the light source (112) may be composed of a vertical-cavity surface-emitting laser (VCSEL).

[0018] The pixel array (122) may include at least one depth pixel (PX) that receives a reflected signal (RX) reflected from an object (1). The depth pixel (PX) can generate a pixel signal (SIG_PX), which is an electrical signal, by accumulating photoelectric charge for a certain period of time, for example, an integration time.

[0019] The depth pixel (PX) can output at least one pixel signal (SIG_PX) by control signals having different phases in one sampling interval. For example, the depth pixel (PX) may have a 4-tap structure and can output four pixel signals (SIG_PX) by photogate control signals having four different phases. Further details regarding the depth pixel (PX) will be described later.

[0020] The pixel array (122) can receive a binning mode control signal (BM_CTRL) from the controller (121). Depending on the binning mode control signal (BM_CTRL), the pixel array (122) can control the timing of transmitting charges generated from multiple taps to a floating diffusion node. For example, the pixel array (122) can acquire a signal from a tap corresponding to a specific phase in a single frame interval while operating in full mode. For example, the pixel array (122) can acquire a signal from multiple taps corresponding to a specific phase in a single frame interval while operating in binning mode.

[0021] The readout circuit (123) can process a pixel signal (SIG_PX) output from a depth pixel (PX) to generate a digital pixel signal (DSIG_PX). In one embodiment, the readout circuit (123) can perform a correlated double sampling (CDS) operation and an analog to digital converting (ADC) operation on the pixel signal (SIG_PX) under the control of a controller (121).

[0022] According to an embodiment of the present disclosure, the depth sensor (100) can perform a binning operation. Binning may refer to an operation of summing and / or averaging a plurality of pixel signals (SIG_PX). A binning operation performed on an analog signal may be referred to as analog binning, and a binning operation performed on a digital signal may be referred to as digital binning.

[0023] The pixel array (122) and / or the readout circuit (123) can perform an analog binning operation or a digital binning operation by means of a binning mode control signal (BM_CTRL) received from the controller (121).

[0024] In one embodiment, the pixel array (122) can read charges generated from a plurality of taps from a single floating diffusion upon receiving a binning mode control signal (BM_CTRL) indicating an analog binning mode. In this case, the plurality of taps may correspond to the same phase. Subsequently, the readout circuit (123) can generate a digital pixel signal (DSIG_PX) by summing the signals obtained from the pixel array (122).

[0025] In one embodiment, the pixel array (122) can read the charge generated at one tap from one floating diffusion upon receiving a binning mode control signal (BM_CTRL) indicating a digital binning mode. The readout circuit (123) can generate a digital pixel signal (DSIG_PX) by digitizing the signal obtained from the pixel array (122) and averaging a plurality of digital signals.

[0026] The number of signals to be binning and / or the number of depth pixels (PX) to be binning can be determined by the controller (121). The readout circuit (123) can perform a binning operation on the pixel signal (SIG_PX) to generate a digital pixel signal (DSIG_PX) and provide the digital pixel signal (DSIG_PX) to the image signal processor (200).

[0027] The controller (121) can control a series of operations of the depth sensor (100). The controller (121) can control the driving circuit (111) to generate an incident signal (TX). In one embodiment, the controller (121) can be configured to calculate the intensity of external light based on a pixel signal (SIG_PX) generated for a reflected signal (RX) reflected from an object (1), determine whether the operating environment of the depth sensor (100) is indoors or outdoors based on the intensity of external light, and perform analog binning or digital binning on the pixel signal (SIG_PX) based on the determined operating environment.

[0028] According to an embodiment of the present disclosure, the controller (121) receives a pixel signal (SIG_PX) from the readout circuit (123) and can generate a binning mode control signal (BM_CTRL) to determine the binning mode of the readout circuit (123) based on the pixel signal (SIG_PX). The controller (121) can measure the intensity of ambient light using the pixel signal (SIG_PX). The controller (121) can generate ambient light information to determine the binning mode based on the intensity of the ambient light. Based on the ambient light information, the binning mode of the readout circuit (123) can be determined as an analog binning mode or a digital binning mode. Meanwhile, ambient light may also be referred to as ambient light. Based on the ambient light information, it can be determined whether the operating environment of the depth sensor (100) is indoors or outdoors. The ambient light information may include, for example, the ambient light parameters of FIG. 8.

[0029] Since ambient light can act as a noise source for the pixel signal (SIG_PX), according to an embodiment of the present disclosure, the binning mode can be changed according to the intensity of the ambient light to reduce noise. This minimizes the noise of the pixel signal (SIG_PX), thereby improving the accuracy of the measured distance or generated depth data.

[0030] Although not illustrated, the receiver (120) may further include a modulator for modulating a reflected signal (RX), a row driver for supplying control signals to depth pixels (PX), a row decoder and a timing controller (121), and a photo gate controller (121) for providing photo gate control signals to depth pixels (PX).

[0031] The image signal processor (200) can generate depth data based on a digital pixel signal (DSIG_PX). Although the image signal processor (200) is shown in FIG. 1 as being inside the image processing device (10), it is not limited thereto and the image signal processor (200) may be implemented outside the image processing device (10).

[0032] FIG. 3 is an exemplary diagram showing the structure of a depth pixel according to an exemplary embodiment of the present disclosure.

[0033] Referring to FIG. 3, the depth pixel (PX) may have a 4-tap structure. A 'tap' may refer to a component including a photogate and a detection area for collecting and detecting photocharges according to a demodulation signal.

[0034] The depth pixel (PX) may include four taps corresponding to each of different phases. For example, the four taps may correspond to phases of 0°, 90°, 180°, and 270°, respectively. That is, the phases of the photogate signals provided to each of the four taps may be 0°, 90°, 180°, and 270°, respectively.

[0035] A depth pixel (PX) may include a floating diffusion node (FD) corresponding to each tab. That is, since the depth pixel (PX) has a 4-tab structure, it may include 4 floating diffusion nodes (FD). Although the floating diffusion nodes (FD) are depicted as being located at the corners of each tab, they are not limited thereto.

[0036] Due to the charge accumulated in the floating diffusion node (FD), four different pixel signals (A1 to A4) can be generated at each tap. In one embodiment, a first pixel signal (A1) may be generated at the first tap corresponding to the 0° phase, a second pixel signal (A2) may be generated at the second tap corresponding to the 90° phase, a third pixel signal (A3) may be generated at the third tap corresponding to the 180° phase, and a fourth pixel signal (A4) may be generated at the fourth tap corresponding to the 270° phase.

[0037] Based on the first to fourth pixel signals (A1 to A4), the distance to the object (1) can be calculated according to mathematical formula 1 and mathematical formula 2.

[0038]

[0039]

[0040] In mathematical formula 1 represents the phase difference between the incident signal (TX) and the reflected signal (RX), and in Equation 2, c is the speed of light, represents the frequency of the incident signal (TX) or reflected signal (RX), and d represents the distance from the object (1).

[0041] Meanwhile, the depth pixel (PX) is not limited to a 4-tap structure and may be a 2-tap structure or a multi-tap structure having multiple taps.

[0042] FIG. 4 is an exemplary diagram showing the structure of a pixel array according to an exemplary embodiment of the present disclosure. The pixel array (122) may correspond to one embodiment of the pixel array (122) of FIG. 1.

[0043] Referring to FIG. 4, the pixel array (122) may include a plurality of depth pixels of a 4-tap structure, for example, a first depth pixel (PX1) and a second depth pixel (PX2).

[0044] The first depth pixel (PX1) may include first to fourth tabs, and the second depth pixel (PX2) may include fifth to eighth tabs. The first tab and the fifth tab may correspond to the same phase, for example, a 0° phase, the second tab and the sixth tab may correspond to the same phase, for example, a 90° phase, the third tab and the fifth tab may correspond to the same phase, for example, a 180° phase, and the fourth tab and the sixth tab may correspond to the same phase, for example, a 270° phase.

[0045] The first depth pixel (PX1) and the second depth pixel (PX2) can be arranged in a mirror-symmetric structure so that tabs corresponding to the same phase are adjacent to each other. For example, the third tab and the fifth tab may be adjacent, and the fourth tab and the sixth tab may be adjacent. The first and second tabs of the first depth pixel (PX1) may be adjacent to other depth pixels.

[0046] Accordingly, multiple adjacent tabs can share a floating diffusion node. For example, the 3rd tab and the 5th tab can share a single floating diffusion node, and the 4th tab and the 6th tab can share a single floating diffusion node.

[0047] FIG. 5 is a circuit diagram schematically showing a portion of a depth pixel according to an exemplary embodiment of the present disclosure, and FIG. 6 is a timing diagram showing a photogate signal according to an exemplary embodiment of the present disclosure.

[0048] Referring to FIG. 5, the depth pixel (PX) may include one photodiode (PD), four phototransistors (PX1 to PX4), and four floating diffusion nodes (FD1 to FD4). Although not illustrated, the depth pixel (PX) may further include at least one of four transfer transistors, four capture transistors, four storage transistors, and one overflow transistor.

[0049] The first to fourth phototransistors (PX1 to PX4) can each be connected between the first to fourth floating diffusion nodes (FD1 to FD4) and the photodiode (PD). The first to fourth phototransistors (PX1 to PX4) can each be controlled by the first to fourth photogate signals (PG1 to PG4). The first to fourth photogate signals (PG1 to PG4) may have different phases. Referring to FIG. 6 as one embodiment, if the phase of the first photogate signal (PG1) is assumed to be 0°, the phases of the second to fourth photogate signals (PG2 to PG4) may be 90°, 180°, and 270°, respectively. That is, the phase difference of the first to fourth photogate signals (PG1 to PG4) may each be 90°.

[0050] When the first to fourth phototransistors (PX1 to PX4) are turned on, photocharges can be generated in the photodiode (PD), and the generated photocharges can be accumulated in the first to fourth floating diffusion nodes (FD1 to FD4). Based on the charges accumulated in the first to fourth floating diffusion nodes (FD1 to FD4), the first to fourth pixel signals (e.g., A1 to A4 in FIG. 3) can be output. The first to fourth pixel signals (A1 to A4) can also be different from each other due to the phase difference of the first to fourth photogate signals (PG1 to PG4).

[0051] FIG. 7 is an exemplary diagram showing the structure of a pixel array according to an exemplary embodiment of the present disclosure.

[0052] Referring to FIG. 7, the depth sensor (e.g., 100 in FIG. 1) can operate in a binning mode, and accordingly, the pixel array (122a) can operate in units of depth pixel groups (GPX). Below, the case of a 2x2 binning mode will be described as an example.

[0053] A depth pixel group (GPX) may include multiple tabs corresponding to the same phase. In one embodiment, it may include a first tab corresponding to a first phase (e.g., 0°), a second tab corresponding to a second phase (e.g., 90°), a third tab corresponding to a third phase (e.g., 180°), and a fourth tab corresponding to a fourth phase (e.g., 270°), each with multiple, for example, four tabs.

[0054] Taps corresponding to the same phase can share a floating diffusion node. Accordingly, pixel group signals generated from multiple taps (e.g., four first taps) can be output through a single floating diffusion node.

[0055] FIG. 8 is a block diagram showing a controller according to an exemplary embodiment of the present disclosure, FIG. 9 is a graph showing the relationship between an ambient light parameter and the intensity of ambient light according to an exemplary embodiment of the present disclosure, FIG. 10 is a graph showing the ambient light parameter according to the operating environment of the depth sensor according to an exemplary embodiment of the present disclosure, and FIG. 11 is a graph showing noise according to the binning mode of the depth sensor according to an exemplary embodiment of the present disclosure.

[0056] Referring to FIG. 8, the controller (300) may include a parameter calculation circuit (310) and a binning mode determination circuit (320).

[0057] The parameter calculation circuit (310) can receive a pixel signal (SIG_PX) from the readout circuit (400). The pixel signal (SIG_PX) may be, for example, the first to fourth pixel signals (A1 to A4) of FIG. 3. The parameter calculation circuit (310) can calculate the intensity of the external light using the pixel signal (SIG_PX) and can calculate the external light parameter based on the intensity of the external light. The external light parameter may refer to a variable that corrects the effect of the distance from the object (1) to the intensity of the external light. Referring to FIG. 9, the external light parameter may be inversely proportional to the intensity of the external light. The external light parameter value (PAR) may be calculated based on the intensity of the incident signal (TX) radiated to the object (1) and the intensity of the external light. For example, the external light parameter value (PAR) may include a value obtained by dividing the intensity of the incident signal (TX) by the intensity of the external light, but is not limited thereto. For example, it may include a value obtained by dividing the intensity of the incident signal (TX) by the square root of the intensity of the ambient light.

[0058] The intensity of the incident signal (TX), the intensity of the ambient light, and the ambient light parameters can be calculated through Equations 3, 4, and 5, respectively.

[0059]

[0060]

[0061]

[0062] In Equation 3, A1 to A4 each represent the first to fourth pixel signals output through the first to fourth taps of the depth pixel, and A represents the intensity of the incident signal (TX). In Equation 4, B represents the intensity of the external light, and in Equation 5, P represents the external light parameter. By using the external light parameter, the parameter calculation circuit (310) can measure the intensity of the external light regardless of the distance from the object (1).

[0063] The binning mode determination circuit (320) can determine the binning mode of the readout circuit (400) based on the external light parameter value (PAR). For example, if the external light parameter value (PAR) exceeds a threshold value, the binning mode can be determined as the analog binning mode, and if the external light parameter value (PAR) is below the threshold value, the binning mode can be determined as the digital binning mode. Meanwhile, the binning mode determination circuit (320) can use two threshold values. This will be described later with reference to FIG. 15.

[0064] The binning mode determination circuit (320) can determine the operating environment of the depth sensor (e.g., 100 in FIG. 1) based on the external light parameter value (PAR), and can generate a binning mode control signal (BM_CTRL) to determine the binning mode of the reading circuit (400) according to the operating environment. For example, when the external light parameter value (PAR) exceeds a threshold value, the operating environment of the depth sensor (100) can be determined to be indoors, and when the external light parameter value (PAR) is below the threshold value, the operating environment of the depth sensor (100) can be determined to be outdoors.

[0065] Meanwhile, referring to FIG. 10, the ambient light parameter value (PAR) is clearly distinguished between indoor and outdoor environments even though the distance from the object (1) changes. Therefore, the ambient light parameter can be used to determine the operating environment of the depth sensor (100).

[0066] According to an embodiment of the present disclosure, the controller (300) determines the binning mode of the readout circuit (400) based on the external light parameter, thereby increasing the level of the pixel signal (SIG_PX) through analog binning in an indoor environment where the intensity of external light is weak, and preventing the depth pixel from being saturated by external light through digital binning in an outdoor environment where the intensity of external light is strong.

[0067] Referring to Fig. 11, depending on the intensity of ambient light, the magnitude of the noise in the data generated through analog binning and the noise in the data generated through digital binning may differ. For example, when the intensity of ambient light is between 0 lux and 26 klux, the noise in the data generated through digital binning may be greater than the noise in the data generated through analog binning. Therefore, it may be advantageous to perform analog binning in this range. Also, when the intensity of ambient light exceeds 26 klux, the noise in the data generated through analog binning may be greater than the noise in the data generated through digital binning. Therefore, it may be advantageous to perform digital binning in this range.

[0068] That is, when the intensity of the external light is below a preset threshold value, noise can be reduced by performing analog binning, and when the intensity of the external light exceeds a preset threshold value, noise can be reduced by performing digital binning. In one embodiment, the binning mode determination circuit (320) can set a threshold value of the external light parameter based on the intensity of the external light at which the magnitude of the noise is reversed, and determine an analog binning mode or a digital binning mode of the depth sensor (100) based on the threshold value.

[0069] Meanwhile, in FIG. 8, the parameter calculation circuit (310) can receive additional pixel group signals generated from a depth pixel group (e.g., GPX in FIG. 7). In this case, the parameter calculation circuit (310) can calculate an ambient light parameter value (PAR) based on the pixel group signals.

[0070] FIG. 12 is a block diagram showing a readout circuit according to an exemplary embodiment of the present disclosure.

[0071] Referring to FIG. 12, the readout circuit (500) may receive a binning mode control signal (BM_CTRL) from a controller (e.g., 300 in FIG. 8) and receive a plurality of pixel group signals from a pixel array (e.g., 122 in FIG. 1). The readout circuit (500) may generate and output a digital pixel signal (DSIG_PX) by performing analog binning or digital binning according to the binning mode control signal (BM_CTRL).

[0072] The readout circuit (500) may include an analog binning circuit (510), an analog to digital converter (ADC) (520), and a digital binning circuit (530). In one embodiment, when the binning mode control signal (BM_CTRL) indicates analog binning as it is at a first logic level, the analog binning circuit (510) and the ADC (520) of the readout circuit (500) may be operated. In one embodiment, when the binning mode control signal (BM_CTRL) indicates digital binning as it is at a second logic level, the ADC (520) and the digital binning circuit (530) of the readout circuit (500) may be operated.

[0073] For analog binning operation, the analog binning circuit (510) receives multiple pixel group signals that are analog signals and can generate a first signal by summing each pixel group signal. The ADC (520) can generate a third signal by converting the first signal or the second signal, which are analog signals, into a digital signal. The readout circuit (500) can output the third signal as a digital pixel signal (DSIG_PX).

[0074] For digital binning operation, the ADC (520) receives multiple pixel group signals that are analog signals and can convert each pixel group signal into a digital signal to generate multiple fourth signals. The digital binning circuit (530) receives multiple fourth signals that are digital signals and can generate a fifth signal by summing the multiple fourth signals. In one embodiment, the digital binning circuit (530) can generate a sixth signal by calculating the average value of the multiple fourth signals. The readout circuit (500) can output the fifth signal or the sixth signal as a digital pixel signal (DSIG_PX).

[0075] FIG. 13 is a flowchart illustrating the operation of a depth sensor according to an exemplary embodiment of the present disclosure.

[0076] Referring to FIG. 1 and FIG. 13 together, a pixel signal (SIG_PX) may be output (S110). In one embodiment, a plurality of pixel signals may be output from a depth pixel (PX). In one embodiment, a plurality of pixel group signals may be output from a depth pixel group (e.g., GPX of FIG. 7).

[0077] The intensity of external light can be measured based on a pixel signal (SIG_PX) (S120). In one embodiment, the intensity of external light can be calculated by averaging a plurality of pixel signals. In one embodiment, the intensity of external light can be calculated by averaging a plurality of pixel group signals.

[0078] An external light parameter value (PAR) can be calculated based on the intensity of external light (S130). In one embodiment, the external light parameter value (PAR) can be calculated based on the intensity of external light and the intensity of an incident signal (TX) radiated to the object (1). For example, the external light parameter value (PAR) can be calculated based on the ratio of the intensity of the incident signal (TX) to the intensity of the external light. The external light parameter value (PAR) can be inversely proportional to the intensity of external light.

[0079] A binning mode can be set according to an external light parameter value (PAR) (S140). In one embodiment, the operating environment of the depth sensor (100) can be determined through the external light parameter value (PAR), and the binning mode of the depth sensor (100) can be set according to the operating environment.

[0080] Subsequently, according to the set binning mode, a pixel group signal is output from a depth pixel group containing multiple depth pixels, and a digital pixel signal can be generated by processing the pixel group signal. As an example, if set to an analog binning mode, the first pixel group signal and the second pixel group signal are summed, and the summed signal can be converted into a digital pixel signal. As an example, if set to a digital binning mode, the first and second pixel group signals are each converted into digital signals, and each digital signal is summed to generate a digital pixel signal.

[0081] FIG. 14 is a flowchart illustrating a method for setting a binning mode of a depth sensor according to an exemplary embodiment of the present disclosure.

[0082] Referring to FIG. 1 and FIG. 14 together, it can be determined whether the ambient light parameter value (PAR) exceeds a preset threshold value (S210). If the ambient light parameter value (PAR) exceeds (or is greater than) the threshold value, the depth sensor (100) can be set to an analog binning mode (S220). If the ambient light parameter value (PAR) is less than (or lower than) the threshold value, the depth sensor (100) can be set to a digital binning mode (S230).

[0083] Meanwhile, multiple threshold values ​​may be used to change the binning mode of the depth sensor (100). This is explained through FIG. 15.

[0084] FIG. 15 is a graph showing a method for setting a binning mode of a depth sensor according to an exemplary embodiment of the present disclosure.

[0085] Referring to FIG. 1 and FIG. 15 together, a first threshold value (TH1) and a second threshold value (TH2) smaller than the first threshold value (TH1) may be used to change the binning mode of the depth sensor (100).

[0086] The case where the ambient light parameter value (PAR) exceeds the first threshold value (TH1) is described. Since the ambient light parameter may be inversely proportional to the intensity of the ambient light, a large ambient light parameter may mean that the intensity of the ambient light is weak. Therefore, the operating environment of the depth sensor (100) can be determined to be indoors. Accordingly, the depth sensor (100) can reduce noise by performing analog binning. That is, it can reduce quantization noise that may occur in the ADC (e.g., 520 in FIG. 12) of the readout circuit (123).

[0087] The case where the ambient light parameter value (PAR) is less than or equal to the second threshold value (TH2) is described. Since the ambient light parameter may be inversely proportional to the intensity of the ambient light, a small ambient light parameter may indicate that the intensity of the ambient light is strong. Therefore, the operating environment of the depth sensor (100) can be determined to be outdoors. Accordingly, the depth sensor (100) can reduce noise by performing digital binning. That is, by securing the light receiving area (full well capacity, FWC) of the floating diffusion node of the depth pixel (PX), saturation of the depth pixel (PX) can be prevented.

[0088] This describes the case where the ambient light parameter value (PAR) exceeds the second threshold value (TH2) and is less than or equal to the first threshold value (TH1). In this case, the reference threshold value may vary depending on whether the ambient light parameter value (PAR) is trending upward or downward.

[0089] For example, if the ambient light parameter value (PAR) is trending downward between the first time point (t1) and the second time point (t2), the second threshold value (TH2) (or the first threshold value (TH1)) can be a reference value. Accordingly, at the second time point (t2), the binning mode of the depth sensor (100) can be changed from an analog binning mode to a digital binning mode.

[0090] For example, if the ambient light parameter value (PAR) shows an increasing trend between the third time point (t3) and the fourth time point (t4), or between the fifth time point (t5) and the sixth time point (t6), the first threshold value (TH1) (or the second threshold value (TH2)) can serve as a reference value. Accordingly, the binning mode of the depth sensor (100) can be maintained in digital binning mode between the third time point (t3) and the fourth time point (t4). Accordingly, the binning mode of the depth sensor (100) may not change between the fourth time point (t4) and the fifth time point (t5). Between the fifth time point (t5) and the sixth time point (t6), since the first threshold value (TH1) is the reference value, the binning mode of the depth sensor (100) can be changed from digital binning mode to analog binning mode at the sixth time point (t6). Meanwhile, the number of threshold values ​​or the threshold value serving as the reference point for changing the binning mode is not limited thereto.

[0091] According to an embodiment of the present disclosure, the depth sensor (100) uses two threshold values ​​(TH1, TH2) so that the binning mode can be prevented from changing sensitively. That is, even when the ambient light parameter value (PAR) changes frequently in an instant, the binning mode change operation can be performed stably.

[0092] FIG. 16 is a block diagram showing an image processing device according to an exemplary embodiment of the present disclosure. Since the image processing device (10a) is similar to the image processing device (10) of FIG. 1, a redundant description is omitted.

[0093] Referring to FIG. 16, an image signal processor (200a) can perform a series of operations to determine the binning mode of a depth sensor (100a). In one embodiment, the image signal processor (200a) can receive a pixel signal (SIG_PX) generated for a reflected signal (RX) reflected from an object (1) from a depth sensor (100a) that measures the distance to an object (1). The image signal processor (200a) can calculate the intensity of ambient light based on the pixel signal (SIG_PX). Based on the intensity of ambient light, the image signal processor (200a) can determine whether to perform an analog binning operation or a digital binning operation of the depth sensor (100a).

[0094] Specifically, the image signal processor (200a) can determine whether the operating environment of the depth sensor (100a) is indoors or outdoors based on the intensity of the external light. If the operating environment of the depth sensor (100a) is determined to be indoors, it can determine to perform an analog binning operation, and if it is determined to be outdoors, it can determine to perform a digital binning operation.

[0095] The image signal processor (200a) may include a parameter calculation circuit (210) and a binning mode determination circuit (220).

[0096] The parameter calculation circuit (210) may be similar to the parameter calculation circuit (310) of FIG. 8, and may receive a pixel signal (SIG_PX) from the depth sensor (100a) and calculate an ambient light parameter value (PAR) based on the pixel signal (SIG_PX).

[0097] The binning mode determination circuit (220) may be similar to the binning mode determination circuit (320) of FIG. 8 and may generate a binning mode control signal (BM_CTRL) to determine the binning mode of the depth sensor (100a) based on an external light parameter value (PAR) and a preset threshold value, and provide it to the readout circuit (123a) of the depth sensor (100a).

[0098] The readout circuit (123a) can generate a digital pixel signal (DSIG_PX) by performing a binning operation based on the binning mode control signal (BM_CTRL). The image signal processor (200a) can receive the digital pixel signal (DSIG_PX) and generate depth data.

[0099] FIG. 17 is a block diagram showing an electronic device including a depth sensor according to an exemplary embodiment of the present disclosure.

[0100] Referring to FIG. 17, the electronic device (1000) may include a depth sensor (1100), an image sensor (1200), a main processor (1300), a working memory (1400), storage (1500), a display device (1600), a user interface (1700), and a communication unit (1800).

[0101] The depth sensor (1100) may correspond to the depth sensor (100, 100a) described with reference to FIGS. 1 to 16. For example, the depth sensor (1100) may generate a pixel signal by receiving a reflected signal to measure the distance to an object outside the electronic device (1000). The depth sensor (1100) may calculate the intensity of external light based on the pixel signal and calculate an external light parameter value. The depth sensor (1100) may determine the binning mode as an analog binning mode or a digital binning mode depending on the external light parameter value. The depth sensor (1100) may generate a digital pixel signal according to the set binning mode and provide it to the main processor (1300). The main processor (1300) may generate depth data based on the digital pixel signal.

[0102] The image sensor (1200) can generate image data, such as raw image data, based on the received optical signal and provide the image data to the main processor (1300).

[0103] The main processor (1300) can control the overall operation of the electronic device (1000) and can detect the movement of an object by processing event data, i.e., event signals, received from the depth sensor (1100). It can also receive image frames from the image sensor (1200) and perform image processing based on preset information.

[0104] The working memory (1400) can store data used for the operation of the electronic device (1000). For example, the working memory (1400) can temporarily store packets or frames processed by the processor (1120). For example, the working memory (1400) may include volatile memory such as DRAM (Dynamic RAM), SDRAM (Synchronous RAM), and / or non-volatile memory such as PRAM (Phase-change RAM), MRAM (Magneto-resistive RAM), ReRAM (Resistive RAM), FRAM (Ferro-electric RAM).

[0105] Storage (1500) can store data requested for storage from the main processor (1300) or other components. Storage (1500) may include non-volatile memory such as flash memory, PRAM, MRAM, ReRAM, FRAM, etc.

[0106] The display device (1600) may include a display panel, a display driving circuit, and a DSI (display serial interface). For example, the display panel may be implemented as various devices such as an LCD (Liquid Crystal Display) device, an LED (Light Emitting Diode) display device, an OLED (Organic LED) display device, an AMOLED (Active Matrix OLED) display device, etc. The display driving circuit may include a timing controller, a source driver, etc., necessary to drive the display panel. A DSI host embedded in the main processor (1300) can perform serial communication with the display panel through the DSI.

[0107] The user interface (1700) may include at least one of input interfaces such as a keyboard, mouse, keypad, button, touch panel, touch screen, touchpad, touch ball, gyroscope sensor, vibration sensor, accelerometer sensor, etc.

[0108] The communication unit (1800) can exchange signals with an external device / system through an antenna (1830). The transceiver (1810) and MODEM (Modulator / Demodulator, 1820) of the communication unit (1800) can process signals exchanged with an external device / system according to wireless communication protocols such as LTE (Long Term Evolution), WIMAX (Worldwide Interoperability for Microwave Access), GSM (Global System for Mobile communication), CDMA (Code Division Multiple Access), Bluetooth, NFC (Near Field Communication), Wi-Fi (Wireless Fidelity), RFID (Radio Frequency Identification), etc.

[0109] Components of an electronic device (1000), such as a depth sensor (1100), an image sensor (1200), a main processor (1300), a working memory (1400), a storage (1500), a display device (1600), a user interface (1700), and a communication unit (1800), can exchange data based on one or more of various interface protocols such as USB (Universal Serial Bus), SCSI (Small Computer System Interface), MIPI, I2C, PCIe (Peripheral Component Interconnect Express), M-PCIe (Mobile PCIe), ATA (Advanced Technology Attachment), PATA (Parallel ATA), SATA (Serial ATA), SAS (Serial Attached SCSI), IDE (Integrated Drive Electronics), EIDE (Enhanced IDE), NVMe (Nonvolatile Memory Express), UFS (Universal Flash Storage), etc.

[0110] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

Claims

Claim 1 A method of operating a depth sensor for measuring distance to an object, comprising: outputting a pixel signal from at least one depth pixel included in a pixel array; calculating the intensity of external light outside the depth sensor from the pixel signal; calculating the intensity of an incident signal transmitted to the object from the pixel signal; generating an external light parameter based on the intensity of the external light and the intensity of the incident signal; and changing the binning mode of the depth sensor to an analog binning mode or a digital binning mode based on the external light parameter. Claim 2 A method of operation of a depth sensor according to claim 1, comprising: a step of outputting a pixel group signal from a depth pixel group including a plurality of depth pixels according to the modified binning mode; and a step of processing the pixel group signal according to the modified binning mode to generate a digital pixel signal. Claim 3 A method of operation of a depth sensor according to claim 2, wherein the step of generating the digital pixel signal comprises: a step of summing the first pixel signal and the second pixel signal to form a summed analog signal when the binning mode is set to the analog binning mode; and a step of converting the summed analog signal into the digital pixel signal. Claim 4 A method of operation of a depth sensor according to claim 2, wherein the step of generating the digital pixel signal comprises: a step of converting the first and second pixel group signals into digital signals respectively when the binning mode is set to a digital binning mode; a step of summing each digital signal to form a summed digital signal; and a step of generating the digital pixel signal from the summed digital signal. Claim 5 A method of operation of a depth sensor according to claim 1, wherein the step of changing to an analog binning mode or a digital binning mode comprises: a step of setting to the analog binning mode when the external light parameter value exceeds a threshold value; and a step of setting to the digital binning mode when the external light parameter value decreases to below the threshold value. Claim 6 A method of operation of a depth sensor according to claim 5, wherein the depth sensor determines the operating environment of the depth sensor as indoors when the external light parameter value exceeds the threshold value, and determines the operating environment of the depth sensor as outdoors when the external light parameter value is less than or equal to the threshold value. Claim 7 A method of operation of a depth sensor according to claim 1, wherein the pixel array includes a first depth pixel having a 4-tap structure, and the step of outputting the pixel signal includes the step of outputting the first to fourth pixel signals by the first to fourth signals each having a first to fourth phase. Claim 8 A method of operation of a depth sensor according to claim 7, characterized in that the first to fourth phases are 0°, 90°, 180°, and 270°, respectively. Claim 9 In claim 7, the step of calculating the intensity of the external light is characterized by calculating the average value of the first to fourth pixel signals. Claim 10 A method of operation of a depth sensor according to claim 1, wherein the depth sensor transmits the incident signal to the object, receives the reflected signal reflected from the object, measures the distance to the object based on the incident signal and the reflected signal, and the step of generating the external light parameter includes the step of generating the external light parameter based on the ratio of the intensity of the incident signal and the intensity of the external light.

Citation Information

Patent Citations

  • Solid state imaging apparatus and on-vehicle control apparatus

    JP2013236170A

  • Method and apparatus for operating an advanced driver assistance system of a vehicle

    EP3562145A1

  • Tetracell image sensor preforming binning

    KR1020200115801A

  • Time of flight sensor binning

    US20160003937A1