How the Depth Sensor Works

Abstract

To provide a depth sensor which changes the binning mode based on ambient light intensity, and an operating method thereof.SOLUTION: A method of operating a depth sensor for measuring distance to an object according to the present invention comprises: outputting a pixel signal from at least one depth pixel included in a pixel array; generating ambient light information based on intensity of ambient light outside the depth sensor measured using the pixel signal; and setting the binning mode of the depth sensor to an analog binning mode or digital binning mode according to the ambient light information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a depth sensor, and more particularly to a depth sensor that changes binning modes and a method of operation thereof. [Background technology]

[0002] In order to obtain distance information of an object, a three-dimensional (3D) camera or a laser detection and ranging (LADAR) technique is used. For example, a time of flight (TOF) method is used in which the distance between an imaging device and a subject is measured by measuring the time it takes for light to travel back and forth, thereby generating a depth image. The TOF method involves irradiating a subject with light of a specific wavelength, measuring or photographing the light reflected from the subject with a photodiode or camera, and processing to extract a depth image.

[0003] At this time, in order to extend the measurement distance, a binning technique is used that combines multiple pixel information into one, but there is a problem in that the resolution is reduced by performing binning. Furthermore, there are two types of binning: analog binning and digital binning, and the main noise sources differ depending on the type. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2007-515869 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in consideration of the problems with the conventional depth sensors described above, and an object of the present invention is to provide a depth sensor and an operating method thereof that change the binning mode based on the intensity of external light. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the operating method of the depth sensor for measuring the distance to an object according to the present invention is characterized by comprising the steps of: outputting a pixel signal from at least one depth pixel included in a pixel array; generating external light information based on the external light intensity outside the depth sensor measured using the pixel signal; and setting the binning mode of the depth sensor to an analog binning mode or a digital binning mode based on the external light information.

[0007] In order to achieve the above-mentioned object, the depth sensor of the present invention has a transmitter that irradiates an incident signal onto an object to measure the distance to the object, and a receiver that receives a reflected signal reflected by the object, wherein the receiver includes a pixel array including a plurality of depth pixels that receive the reflected signal, a reading circuit that receives pixel signals generated in the pixel array, and a controller configured to calculate external light intensity based on the pixel signal and determine an analog binning mode or a digital binning mode of the reading circuit based on the external light intensity, wherein the reading circuit operates in the analog binning mode or the digital binning mode and generates a digital pixel signal based on the pixel signal.

[0008] In addition, the depth sensor for measuring the distance to an object according to the present invention, which has been made to achieve the above-mentioned object, is characterized in that it includes a controller configured to calculate external light intensity based on a pixel signal generated from a reflected signal reflected by the object, determine whether the operating environment of the depth sensor is indoors or outdoors based on the external light intensity, and perform analog binning or digital binning on the pixel signal based on the determined operating environment.

[0009] In order to achieve the above object, an operating method of an image signal processor (ISP) according to the present invention is characterized in that it includes the steps of receiving a pixel signal generated from a depth sensor that measures a distance to an object, a step of calculating external light intensity based on the pixel signal, and a step of determining whether to perform an analog binning operation or a digital binning operation of the depth sensor based on the external light intensity.

[0010] In addition, an image signal processor (ISP) according to the present invention, which has been made to achieve the above-mentioned object, is characterized in that it includes a parameter calculation circuit configured to receive a pixel signal generated from a depth sensor that measures a distance to an object, the pixel signal being generated in response to a reflected signal reflected by the object, and to calculate an external light parameter value reflecting external light intensity based on the pixel signal, and a binning mode determination circuit that determines that the depth sensor should perform an analog binning operation if the external light parameter value exceeds a threshold, and determines that the depth sensor should perform a digital binning operation if the external light parameter value is equal to or less than the threshold. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of an image processing device according to an embodiment of the present invention. [Diagram 2] FIG. 4 is a timing diagram illustrating incident and reflected signals according to an embodiment of the present invention. [Diagram 3]2 is an exemplary diagram showing a schematic structure of a depth pixel according to an embodiment of the present invention; [Figure 4] 1 is an exemplary diagram showing a schematic structure of a pixel array according to an embodiment of the present invention; [Diagram 5] FIG. 2 is a circuit diagram that illustrates a schematic of a portion of a depth pixel according to an embodiment of the present invention. [Figure 6] FIG. 4 is a timing diagram illustrating photogate signals according to an embodiment of the present invention. [Figure 7] 1 is an exemplary diagram showing a schematic structure of a pixel array according to an embodiment of the present invention; [Figure 8] FIG. 2 is a block diagram showing a schematic configuration of a controller according to an embodiment of the present invention. [Figure 9] 4 is a graph illustrating a relationship between an external light parameter and an external light intensity according to an embodiment of the present invention. [Figure 10] 4 is a graph showing external light parameters according to an operating environment of a depth sensor according to an embodiment of the present invention. [Figure 11] 11 is a graph illustrating noise due to binning modes of a depth sensor according to an embodiment of the present invention. [Figure 12] 1 is a block diagram showing a schematic configuration of a reading circuit according to an embodiment of the present invention. [Figure 13] 5 is a flowchart illustrating an operation of a depth sensor according to an embodiment of the present invention. [Figure 14] 1 is a flowchart illustrating a method for setting a binning mode of a depth sensor according to an embodiment of the present invention. [Figure 15] 11 is a graph illustrating a method for setting a binning mode of a depth sensor according to an embodiment of the present invention. [Figure 16] 1 is a block diagram showing a schematic configuration of an image processing device according to an embodiment of the present invention. [Figure 17] 1 is a block diagram showing a schematic configuration of an electronic device including a depth sensor according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Next, specific examples of embodiments for carrying out the depth sensor and the operating method thereof according to the present invention will be described with reference to the drawings. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention, and FIG. 2 is a timing diagram showing incident and reflected signals according to the embodiment of the present invention. 1 and 2, the image processing device 10 includes a depth sensor 100 and an image signal processor (ISP) 200. The image processing device 10 generates depth data by measuring the distance from an externally located object 1 .

[0014] The depth sensor 100 operates by a time of flight (TOF) method, and measures the distance to the object 1 based on the phase difference between an incident signal TX and a reflected signal RX, as shown in FIG. The depth sensor 100 includes a transmitter 110 that irradiates an incident signal TX onto the object 1 to measure the distance to the object 1, and a receiver 120 that receives a reflected signal RX reflected by the object 1 and generates a digital pixel signal (DSIG_PX) based on the reflected signal RX. The transmitter 110 includes a driver circuit 111 , a light source 112 , and an optical system 113 , and the receiver 120 includes a controller 121 , a pixel array 122 , and a read circuit 123 .

[0015] The drive circuit 111 generates a clock signal for driving the light source 112 under the control of the controller 121 . The light source 112 is responsive to the clock signal to generate a modulated optical signal and irradiate it onto the object 1 via an optical system 113 . The clock signal, or the modulated optical signal, may be a sine wave or a square wave. The light source 112 is configured by an LED (light emitting diode), an OLED (organic light emitting diode), an AMOLED (active-matrix organic light emitting diode), a laser diode, or the like. In one embodiment, the light source 112 of the present invention comprises a vertical-cavity surface-emitting laser (VCSEL).

[0016] The pixel array 122 includes at least one depth pixel PX that receives a reflected signal RX reflected from the object 1 . The depth pixel PX generates an electrical signal, a pixel signal (SIG_PX), by accumulating photocharges for a certain period of time, for example an integration time. The depth pixel PX outputs at least one pixel signal (SIG_PX) in accordance with control signals having mutually different phases during one sampling period. For example, the depth pixel PX has a 4-tap structure and outputs four pixel signals (SIG_PX) according to four photogate control signals having different phases. Details relating to depth pixels PX will be described later.

[0017] The pixel array 122 receives a binning mode control signal (BM_CTRL) from the controller 121. The binning mode control signal (BM_CTRL) controls the timing at which the pixel array 122 transfers charges generated from multiple taps to a floating diffusion node. For example, the pixel array 122 operates in full mode to obtain a signal from a tap corresponding to a specific phase during one frame period. For example, the pixel array 122 operates in a binning mode to obtain signals from multiple taps corresponding to specific phases during one frame period.

[0018] The read circuit 123 processes the pixel signal (SIG_PX) output from the depth pixel PX to generate a digital pixel signal (DSIG_PX). In one embodiment, the read circuit 123 performs correlated double sampling (CDS) and analog to digital converter (ADC) operations related to the pixel signal (SIG_PX) under the control of the controller 121.

[0019] According to an embodiment of the present invention, the depth sensor 100 performs a binning operation. Binning refers to the operation of adding and / or averaging multiple pixel signals (SIG_PX). A binning operation performed on an analog signal is referred to as analog binning, whereas a binning operation performed on a digital signal is referred to as digital binning. The pixel array 122 and / or the read circuit 123 perform an analog binning operation or a digital binning operation depending on a binning mode control signal (BM_CTRL) received from the controller 121 .

[0020] In one embodiment, the pixel array 122 receives a binning mode control signal (BM_CTRL) indicating an analog binning mode, and reads the charges generated from multiple taps from one floating diffusion. In this case, multiple taps correspond to the same phase. The read circuit 123 then generates a digital pixel signal (DSIG_PX) by summing the signals obtained from the pixel array 122 .

[0021] In one embodiment, the pixel array 122 receives a binning mode control signal (BM_CTRL) indicating a digital binning mode, and reads the charge generated from one tap from one floating diffusion. The read circuit 123 digitizes the signals acquired from the pixel array 122 and generates a digital pixel signal (DSIG_PX) by averaging a plurality of digital signals.

[0022] The number of signals to be binned and / or the number of depth pixels PX to be binned are determined by the controller 121. The read circuit 123 performs a binning operation on the pixel signal (SIG_PX) to generate a digital pixel signal (DSIG_PX), and provides the digital pixel signal (DSIG_PX) to an image signal processor (ISP) 200 .

[0023] The controller 121 controls a series of operations of the depth sensor 100 . The controller 121 controls the driver circuit 111 to generate the incident signal TX. In one embodiment, the controller 121 is configured to calculate the external light intensity based on the pixel signal (SIG_PX) generated based on the reflection signal RX reflected by the object 1, determine whether the operating environment of the depth sensor 100 is indoors or outdoors based on the external light intensity, and perform analog binning or digital binning on the pixel signal (SIG_PX) based on the determined operating environment.

[0024] According to an embodiment of the present invention, the controller 121 receives a pixel signal (SIG_PX) from the read circuit 123, and generates a binning mode control signal (BM_CTRL) for determining a binning mode of the read circuit 123 based on the pixel signal (SIG_PX). The controller 121 uses the pixel signal (SIG_PX) to measure the intensity of ambient light. The controller 121 generates external light information for determining a binning mode based on the external light intensity. Based on the external light information, the binning mode of the reading circuit 123 is determined to be either the analog binning mode or the digital binning mode.

[0025] On the other hand, external light is also called ambient light. Based on the external light information, it is determined whether the operating environment of the depth sensor 100 is indoors or outdoors. The ambient light information may include, for example, the ambient light parameters of FIG. Since external light also acts as a noise source for the pixel signal (SIG_PX), according to an embodiment of the present invention, the binning mode is changed according to the external light intensity to reduce noise. This can minimize noise in the pixel signal (SIG_PX), thereby improving the accuracy of the measured distance or generated depth data.

[0026] Although not shown in the figure, the receiver 120 may further include a modulator for modulating the reflected signal RX, a row driver, a row decoder and timing controller for supplying control signals to the depth pixels PX, and a photogate controller for providing photogate control signals to the depth pixels PX. The image signal processor (ISP) 200 generates depth data based on the digital pixel signal (DSIG_PX). In FIG. 1, the image signal processor (ISP) 200 is shown as being internal to the image processing device 10, but is not limited thereto, and the image signal processor (ISP) 200 may be embodied externally to the image processing device 10.

[0027] FIG. 3 is a diagram illustrating a schematic structure of a depth pixel according to an embodiment of the present invention. Referring to FIG. 3, the depth pixel PX has a 4-tap structure. "Tap" refers to a component that includes a photogate and detection area for collecting and detecting a photocharge with a demodulation signal. A depth pixel PX includes four taps, each corresponding to a different phase. For example, the four taps correspond to phases that are 0°, 90°, 180°, and 270°, respectively. That is, the phases of the photogate signals provided to each of the four taps are 0°, 90°, 180°, and 270°, respectively.

[0028] The depth pixel PX includes a floating diffusion node FD corresponding to each tap. That is, the depth pixel PX has a 4-tap structure and therefore includes four floating diffusion nodes FD. The floating diffusion nodes FD are shown in the figures as being located at the corners of each tap, but are not limited to such. Four different pixel signals (A1 to A4) are generated from each tap based on the charges accumulated in the floating diffusion node FD. In one embodiment, a first tap corresponding to a 0° phase generates a first pixel signal A1, a second tap corresponding to a 90° phase generates a second pixel signal A2, a third tap corresponding to a 180° phase generates a third pixel signal A3, and a fourth tap corresponding to a 270° phase generates a fourth pixel signal A4.

[0029] The distance to the object 1 is calculated based on the first pixel signal A1 to the fourth pixel signal A4 by the following formulas 1 and 2.

number

number

[0030] In Equation 1, φ means the phase difference between the incident signal TX and the reflected signal RX, and in Equation 2, c means the luminous flux, and f m denotes the frequency of the incident signal TX or the reflected signal RX, and d denotes the distance to the object 1. It should be noted that the depth pixel PX is not limited to a 4-tap structure, but may be a 2-tap structure or a multi-tap structure having multiple taps.

[0031] FIG. 4 is a diagram illustrating a schematic structure of a pixel array according to an embodiment of the present invention. Pixel array 122 corresponds to one embodiment of pixel array 122 in FIG. Referring to FIG. 4, the pixel array 122 includes a plurality of depth pixels having a 4-tap structure, for example including a first depth pixel PX1 and a second depth pixel PX2.

[0032] The first depth pixel PX1 includes the first tap to the fourth tap, and the second depth pixel PX2 includes the fifth tap to the eighth tap. The first tap and the fifth tap correspond to the same phase, e.g., a 0° phase, the second tap and the sixth tap correspond to the same phase, e.g., a 90° phase, the third tap and the fifth tap correspond to the same phase, e.g., a 180° phase, and the fourth tap and the sixth tap correspond to the same phase, e.g., a 270° phase. The first depth pixel PX1 and the second depth pixel PX2 are arranged in a mirror symmetrical structure so that taps corresponding to the same phase are adjacent to each other.

[0033] For example, the third tap and the seventh tap are adjacent, and the fourth tap and the eighth tap are adjacent. The first and second taps of the first depth pixel PX1 are adjacent to different depth pixels. Thereby, adjacent taps share a floating diffusion node. For example, the third tap and the seventh tap share one floating diffusion node, and the fourth tap and the eighth tap share one floating diffusion node.

[0034] FIG. 5 is a schematic circuit diagram of a portion of a depth pixel according to an embodiment of the present invention, and FIG. 6 is a timing diagram illustrating photogate signals according to an embodiment of the present invention. Referring to FIG. 5, the depth pixel PX includes one photodiode PD, four phototransistors (PX1 to PX4), and four floating diffusion nodes (FD1 to FD4). Although not shown in the figure, the depth pixel PX may further include at least one of four transmission transistors, four capture transistors, four storage transistors, and one overflow transistor.

[0035] The first phototransistor PX1 to the fourth phototransistor PX4 are connected between the first floating diffusion node FD1 to the fourth floating diffusion node FD4 and the photodiode PD, respectively. The first to fourth phototransistors PX1 to PX4 are controlled by a first to fourth photogate signals PG1 to PG4, respectively. The first photogate signal PG1 to the fourth photogate signal PG4 have different phases from one another.

[0036] As an embodiment, referring to FIG. 6, assuming that the phase of the first photogate signal PG1 is 0°, the phases of the second photogate signal PG2 to the fourth photogate signal PG4 are 90°, 180°, and 270°, respectively. That is, the phase differences between the first photogate signal PG1 to the fourth photogate signal PG4 are each 90°. When the first phototransistor PX1 through the fourth phototransistor PX4 are turned on, photocharges are generated in the photodiode PD, and the generated photocharges are accumulated in the first floating diffusion node FD1 through the fourth floating diffusion node FD4. Based on the charges accumulated in the first floating diffusion node FD1 to the fourth floating diffusion node FD4, for example, the first pixel signal A1 to the fourth pixel signal A4 (FIG. 3) are output. Due to the phase differences between the first photogate signal PG1 to the fourth photogate signal PG4, the first pixel signal A1 to the fourth pixel signal A4 also differ from one another.

[0037] FIG. 7 is a diagram illustrating a schematic structure of a pixel array according to an embodiment of the present invention. Referring to FIG. 7, for example, the depth sensor 100 (FIG. 1) operates in a binning mode, whereby the pixel array 122a operates in depth pixel group GPX units. In the following, the "2x2" binning mode will be described as an example. The depth pixel group GPX includes multiple taps that correspond to the same phase.

[0038] In one embodiment, the tap includes a first tap corresponding to a first phase (e.g., 0°), a second tap corresponding to a second phase (e.g., 90°), a third tap corresponding to a third phase (e.g., 180°), and a fourth tap corresponding to a fourth phase (e.g., 270°), each of which is multiple, for example, four. Taps corresponding to the same phase share a floating diffusion node. Thereby, pixel group signals generated from multiple taps (for example, the first four taps) are output via one floating diffusion node.

[0039] Figure 8 is a block diagram showing the general configuration of a controller according to an embodiment of the present invention, Figure 9 is a graph showing the relationship between external light parameters and external light intensity according to an embodiment of the present invention, Figure 10 is a graph showing external light parameters depending on the operating environment of a depth sensor according to an embodiment of the present invention, and Figure 11 is a graph showing noise due to the binning mode of a depth sensor according to an embodiment of the present invention. Referring to FIG. 8, the controller 300 includes a parameter calculation circuit 310 and a binning mode determination circuit 320 .

[0040] The parameter calculation circuit 310 receives the pixel signal (SIG_PX) from the reading circuit 400 . The pixel signal (SIG_PX) is, for example, the first pixel signal A1 to the fourth pixel signal A4 in FIG. The parameter calculation circuit 310 uses the pixel signal (SIG_PX) to calculate the external light intensity, and calculates external light parameters based on the external light intensity. The external light parameter means a variable that corrects the effect of the distance from the object 1 on the intensity of external light.

[0041] Referring to FIG. 9, the external light parameter is inversely proportional to the external light intensity. The external light parameter value PAR is calculated based on the intensity of the incident signal TX irradiated to the object 1 and the external light intensity. For example, the ambient light parameter value PAR may include, but is not limited to, the intensity of the incident signal TX divided by the ambient light intensity. For example, it includes the intensity of the incident signal TX divided by the square root of the external light intensity.

[0042] The intensity of the incident signal TX, the external light intensity, and the external light parameters are calculated via Equation 3, Equation 4, and Equation 5 shown below, respectively.

number

number

number

[0043] In Equation 3, A1 to A4 respectively represent the first to fourth pixel signals outputted through the first to fourth taps of the depth pixel, and A represents the intensity of the incident signal TX. In Equation 4, B represents the external light intensity, and in Equation 5, P represents the external light parameter.

[0044] The parameter calculation circuit 310 measures the intensity of the external light, regardless of the distance to the object 1, by utilizing the external light parameters. The binning mode determination circuit 320 determines the binning mode of the reading circuit 400 based on the external light parameter value PAR. For example, when the external light parameter value PAR exceeds a threshold, the binning mode is determined to be the analog binning mode, and when the external light parameter value PAR is equal to or smaller than the threshold, the binning mode is determined to be the digital binning mode. It should be noted that the binning mode decision circuit 320 can use two thresholds. This will be described later with reference to FIG.

[0045] The binning mode determination circuit 320 determines, for example, the operating environment of the depth sensor 100 (FIG. 1) based on the external light parameter value PAR, and generates a binning mode control signal (BM_CTRL) for determining the binning mode of the reading circuit 400 based on this operating environment. For example, if the external light parameter value PAR exceeds a threshold, the operating environment of the depth sensor 100 is determined to be indoors, and if the external light parameter value PAR is equal to or less than the threshold, the operating environment of the depth sensor 100 is determined to be outdoors.

[0046] In addition, referring to FIG. 10, the external light parameter value PAR is clearly differentiated between indoor and outdoor environments, even though the distance to the object 1 changes. Therefore, the ambient light parameters are used to determine the operating environment of the depth sensor 100. According to an embodiment of the present invention, the controller 300 determines the binning mode of the reading circuit 400 based on the external light parameters, so that when the external light intensity is weak indoors, the level of the pixel signal (SIG_PX) is increased through analog binning, and when the external light intensity is strong outdoors, the depth pixel can be prevented from being saturated by the external light through digital binning.

[0047] Referring to FIG. 11, the magnitude of noise in data generated through analog binning and the magnitude of noise in data generated through digital binning differ from each other depending on the external light intensity. For example, when the external light intensity is between 0 lux and 26 klux, the noise in the data generated through digital binning is greater than the noise in the data generated through analog binning. Therefore, in this range it is advantageous to perform analog binning. In addition, when the external light intensity exceeds 26 klux, the noise of the data generated through analog binning is greater than the noise of the data generated through digital binning. In this range it is therefore advantageous to perform digital binning.

[0048] In other words, when the external light intensity is below a preset threshold, noise is reduced by performing analog binning, and when the external light intensity exceeds the preset threshold, noise is reduced by performing digital binning. In one embodiment, the binning mode determination circuit 320 sets a threshold value for an external light parameter based on the external light intensity at which the noise magnitude is inverted, and determines the analog binning mode or digital binning mode of the depth sensor 100 based on the threshold value. It should be noted that in FIG. 8, the parameter calculation circuit 310 can further receive a pixel group signal generated, for example, by the depth pixel group GPX (FIG. 7). In this case, the parameter calculation circuit 310 calculates the external light parameter value PAR based on the pixel group signal.

[0049] FIG. 12 is a block diagram showing a schematic configuration of a reading circuit according to an embodiment of the present invention. 12, a read circuit 500 receives a binning mode control signal (BM_CTRL), for example from the controller 300 (FIG. 8), and receives a number of pixel group signals, for example from the pixel array 122 (FIG. 1). The reading circuit 500 performs analog binning or digital binning in response to a binning mode control signal (BM_CTRL) to generate and output a digital pixel signal (DSIG_PX).

[0050] The read circuit 500 includes an analog binning circuit 510 , an ADC (analog to digital converter) 520 , and a digital binning circuit 530 . In one embodiment, when the binning mode control signal (BM_CTRL) is at a first logic level to indicate analog binning, the analog binning circuit 510 and the ADC 520 of the read circuit 500 are enabled. In one embodiment, when the binning mode control signal (BM_CTRL) indicates digital binning by a second logic level, the ADC 520 and the digital binning circuit 530 of the read circuit 500 are operated.

[0051] For an analog binning operation, the analog binning circuit 510 receives a plurality of pixel group signals, which are analog signals, and sums each pixel group signal to generate a first signal. The ADC 520 converts the first signal or the second signal, which is an analog signal, into a digital signal to generate a third signal. The reading circuit 500 outputs the third signal as a digital pixel signal (DSIG_PX). For the digital binning operation, the ADC 520 receives the multiple pixel group signals, which are analog signals, converts each pixel group signal into a digital signal, and generates multiple fourth signals.

[0052] The digital binning circuit 530 receives the plurality of fourth signals, which are digital signals, and sums the plurality of fourth signals to generate a fifth signal. In one embodiment, the digital binning circuit 530 calculates the average value of a plurality of the fourth signals to generate a sixth signal. The reading circuit 500 outputs the fifth signal or the sixth signal as a digital pixel signal (DSIG_PX).

[0053] FIG. 13 is a flowchart illustrating the operation of the depth sensor according to the embodiment of the present invention. Referring to both FIG. 1 and FIG. 13, a pixel signal (SIG_PX) is output (step S110). In one embodiment, a plurality of pixel signals are output from the depth pixel PX. In one embodiment, for example, a plurality of pixel group signals are output from the depth pixel group GPX (FIG. 7).

[0054] The external light intensity is measured based on the pixel signal (SIG_PX) (step S120). In one embodiment, the ambient light intensity is calculated by averaging multiple pixel signals. In one embodiment, the ambient light intensity is calculated by averaging multiple pixel group signals. An external light parameter value PAR is calculated based on the external light intensity (step S130). In one embodiment, the external light parameter value PAR is calculated based on the external light intensity and the intensity of the incident signal TX irradiated on the object 1 . For example, the external light parameter value PAR is calculated based on the ratio between the intensity of the incident signal TX and the intensity of the external light. The external light parameter value PAR is inversely proportional to the external light intensity. The binning mode is set based on the external light parameter value PAR (step S140). In one embodiment, the operating environment of the depth sensor 100 is determined via the external light parameter value PAR, and the binning mode of the depth sensor 100 is set according to the operating environment.

[0055] Then, a pixel group signal is output from a depth pixel group including a plurality of depth pixels according to the set binning mode, and the pixel group signal is processed to generate a digital pixel signal. For example, when set to an analog binning mode, the first pixel group signal and the second pixel group signal are summed, and the summed signal is converted into a digital pixel signal. For example, when the digital binning mode is selected, the first pixel group signal and the second pixel group signal are converted into digital signals, and the digital signals are summed to generate a digital pixel signal.

[0056] FIG. 14 is a flowchart illustrating a method for setting a binning mode of a depth sensor according to an embodiment of the present invention. Referring to both FIG. 1 and FIG. 14, it is determined whether the external light parameter value PAR exceeds a preset threshold value (step S210). If the ambient light parameter value PAR exceeds (or is equal to or greater than) the threshold value, the depth sensor 100 is set to an analog binning mode (step S220). If the external light parameter value PAR is equal to or less than the threshold value (or is less than the threshold value), the depth sensor 100 is set to a digital binning mode (step S230). It should be noted that multiple thresholds may be used to change the binning mode of the depth sensor 100. This will be explained with reference to FIG.

[0057] FIG. 15 is a graph illustrating a method for setting a binning mode of a depth sensor according to an embodiment of the present invention. 1 and 15, a first threshold TH1 and a second threshold TH2, which is smaller than the first threshold TH1, are used to change the binning mode of the depth sensor 100.

[0058] A case where the external light parameter value PAR exceeds the first threshold value TH1 will be described. The external light parameter is inversely proportional to the external light intensity, and a large external light parameter means that the external light intensity is weak. Therefore, the operating environment of the depth sensor 100 is determined to be indoors. Therefore, the depth sensor 100 reduces noise by performing analog binning. That is, for example, quantization noise that may occur in the ADC 520 (FIG. 12) of the read circuit 123 is reduced.

[0059] A case where the external light parameter value PAR is equal to or smaller than the second threshold value TH2 will be described. The external light parameter is inversely proportional to the external light intensity, and a small external light parameter means that the external light intensity is strong. Therefore, the operating environment of the depth sensor 100 is determined to be outdoors. Therefore, the depth sensor 100 reduces noise by performing digital binning. That is, the light receiving area (full well capacity: FWC) of the floating diffusion node of the depth pixel PX is secured, and saturation of the depth pixel PX is prevented.

[0060] A case where the external light parameter value PAR exceeds the second threshold value TH2 and is equal to or smaller than the first threshold value TH1 will be described. In this case, the reference threshold value may differ depending on whether the external light parameter value PAR is tending to increase or decrease. For example, if the external light parameter value PAR has a tendency to decrease during the period from the first point in time t1 to the second point in time t2, the second threshold value TH2 (or the first threshold value TH1) becomes the reference value. Therefore, at the second time point t2, the binning mode of the depth sensor 100 is changed from the analog binning mode to the digital binning mode.

[0061] For example, if the external light parameter value PAR tends to increase between the third time point t3 and the fourth time point t4 and between the fifth time point t5 and the sixth time point t6, the first threshold value TH1 (or the second threshold value TH2) becomes the reference value. Therefore, between the third time point t3 and the fourth time point t4, the binning mode of the depth sensor 100 is maintained in the digital binning mode. As a result, the binning mode of the depth sensor 100 does not change between the fourth time point t4 and the fifth time point t5. During the period from the fifth time point t5 to the sixth time point t6, when the first threshold value TH1 is the reference value, the binning mode of the depth sensor 100 is changed from the digital binning mode to the analog binning mode at the sixth time point t6. It should be noted that the number of thresholds or the threshold that becomes the reference point at which the binning mode is changed are not limited to those.

[0062] According to an embodiment of the present invention, the depth sensor 100 can prevent the binning mode from being changed too sensitively by using two thresholds (TH1, TH2). That is, even if the external light parameter value PAR changes instantaneously and frequently, the binning mode change operation can be stably executed.

[0063] FIG. 16 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention. The image processing device 10a is similar to the image processing device 10 of FIG. 1, so a duplicated description will be omitted. Referring to FIG. 16, the image signal processor (ISP) 200a performs a series of operations to determine the binning mode of the depth sensor 100a.

[0064] In one embodiment, the image signal processor (ISP) 200a receives a pixel signal (SIG_PX) generated for a reflected signal RX reflected by the object 1 from a depth sensor 100a that measures the distance to the object 1. The image signal processor (ISP) 200a calculates the external light intensity based on the pixel signal (SIG_PX). The image signal processor (ISP) 200a determines whether to perform an analog or digital binning operation of the depth sensor 100a based on the external light intensity.

[0065] Specifically, the image signal processor (ISP) 200a determines whether the operating environment of the depth sensor 100a is indoors or outdoors based on the external light intensity. If the operating environment of the depth sensor 100a is determined to be indoors, it decides to perform an analog binning operation, and if it is determined to be outdoors, it decides to perform a digital binning operation. The image signal processor (ISP) 200 a includes a parameter calculation circuit 210 and a binning mode decision circuit 220 .

[0066] The parameter calculation circuit 210 is similar to the parameter calculation circuit 310 of FIG. 8, and receives a pixel signal (SIG_PX) from the depth sensor 100a, and calculates an external light parameter value PAR based on the pixel signal (SIG_PX). The binning mode determination circuit 220 is similar to the binning mode determination circuit 320 of FIG. 8, and generates a binning mode control signal (BM_CTRL) for determining the binning mode of the depth sensor 100a based on the external light parameter value PAR and a preset threshold value, and provides the signal to the reading circuit 123a of the depth sensor 100a. The read circuit 123a generates a digital pixel signal (DSIG_PX) by performing a binning operation based on the binning mode control signal (BM_CTRL). The image signal processor (ISP) 200a receives the digital pixel signal (DSIG_PX) and generates depth data.

[0067] FIG. 17 is a block diagram showing a schematic configuration of an electronic device including a depth sensor according to an embodiment of the present invention. Referring to FIG. 17, the electronic device 1000 includes a depth sensor 1100, an image sensor 1200, a main processor 1300, a working memory 1400, a storage 1500, a display device 1600, a user interface 1700, and a communication unit 1800.

[0068] The depth sensor 1100 corresponds to the depth sensors (100, 100a) described with reference to FIGS. For example, the depth sensor 1100 generates a pixel signal by receiving a reflected signal to measure the distance to an object outside the electronic device 1000. The depth sensor 1100 calculates the external light intensity based on the pixel signal, and calculates the external light parameter value. The depth sensor 1100 determines the binning mode to be an analog binning mode or a digital binning mode depending on the external light parameter value. The depth sensor 1100 generates digital pixel signals according to the set binning mode and provides them to the main processor 1300.

[0069] The main processor 1300 generates depth data based on the digital pixel signals. The image sensor 1200 generates image data, for example raw image data, based on the received optical signals and provides the image data to a main processor 1300 . The main processor 1300 controls the overall operation of the electronic device 1000, processes event data, i.e., event signals, received from the depth sensor 1100, and detects object movements. It also receives image frames from the image sensor 1200 and performs image processing based on preset information.

[0070] The working memory 1400 stores data used for the operation of the electronic device 1000 . For example, the working memory 1400 temporarily stores packets or frames that have been processed by the main processor 1300 . For example, the working memory 1400 may include volatile memory such as dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and / or non-volatile memory such as phase change random access memory (PRAM), magnetic random access memory (MRAM), resistive random access memory (ReRAM), or ferroelectric random access memory (FRAM®). The storage 1500 stores data requested to be stored by the main processor 1300 or other components. The storage 1500 may include non-volatile memory such as flash memory, PRAM, MRAM, ReRAM, or FRAM (registered trademark).

[0071] The display device 1600 includes a display panel, a display driving circuit, and a display serial interface (DSI). For example, the display panel may be implemented by a variety of devices, such as a liquid crystal display (LCD) device, a light emitting diode (LED) display device, an organic light emitting diode (OLED) display device, or an active matrix OLED organic light emitting diode (AMOLED) display device. The display driver circuit includes a timing controller, a source driver, and the like required to drive the display panel. The DSI host built into the main processor 1300 performs serial communication with the display panel via the DSI. The user interface 1700 may include at least one of the following input interfaces: a keyboard, a mouse, a keypad, a button, a touch panel, a touch screen, a touchpad, a touch ball, a gyroscope sensor, a vibration sensor, and an acceleration sensor.

[0072] The communication unit 1800 exchanges signals with external devices / systems via an antenna 1830 . The transceiver 1810 and modem (modulator / demodulator) 1820 of the communication unit 1800 process signals exchanged with external devices / systems according to wireless communication protocols such as LTE (long term evolution), WIMAX (worldwide interoperability for microwave access), GSM (global system for mobile communication), CDMA (code division multiple access), Bluetooth (registered trademark), NFC (near field communication), Wi-Fi (wireless fidelity), and RFID (radio frequency identification).

[0073] The components of the electronic device 1000, such as the depth sensor 1100, the image sensor 1200, the main processor 1300, the working memory 1400, the storage 1500, the display device 1600, the user interface 1700, and the communication unit 1800, may be implemented using a USB (universal serial bus), a SCSI (small computer system interface), a MIPI (mobile industry processor interface), an I 2Data is exchanged based on one or more of a variety of interface protocols, such as C (inter-integrated circuit), PCIe (peripheral component interconnect express), M-PCIe (mobile PCIe), ATA (advanced technology attachment), PATA (parallel ATA), SATA (serial ATA), SAS (serial attached SCSI), IDE (integrated drive electronics), EIDE (enhanced IDE), NVMe (nonvolatile memory express), and UFS (universal flash storage).

[0074] The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the technical scope of the present invention. [Explanation of symbols]

[0075] 1 object 10, 10a Image processing device 100, 100a, 1100 Depth Sensor 110 Transmitter 111 Drive circuit 112 Light source 113 Optical system 120 Receiver 121, 300 Controller 122 pixel array 123, 123a, 400, 500 reading circuit 200, 200a Image Signal Processor (ISP) 210, 310 Parameter calculation circuit 220, 320 Binning mode decision circuit 510 Analog Binning Circuit 520 ADC 530 Digital Binning Circuit 1000 electronic equipment 1200 Image Sensor 1300 Main Processor 1400 Working Memory 1500 Storage 1600 display device 1700 User Interface 1800 Communications Department 1810 Transmitter / Receiver 1820 Modem 1830 Antenna

Claims

1. 1. A method of operating a depth sensor for measuring a distance to an object, comprising: outputting pixel signals from a plurality of depth pixels included in a pixel array; obtaining a first intensity corresponding to an external light intensity outside the depth sensor from the pixel signal; illuminating the object with an incident signal at a second intensity; calculating an ambient light parameter value comprising a ratio of the second intensity to the first intensity; and setting a binning mode of the depth sensor to an analog binning mode or a digital binning mode based on the external light parameter value.

2. A step of outputting a pixel group signal from a depth pixel group including the plurality of depth pixels according to the set binning mode; and processing the pixel group signals to generate digital pixel signals according to the set binning mode.

3. The step of generating the digital pixel signal comprises: summing a first pixel signal and a second pixel signal to form a summed analog signal when the binning mode is set to the analog binning mode; and converting said summed analog signal to said digital pixel signal.

4. The step of generating the digital pixel signal comprises: if the binning mode is set to a digital binning mode, converting the first pixel signal and the second pixel group signal into a digital signal, respectively; summing the digital signals to form a summed digital signal; and generating said digital pixel signal from said summed digital signal.

5. The step of setting the analog binning mode or the digital binning mode includes a step of setting the binning mode to the analog binning mode when the external light parameter value exceeds a threshold value; and if the external light parameter value is less than or equal to the threshold, setting the binning mode to the digital binning mode.

6. The depth sensor determines that the operating environment of the depth sensor is indoors when the external light parameter value exceeds the threshold value; The method for operating a depth sensor according to claim 5 , further comprising determining that the operating environment of the depth sensor is outdoors when the external light parameter value is equal to or less than the threshold value.

7. The pixel array includes a first depth pixel having a 4-tap structure; The step of outputting the pixel signal includes:

2. The method of claim 1, further comprising the step of outputting first to fourth pixel signals according to first to fourth signals having first to fourth phases, respectively.

8. A method for operating a depth sensor as described in claim 7, characterized in that the first to fourth phases are 0°, 90°, 180°, and 270°, respectively.

9. A method for operating a depth sensor as described in claim 7, characterized in that the step of calculating the external light parameter value includes calculating an average value of the first to fourth pixel signals.

10. The depth sensor irradiates an incident signal to the object, receives a reflected signal reflected by the object, and measures a distance to the object based on the incident signal and the reflected signal. The method of claim 1 , wherein the calculating the external light parameter value comprises calculating the external light parameter value based on the intensity of the incident signal and the external light intensity.

Images