Multifunctional Time-of-Flight Sensor and its Operation Method

Abstract

A time-of-flight (ToF) sensor and a method of operating it are provided. The ToF sensor includes at least one depth pixel with a multi-tap structure and a light source that illuminates an object. The operating mode of the ToF sensor is determined among a distance detection mode for sensing the distance to a target and multiple additional operating modes. Based on the determined operating mode, the multiple taps of the depth pixel and the light source are controlled such that the multiple taps generate multiple sampling data corresponding to the determined operating mode. Based on the multiple sampling data, a sensing result corresponding to the selected operating mode is determined. In addition to the function of the ToF sensor in measuring the distance to an object, multiple functions can be effectively performed by controlling the multiple taps of the depth pixel and the light source according to the operating mode.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0088776, filed on July 17, 2020, with the Korean Intellectual Property Office (KIPO), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The exemplary embodiments of this disclosure generally relate to semiconductor integrated circuits, and more specifically, to multifunctional time-of-flight (ToF) sensors and methods of operating ToF sensors. Background Technology

[0004] In recent years, there has been increasing interest in sensing technologies that acquire three-dimensional information about objects, leading to the development of various 3D cameras. Among these, Time-of-Flight (ToF) sensors offer a simple circuit structure and high distance resolution. ToF sensors illuminate an object with transmitted light from a light source and calculate the distance to the object by measuring the phase difference of the time of flight of the received light reflected from the object using a demodulated signal. Various electronic devices, including mobile phones, can incorporate sensors with different functions in addition to ToF sensors, thus increasing the size of the electronic devices. Summary of the Invention

[0005] Some example embodiments may provide a time-of-flight (ToF) sensor capable of performing a variety of functions, as well as methods for operating the ToF sensor to perform a variety of functions.

[0006] According to one aspect of an example embodiment, a method for operating a time-of-flight (ToF) sensor (including at least one depth pixel having a multi-tap structure and a light source that emits light onto an object) includes: determining an operating mode of the ToF sensor from a distance detection mode for sensing the distance to the object and a plurality of additional operating modes; controlling the plurality of taps of the depth pixel and the light source based on the determined operating mode, such that the plurality of taps generate a plurality of sample data corresponding to the determined operating mode; and determining a sensing result corresponding to the determined operating mode based on the plurality of sample data.

[0007] According to one aspect of another example embodiment, a time-of-flight (ToF) sensor includes a light source configured to illuminate an object with emitted light; a pixel array including at least one depth pixel having a multi-tap structure; a line scanning circuit configured to generate a plurality of sampling control signals applied to the plurality of taps of the depth pixel; and a controller configured to control the light source, pixel array, and line scanning circuit based on a mode signal indicating a distance detection mode for sensing a distance to the object and a plurality of additional operating modes, such that the plurality of taps generate a plurality of sampling data corresponding to the selected operating mode.

[0008] According to one aspect of another example embodiment, a method of operating a time-of-flight (ToF) sensor (including at least one depth pixel having a multi-tap structure and a light source for illuminating an object with emitted light) includes: determining an operating mode of the ToF sensor from a distance detection mode for sensing the distance to the object and a plurality of additional operating modes; applying a plurality of sampling control signals of different phases to a plurality of taps of the depth pixel during an integration period for collecting photocharge generated by incident light, based on the determined operating mode being a distance detection mode; and dividing the integration period into a plurality of shot periods based on the determined operating mode being one of the plurality of additional operating modes, and selectively activating the emitted light and the plurality of sampling control signals during the plurality of shot periods based on the selected operating mode.

[0009] According to one aspect of another example embodiment, an apparatus for operating a time-of-flight (ToF) sensor includes a memory storing instructions and at least one processor configured to execute the instructions to: determine an operating mode of the ToF sensor from among a distance detection mode for sensing the distance to an object and a plurality of additional operating modes; based on the determined operating mode being a distance detection mode, control the application of a plurality of sampling control signals of different phases to a plurality of taps of a depth pixel during an integration period for collecting photocharge generated by incident light; and based on the determined operating mode being one of the plurality of additional operating modes, divide the integration period into a plurality of excitation periods, and control the selective activation of emitted light and the plurality of sampling control signals during the plurality of excitation periods based on the determined operating mode.

[0010] According to one aspect of another example embodiment, a non-transitory computer-readable recording medium has recorded thereon instructions executable by at least one processor to perform a method of operating a ToF sensor, the method comprising: determining an operating mode of the ToF sensor from a distance detection mode for sensing the distance to an object and a plurality of additional operating modes; applying a plurality of sampling control signals of different phases to a plurality of taps of a depth pixel during an integration period for collecting photocharge generated by incident light, based on the determined operating mode being a distance detection mode; and dividing the integration period into a plurality of excitation periods based on the determined operating mode being one of the plurality of additional operating modes, and selectively activating emitted light and the plurality of sampling control signals during the plurality of excitation periods based on the determined operating mode.

[0011] In addition to the original function of the ToF sensor in measuring distance, the ToF sensor and the method of operating the ToF sensor according to the example embodiment can also perform multiple functions by controlling multiple taps of the depth pixels and the light source depending on the operating mode. Attached Figure Description

[0012] The exemplary embodiments of this disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a flowchart illustrating a method for operating a time-of-flight (ToF) sensor according to an example embodiment;

[0014] Figure 2 This is a block diagram illustrating a ToF sensor according to an example embodiment;

[0015] Figure 3 and Figure 4 This is a diagram illustrating example methods for measuring and calculating distances to objects;

[0016] Figure 5 This is a circuit diagram illustrating an example embodiment of a depth pixel including a ToF sensor according to an example embodiment;

[0017] Figure 6 This is a timing diagram illustrating the operation of a ToF sensor in distance detection mode according to an example embodiment;

[0018] Figure 7 This is a flowchart illustrating a method for operating a ToF sensor in object detection mode according to an example embodiment;

[0019] Figure 8 This is a timing diagram illustrating the operation of a ToF sensor in target detection mode according to an example embodiment;

[0020] Figure 9This is a flowchart illustrating a method for operating a ToF sensor in motion detection mode according to an example embodiment;

[0021] Figures 10 to 13 This is a timing diagram illustrating the operation of a ToF sensor in motion detection mode according to an example embodiment;

[0022] Figure 14 This is a flowchart illustrating a method for operating a ToF sensor in combined detection mode according to an example embodiment;

[0023] Figure 15 and Figure 16 This is a timing diagram illustrating the operation of a ToF sensor in combined detection mode according to an example embodiment;

[0024] Figure 17 This is a flowchart illustrating a method for operating a ToF sensor in wide dynamic range (WDR) mode according to an example embodiment;

[0025] Figures 18 to 21 This is a timing diagram illustrating the operation of a ToF sensor in WDR mode according to an example embodiment;

[0026] Figure 22 This is a circuit diagram showing a depth pixel with a four-tap structure according to an example embodiment;

[0027] Figure 23 This is a timing diagram illustrating an example embodiment of the operation of depth pixels in distance detection mode;

[0028] Figure 24 and Figure 25 This is a diagram illustrating an example embodiment of the structure of a floating diffusion region sharing a depth pixel;

[0029] Figure 26 This is a block diagram illustrating an electronic device according to an example embodiment.

[0030] Figure 27 It shows that it is included Figure 26 A block diagram of a camera module in an electronic device. Detailed Implementation

[0031] Various exemplary embodiments will be described more fully below with reference to the accompanying drawings, which illustrate some exemplary embodiments. In the drawings, the same reference numerals refer to the same elements throughout. Repeated descriptions may be omitted. It should be understood below that expressions such as “at least one of…”, when preceding a list of elements, modify the entire list of elements, not the individual elements in the list. For example, the expressions “at least one of [A], [B], and [C]” or “at least one of [A], [B], or [C]” mean only A, only B, only C, A and B, B and C, A and C, or A, B, and C.

[0032] Figure 1 This is a flowchart illustrating a method for operating a time-of-flight (ToF) sensor according to an example embodiment.

[0033] refer to Figure 1 In order to operate a time-of-flight (ToF) sensor comprising at least one depth pixel having a multi-tap structure and a light source that illuminates an object, a selected operating mode of the ToF sensor is determined among a distance detection mode for sensing the distance to the object and a plurality of additional operating modes (or at least one additional operating mode) (S100). Referring below... Figures 2 to 6 Describes depth pixels, ToF sensors, and distance detection modes that can correspond to the original functions of ToF sensors.

[0034] According to an example embodiment, the additional operating modes may include at least one of the following: an object detection mode for sensing the presence of an object, a motion detection mode for sensing the motion of an object, a combined detection mode for simultaneously sensing the presence and motion of an object, and a wide dynamic range (WDR) mode for sensing an object having multiple sensing sensitivities. Reference is made below. Figures 7 to 21 Describes several additional operating modes according to the example embodiments.

[0035] Based on the selected operation mode, multiple taps and light sources of depth pixels are controlled, so that multiple taps generate multiple sampling data corresponding to the selected operation mode (S200).

[0036] In distance detection mode, during the integration period for collecting photocharge generated by the incident light, a light source can be used to generate emitted light, and multiple sampling control signals of different phases can be used to measure distance. Conversely, in at least one (or all) of multiple additional operating modes, the integration period can be divided into multiple excitation periods, and multiple sampling control signals can be selectively activated during the multiple excitation periods based on the selected operating mode.

[0037] The sensing result corresponding to the selected operating mode is determined based on multiple sampled data (S300). Depending on the operating mode, the multiple sampled data may include different information. See below for reference. Figures 7 to 21 A method for determining sensing results according to an example embodiment is described.

[0038] Therefore, in addition to the distance measurement function of the ToF sensor, the ToF sensor and the method of operating the ToF sensor according to the example embodiment can also perform multiple functional modes by controlling multiple taps of depth pixels and light sources depending on the operating mode.

[0039] In the following text, see references Figures 2 to 6 Describe the configuration and operation of the ToF sensor, and refer to the following: Figures 7 to 21 Describe the additional features of using a ToF sensor.

[0040] Figure 2 This is a block diagram illustrating a ToF sensor 100 according to an example embodiment.

[0041] refer to Figure 2 The ToF sensor 100 includes a sensing unit (e.g., a sensor), a controller 150, and a light source module 200 (e.g., a light source). The sensing unit may include a pixel array 110, an analog-to-digital converter (ADC) unit 120 (or ADC), a row scanning circuit 130, and a column scanning circuit 140.

[0042] The pixel array 110 may include depth pixels that receive light RL reflected from the object OBJ after being emitted by the light source module 200 to the object OBJ. The depth pixels can convert the received light RL into an electrical signal. The depth pixels can provide information about the distance between the object OBJ and the ToF sensor 100 and / or black-and-white image information.

[0043] The pixel array 110 may further include color pixels for providing color image information. In this case, the ToF sensor 100 may be a three-dimensional color image sensor that provides color image information and depth information. According to an example embodiment, infrared filters and / or near-infrared filters may be formed on the depth pixels, and color filters (e.g., red, green, and blue filters) may be formed on the color pixels. According to an example embodiment, the ratio of the number of depth pixels to the number of color pixels may vary as needed or by design.

[0044] The ADC unit 120 can convert analog signals output from the pixel array 110 into digital signals. According to an example embodiment, the ADC unit 120 can perform column-to-digital conversion, which uses multiple analog-to-digital converters, each coupled to a plurality of column lines, to convert the analog signals in parallel. According to an example embodiment, the ADC unit 120 can perform a single analog-to-digital conversion, which uses a single analog-to-digital converter to convert the analog signals sequentially.

[0045] According to an example embodiment, the ADC unit 120 may further include a correlated double sampling (CDS) unit for extracting valid signal components. The CDS unit can perform analog double sampling, which extracts valid signal components based on the difference between an analog reset signal including a reset component and an analog data signal including a signal component. Alternatively, the CDS unit can perform digital double sampling, which converts the analog reset signal and the analog data signal into two digital signals and extracts valid signal components based on the difference between the two digital signals. Additionally, the CDS unit can perform dual correlated double sampling, which performs both analog and digital double sampling simultaneously.

[0046] The line scanning circuit 130 can receive control signals from the controller 150 and can control the line address and line scanning of the pixel array 110. To select a line among multiple line lines, the line scanning circuit 130 can apply a signal to the pixel array 110 to activate the selected line line. According to an example embodiment, the line scanning circuit 130 may include a line decoder that selects the line lines of the pixel array 110 and a line driver that applies the signal to activate the selected line line.

[0047] The column scan circuit 140 can receive control signals from the controller 150 and can control the column address and column scan of the pixel array 110. The column scan circuit 140 can output digital output signals from the ADC unit 120 to a digital signal processing circuit (not shown) and / or an external host (not shown). For example, the column scan circuit 140 can provide a horizontal scan control signal to the ADC unit 120 to sequentially select multiple analog-to-digital converters included in the ADC unit 120.

[0048] The controller 150 can control the ADC unit 120, the row scanning circuit 130, the column scanning circuit 140, and the light source module 200. The controller 150 can provide control signals, such as at least one of clock signals and timing control signals, to the ADC unit 120, the row scanning circuit 130, the column scanning circuit 140, and the light source module 200. The controller 150 may include at least one of control logic circuitry, a phase-locked loop circuit, a timing control circuit, and a communication interface circuit.

[0049] The light source module 200 can emit light of a desired (or, optionally, predetermined) wavelength. For example, the light source module 200 can emit infrared and / or near-infrared light. The light source module 200 may include a light source 210 and a lens 220. The light source 210 can be controlled by the controller 150 to emit emitted light TL with a desired intensity and / or characteristics (e.g., periodicity). For example, the intensity and / or characteristics of the emitted light TL can be controlled so that the emitted light TL has a pulse wave, sine wave, cosine wave, etc. The light source 210 can be implemented by a light-emitting diode (LED), a laser diode, etc.

[0050] In the following text, the first operation (e.g., normal operation) of the ToF sensor 100 according to an example embodiment is described.

[0051] The controller 150 can control the light source module 200 to emit emitted light TL with periodic intensity. The emitted light TL emitted by the light source module 200 can be reflected back to the ToF sensor 100 from the object OBJ as received light RL. The received light RL can be incident on a depth pixel, and the depth pixel can be activated by the row scan circuit 130 to output an analog signal corresponding to the received light RL. The ADC unit 120 can convert the analog signal output from the depth pixel into sampled data SDATA. The sampled data SDATA can be provided to the controller 150 by the column scan circuit 140 and / or the ADC unit 120.

[0052] The controller 150 can calculate the distance from the object OBJ to the ToF sensor 100, the horizontal position of the object OBJ, the vertical position of the object OBJ, and / or the size of the object OBJ based on the sampled data SDATA. The controller 150 can control the emission angle or projection (or incident) area of ​​the emitted light TL based on the distance, horizontal position, vertical position, and / or size of the object OBJ. For example, the controller 150 can control the spacing between the light source 210 and the lens 220, the relative positions (or arrangement) of the light source 210 and the lens 220, the refractive index of the lens 220, the curvature of the lens 220, etc.

[0053] The emitted light TL illuminating the object OBJ can be reflected, and the reflected light RL can be incident on the depth pixels in the pixel array 110. The depth pixels can output an analog signal corresponding to the reflected light RL, and the ADC unit 120 can convert the analog signal into digital data or sampled data SDATA. The sampled data SDATA and / or depth information can be provided to the controller 150, digital signal processing circuitry, and / or an external host. According to an example embodiment, the pixel array 110 may include color pixels and can provide color image information and depth information to the digital signal processing circuitry and / or an external host.

[0054] An external host or processor can determine the selected operating mode of the ToF sensor 100 according to various scenarios and provide a mode signal MD indicating the selected operating mode to the ToF sensor 100. Based on the mode signal MD, the controller 150 of the ToF sensor 100 can control the light source 210, the pixel array 110, and the line scanning circuit 130 to perform operations corresponding to the selected operating mode.

[0055] In some example embodiments, the controller 150 or an external host can change the selected operating mode based on sensing results of the currently selected operating mode. For example, the ToF sensor 100 can operate in one of the additional operating modes, and the selected operating mode can be switched to a distance detection mode if needed, desired, or controlled. In some example embodiments, the selected operating mode can be changed step-by-step, for example, from object detection mode to motion detection mode, and from motion detection mode to distance detection mode.

[0056] Figure 3 and Figure 4 This is a diagram illustrating an example method for measuring and calculating distances to objects.

[0057] refer to Figure 2 and 3 The emitted light TL emitted by the light source module 200 can have periodic intensity and / or characteristics. For example, the intensity of the emitted light TL (i.e., the number of photons per unit area) can have a sinusoidal waveform.

[0058] The emitted light TL emitted by the light source module 200 can be reflected from the object OBJ and then incident on the pixel array 110 as the received light RL. The pixel array 110 can periodically sample the received light RL. According to an example embodiment, during each period of the received light RL (e.g., corresponding to the period of the emitted light TL), the pixel array 110 can sample the received light RL by, for example, sampling at two sampling points with a phase difference of about 180 degrees, at four sampling points with a phase difference of about 90 degrees, or at more than four sampling points. For example, the pixel array 110 can extract four samples A0, A1, A2, and A3 of the received light RL in each period with phases of 90 degrees (or about 90 degrees), 180 degrees (or about 180 degrees), 270 degrees (or about 270 degrees), and 360 degrees (or about 360 degrees), respectively.

[0059] The received light RL can have an offset B, which is different from the offset of the emitted light TL emitted by the light source module 200 due to background light, noise, etc. The offset B of the received light RL can be calculated using Equation 1.

[0060] [Equation 1]

[0061]

[0062] Here, A0 represents the intensity of the received light RL sampled at approximately 90 degrees of phase of the emitted light TL, A1 represents the intensity of the received light RL sampled at approximately 180 degrees of phase of the emitted light TL, A2 represents the intensity of the received light RL sampled at approximately 270 degrees of phase of the emitted light TL, and A3 represents the intensity of the received light RL sampled at approximately 360 degrees of phase of the emitted light TL.

[0063] Due to losses (e.g., optical losses), the amplitude A of the received light RL can be lower than the amplitude A of the emitted light TL emitted by the light source module 200. The amplitude A of the received light RL can be calculated using Equation 2.

[0064] [Equation 2]

[0065]

[0066] Based on the amplitude A of the received light RL, black and white image information about the object OBJ can be provided by the individual depth pixels included in the pixel array 110.

[0067] The received light RL can be delayed by a phase difference Φ relative to the transmitted light TL, which corresponds to, for example, twice the distance to the object OBJ of the t-ToF sensor 100. The phase difference Φ between the transmitted light TL and the received light RL can be calculated using Equation 3.

[0068] [Equation 3]

[0069]

[0070] The phase difference Φ between the emitted light TL and the received light RL can, for example, correspond to the time of flight (TF). The distance between the object OBJ and the ToF sensor 100 can be calculated using the equation "R = c * TF / 2", where R represents the distance from the object OBJ and c represents the speed of light. Furthermore, the distance between the object OBJ and the ToF sensor 100 can also be calculated using Equation 4, using the phase difference Φ between the emitted light TL and the received light RL.

[0071] [Equation 4]

[0072]

[0073] Here, f represents the modulation frequency, which is the frequency of the intensity of the transmitted light TL (or the frequency of the intensity of the received light RL).

[0074] As described above, the ToF sensor 100 according to the example embodiment can obtain depth information about the object OBJ using the emitted light TL emitted by the light source module 200. Although Figure 3 The illustration shows an emitted light TL with a sinusoidal waveform of intensity; however, it should be understood that one or more other embodiments are not limited thereto. For example, according to the example embodiment, the ToF sensor 100 can use emitted light TL with various types of waveforms of intensity. Furthermore, the ToF sensor 100 can extract depth information based on the waveform of the emitted light TL's intensity, the structure of depth pixels, etc.

[0075] Figure 4 An example of modulation and demodulation timing for a depth pixel with a four-tap structure is shown, where the operation of the ToF sensor can be varied.

[0076] refer to Figure 4 The emitted light TL from the light source 210 can be output synchronously with the signal provided from the controller 150. First to fourth demodulated signals DEM1 to DEM4 can be generated synchronously with this signal from the controller 150. The first to fourth demodulated signals DEM1 to DEM4 have phase differences of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. (Refer to the above) Figure 3 The four samples A0, A1, A2 and A3 of the received optical RL can be sampled in each cycle at phases of approximately 90 degrees, approximately 180 degrees, approximately 270 degrees and approximately 360 degrees, respectively.

[0077] Figure 4An example is shown where the phase of the first demodulated signal DEM1 is consistent with the phase of the emitted light TL. The first to fourth demodulated signals DEM1 to DEM4 can be applied to the first to fourth demodulation transfer gates, as described below.

[0078] although Figure 3 and Figure 4 The principles described here for measuring and calculating distances to objects using a ToF sensor should be understood, and one or more other example embodiments are not limited thereto. The duty cycle of the emitted light TL and the number, phase difference, and duty cycle of the demodulated signal can be varied.

[0079] In the following text, see references Figure 5 and 6 Describes a depth pixel with a four-tap structure and operations on that depth pixel. It should be understood that one or more other example embodiments are not limited to a four-tap structure and can be applied to any depth pixel having three or more taps.

[0080] Figure 5 This is a circuit diagram illustrating an example embodiment of depth pixels including a ToF sensor.

[0081] refer to Figure 5 The depth pixel PX1 may include: a first photogate PGA and transistors TMA, TS1 and TT1 corresponding to the first tap TA; a second photogate PGB and transistors TMB, TS1 and TT1 corresponding to the second tap TB; a third photogate PGC and transistors TMC, TS2 and TT2 corresponding to the third tap TC; a fourth photogate PGD and transistors TMD, TS2 and TT2 corresponding to the fourth tap TD; transistors TRS1, TRS2, TSF1, TSF2, TSL1 and TSL2 corresponding to the readout circuit; and an overflow gate OG and a photodiode PD corresponding to the shared circuit.

[0082] Each of transistors TMA, TMB, TMC, TMD, TS1, TS2, TT1, TT2, TRS1, and TRS2 may include a gate disposed above a semiconductor substrate and source and drain regions disposed on either side of the gate disposed in the semiconductor substrate. The gates of transistors TMA, TMB, TMC, TMD, TS1, TS2, TT1, TT2, and TRS1 correspond to the first demodulation transfer gate TGA, the second demodulation transfer gate TGB, the third demodulation transfer gate TGC, the fourth demodulation transfer gate TGD, the memory gate SG, the FD transfer gate TG, and the reset gates RG1 and RG2, respectively.

[0083] The first to fourth photogate signals SPGA~SPGD are applied to the first to fourth photogates PGA~PGD, the overflow gate voltage VOG is applied to the overflow gate OG, the storage control signals SSG1 and SSG2 are applied to the storage gates SG1 and SG2, the demodulation transmission control signals STGA~STGD are applied to the demodulation transmission gates TGA~TGD, the FD transmission control signals STG1 and STG2 are applied to the FD transmission gates TG1 and TG2, the reset signals SRG1 and SRG2 are applied to the reset gates RG1 and RG2, and the selection signals SEL1 and SEL2 are applied to the gates of the selection transistors TSL1 and TSL2. (Refer to the above...) Figure 2 The controller 150 can provide photogate signals SPGA~SPGD, overflow gate voltage VOG, demodulation transmission control signals STGA~STGD, storage control signals SSG1 and SSG2, FD transmission control signals STG1 and STG2, reset signals SRG1 and SRG2, and selection signals SEL1 and SEL2 from the row scanning circuit 130.

[0084] The first to fourth photogate signals SPGA~SPGD correspond to the above reference. Figure 1 The sampling control signal is described above. The row scanning circuit 130 can generate first to fourth photogate signals SPGA to SPGD corresponding to the sampling control signal based on the selected operating mode as described below.

[0085] Storage gates SG1 and SG2 are one of the charge storage structures used to temporarily store photoelectric charge transferred from the common photogate CPG before transferring it to the floating diffusion regions FDA, FDB, FDC, and FDD. In some example embodiments, the charge storage structure can be implemented using storage gates SG1 and SG2 alone. In other example embodiments, the charge storage structure can be implemented using storage gates SG1 and SG2 and a storage diode formed in the semiconductor substrate beneath storage gates SG1 and SG2. Using such a charge storage structure, true correlated double sampling (CDS) can be performed, and noise in the readout signal can be reduced. According to example embodiments, FD transfer gates TG1 and TG2 and / or storage gates SG1 and SG2 can be omitted.

[0086] The charge stored in the floating diffusion regions FDA, FDB, FDC, and FDD can be provided as output signals, i.e., sampled data SOA to SOD, using source follower transistors TSF1 and TSF2 and select transistors TSL1 and TSL2.

[0087] Figure 6 This is a timing diagram illustrating the operation of a ToF sensor in distance detection mode according to an example embodiment.

[0088] refer to Figure 2 , Figure 5 and Figure 6 In distance detection mode, the light source 210 can generate emitted light TL modulated by the modulation frequency during the integration period TINT used to collect photocharge generated by the incident light. The line scanning circuit 130 can apply first to fourth sampling control signals, i.e., first to fourth photogate signals SPGA to SPGD with different phases, to first to fourth photogates PGA to PGD corresponding to the first to fourth taps TA to TD. In some example embodiments, the phase of the first photogate signal STGA can be synchronized with the phase of the emitted light TL. In some example embodiments, the phase difference between the first and second photogate signals STGA and STGB can be 90 degrees, the phase difference between the first and third photogate signals STGA and STGC can be 180 degrees, and the phase difference between the first and fourth photogate signals STGA and STGD can be 270 degrees. Using photogate signals STGA to STGB with different phases can be referenced as above. Figure 3 and 4 The distance to object OBJ is measured.

[0089] The overflow voltage VOG applied to the overflow gate OG can have a turn-off voltage level to prevent photocharge from draining from the photodiode PD during the integration period TINT. The demodulation transmission control signals STGA~STGD and the storage control signals SSG1 and SSG2 are activated during the integration period TINT. Therefore, the photocharge collected by the first to fourth photodiode signals SPGA~SPGD can be stored in the semiconductor substrate under the storage gates SG1 and SG2, respectively.

[0090] During other time periods, such as the reset period TRST for initializing the depth pixel PX1 and the readout period TRD for measuring the amount of photocharge collected during the integration period TINT, the overflow gate voltage VOG can have a turn-on voltage level VON to discharge the photocharge from the photodiode PD. The collected photocharge can be discharged to the terminal of the power supply voltage VDD during the periods TRST and TRD, excluding the integration period TINT. Therefore, the overflow gate OG can be used to implement a global shutter function.

[0091] At the first time point t10 during the readout period TRD (at which time the reset signal SRG1 is deactivated and the select signal SEL1 is activated), the first and second reset state data of the first and second taps TA and TB can be output through column lines COL1 and COL2, respectively. At the second time point t11 ​​during the readout period TRD (at which time the FD transfer control signal STG1 is activated and the storage control signal SSG1 is deactivated), the photocharge stored by the storage gate SG can be transferred to the floating diffusion regions FDA and FDB, and the first and second sampling data SOA and SOB of the first and second taps TA and TB can be output through column lines COL1 and COL2, respectively.

[0092] At the third time point t20 during the readout period TRD (at which time the reset signal SRG2 is deactivated and the selection signal SEL2 is activated), the third and fourth reset state data of the third and fourth taps TC and TD can be output through column lines COL1 and COL2, respectively. At the fourth time point t21 during the readout period TRD (at which time the FD transfer control signal STG2 is activated and the storage control signal SSG2 is deactivated), the photocharge stored by the storage gate SG can be transferred to the floating diffusion regions FDC and FDD, and the third and fourth sampling data SOC and SOD of the third and fourth taps TC and TD can be output through column lines COL1 and COL2, respectively.

[0093] In the following text, information about can be omitted. Figure 8 , Figure 10 , Figure 11 , Figure 12 , Figure 13 , Figure 15 , Figure 16 , Figure 18 , Figure 19 , Figure 20 and Figure 21 Redundant description of the timing diagram, where signals other than the photogate signals SPGA~SPGD are related to the reference. Figure 6 The signals described are the same.

[0094] Figure 7 This is a flowchart illustrating a method for operating a ToF sensor in object detection mode according to an example embodiment. Figure 8 This is a timing diagram showing the operation of a ToF sensor in object detection mode according to an example embodiment.

[0095] refer to Figure 7 and Figure 8When the selected operating mode is the object detection mode for sensing the presence of an object, the integration period TINT can be divided into the first excitation period TS1 and the second excitation period TS2 (S211).

[0096] The emitted light TL can be deactivated during the first excitation period TS1 (S212), and ambient light sampling data corresponding to the ambient light can be generated by activating at least one of the multiple sampling control signals for controlling multiple taps during the first excitation period TS1 (S213). For example, Figure 8 As shown, the photogate signals SPGA and SPGD corresponding to at least one first signal applied to the first and fourth taps TA and TD can be activated during the first excitation period TS1, and the first sampled data SOA and the fourth sampled data SOD can correspond to ambient light sampled data.

[0097] The emitted light TL can be activated during the second excitation period TS2 (S214), and object sampling data corresponding to the object can be generated by activating at least one second signal among a plurality of sampling control signals during the second excitation period TS2 (S215). For example, as Figure 8 As shown, the photogate signals SPGB and SPGC corresponding to at least one second signal applied to the second and third taps TB and TC can be activated during the second excitation period TS2, and the second sampled data SOB and the third sampled data SOC can correspond to the object sampled data.

[0098] Therefore, the integration period TINT can be divided into a first excitation period TS1 and a second excitation period TS2, and multi-tap operation can be performed during the second excitation period TS2 using the emitted light TL, which includes a single pulse. (See reference...) Figure 6 The operations during the readout period TRD can provide ambient light sampling data SOA and SOD, as well as object sampling data SOB and SOC.

[0099] The sensing result corresponding to the object detection mode can be determined based on ambient light sampling data SOA and SOD, and object sampling data SOB and SOC. In some example embodiments, the presence of an object within a reference distance can be determined based on the value SOB+SOC-SOA-SOD (subtracting ambient light sampling data SOA and SOD from object sampling data SOB and SOC). For example, if the value SOB+SOC-SOA-SOD is greater than a reference value corresponding to the reference distance, it can be determined that the object exists within the reference distance. In some example embodiments, the above reference value can be omitted. Figure 6The readout of reset state data at the first and third time points t10 and t20 is described to improve the readout speed. In this case, CDS is not executed, but the reset deviation of taps TA to TD can be eliminated to some extent in the value SOB+SOC-SOA-SOD itself, and the advantage of high-speed readout can be taken advantage of.

[0100] Figure 9 This is a flowchart illustrating a method for operating a ToF sensor in motion detection mode according to an example embodiment, and Figures 10 to 13 This is a timing diagram illustrating the operation of a ToF sensor in motion detection mode according to one or more example embodiments.

[0101] refer to Figure 9 , Figure 10 and Figure 11 When the selected operation mode is a motion detection mode for sensing the motion of the object, the integration period TINT can be divided into a first excitation period TS1 and a second excitation period TS2 (S221).

[0102] First object sampling data corresponding to the object can be generated (S222) by activating at least one first signal among a plurality of sampling control signals used to control a plurality of taps during the first excitation period TS1. For example, as Figure 10 and 11 As shown, the photogate signals SPGA and SPGD corresponding to at least one first signal applied to the first and fourth taps TA and TD can be activated during the first excitation period TS1, and the first sampled data SOA and the fourth sampled data SOD can correspond to the first object sampled data.

[0103] Second object sampling data corresponding to the object can be generated by activating at least one of the multiple sampling control signals during the second excitation period TS2 (S223). For example, as Figure 10 and Figure 11 As shown, the photogate signals SPGB and SPGC corresponding to at least one second signal applied to the second and third taps TB and TC can be activated during the second excitation period TS2, and the second sampled data SOB and the third sampled data SOC can correspond to the second object sampled data.

[0104] In some example embodiments, such as Figure 10 As shown, it can be controlled Figure 2 The light source 210 is used to activate the emitted light TL during the integration period TINT to include a single pulse. In this case, the first object sampling data SOA and SOD and the second object sampling data SOB and SOC correspond to the active signal associated with the emitted light TL.

[0105] In some example embodiments, such as Figure 11 As shown, it can be controlled Figure 2 The light source 210 in the middle is used to deactivate the emitted light TL during the integration period TINT. In this case, the first object sampling data SOA and SOD and the second object sampling data SOB and SOC correspond to passive signals independent of the emitted light TL.

[0106] The sensing results corresponding to the motion detection mode can be determined based on the first object sampling data SOA and SOD and the second object sampling data SOB and SOC. In some example embodiments, the motion of the object can be determined based on the difference between the second object sampling data SOB and SOC and the first object sampling data SOA and SOD, SOB+SOC-SOA-SOD. For example, if the difference SOB+SOC-SOA-SOD is positive, it can be determined that the object is moving towards the ToF sensor. Conversely, if the difference SOB+SOC-SOA-SOD is negative, it can be determined that the object is moving away from the ToF sensor. In some example embodiments, the above references can be omitted. Figure 6 The readout of reset state data at the first and third time points t10 and t20 is described to improve the readout speed. In this case, CDS is not executed, but the reset deviation of taps TA to TD can be eliminated to some extent in the value SOB+SOC-SOA-SOD itself, and the advantage of high-speed readout can be taken advantage of.

[0107] According to the example embodiment, the integration period TINT can be divided into multiple excitation periods, for example, such as Figure 12 and Figure 13 The first to fourth excitation periods TS1 to TS4 are shown. Multiple object sampling data SOA to SOD corresponding to the object can be generated by sequentially activating multiple sampling control signals (i.e., the first to fourth photogate signals SPGA to SPGD) used to control multiple taps TA to TD during the multiple excitation periods TS1 to TS4. When detecting high-speed motion, the integration period TINT can be divided into more excitation periods to obtain further subdivided object sampling data.

[0108] Figure 14 This is a flowchart illustrating a method for operating a ToF sensor in combined detection mode according to an example embodiment, and Figure 15 and Figure 16 This is a timing diagram illustrating the operation of a ToF sensor in combined detection mode according to one or more example embodiments.

[0109] refer to Figure 14 and Figure 15 When (or based on) the selected operation mode is a combined detection mode that simultaneously senses the presence and motion of an object, the integration period TINT can be divided into a first excitation period TS1, a second excitation period TS2, and a third excitation period TS3 (S231).

[0110] The emitted light TL can be deactivated during the first excitation period TS1 (S232), and ambient light sampling data corresponding to the ambient light can be generated by activating at least one first signal of a plurality of sampling control signals for controlling a plurality of taps during the first excitation period TS1 (S233). For example, as Figure 15 As shown, the photogate signals SPGA and SPGD corresponding to at least one first signal applied to the first and fourth taps TA and TD can be activated during the first excitation period TS1, and the first sampled data SOA and the fourth sampled data SOD can correspond to ambient light sampled data or ambient signals.

[0111] The emitted light TL can be activated during the second excitation period TS2 and the third excitation period TS3 (S234).

[0112] First object sampling data corresponding to the object can be generated by activating at least one second signal among a plurality of sampling control signals during the second excitation period TS2 (S235). For example, as Figure 15 As shown, the photogate signal SPGB corresponding to at least one second signal applied to the second tap TB can be activated during the second excitation period TS2, and the second sampled data SOB can correspond to the first object sampled data corresponding to the active signal associated with the emitted light TL.

[0113] Second object sampling data corresponding to the object can be generated by activating at least one third signal among multiple sampling control signals during the third excitation period TS3 (S236). For example, as Figure 15 As shown, the photogate signal SPGC corresponding to at least one third signal applied to the third tap TC can be activated during the third excitation period TS3, and the third sampled data SOC can correspond to the second object sampled data corresponding to the active signal associated with the emitted light TL.

[0114] According to an example embodiment, at least one of the plurality of sampling control signals can be deactivated during the integration period TINT to generate noise sampling data indicating the sensed noise of depth pixels. For example, as Figure 16As shown, the photogate signal SPGD applied to the fourth tap TD can be deactivated during the integration period TINT, and the fourth sampled data SOD can be used as noise sampling data or a noise signal. Sensing noise of depth pixels can include dark noise, light leakage noise, offset noise, etc.

[0115] As per the above reference Figure 7 and Figure 8 The sensing result corresponding to the object detection mode can be determined based on the first and second object sampling data SOB and SOC minus the ambient light sampling data SOA and SOD, resulting in the value SOB+SOC-SOA-SOD. Additionally, as mentioned above... Figures 9 to 13 The sensing result corresponding to the motion detection mode can be determined based on the difference between the first object sampling data SOB and the second object sampling data SOC, SOC-SOB. Figure 16 In an example embodiment, further limited sensing results can be obtained by using noise sampling data SOD to compensate ambient light sampling data SOA, first object sampling data SOB, and second object sampling data SOC.

[0116] Figure 17 This is a flowchart illustrating a method for operating a ToF sensor in wide dynamic range (WDR) mode according to an example embodiment, and Figures 18 to 21 This is a timing diagram illustrating the operation of a ToF sensor in WDR mode according to one or more example embodiments.

[0117] The WDR (Dynamic Dynamic Range) scheme is used to simultaneously capture both the dark and bright areas of an image. By combining data from the higher sensitivity areas in the darker areas with data from the lower sensitivity areas in the brighter areas, backlighting issues can be addressed, resulting in a sharper image. The WDR scheme can provide better results than the backlight compensation (BLC) scheme. Dynamic range indicates the ratio between the brightest and darkest areas within the resolvable range.

[0118] refer to Figures 17 to 21 When the selected operating mode is WDR mode for sensing objects with multiple sensing sensitivities, the integration period TINT can be divided into a first excitation period TS1, a second excitation period TS2 which is longer than the first excitation period TS1, and a third excitation period TS3 which is longer than the second excitation period TS2 (S241).

[0119] First sensitivity sampling data can be generated by activating at least one first signal among a plurality of sampling control signals used to control a plurality of taps during the first excitation period TS1 (S242). For example, as Figures 18 to 21 As shown, the photogate signal SPGA corresponding to at least one first signal applied to the first tap TA can be activated during the first excitation period TS1, and the first sampling data SOA can correspond to the first sensitivity sampling data of short exposure.

[0120] The second sensitivity sampling data (T243) can be generated by activating at least one of a plurality of sampling control signals during the second excitation period TS2. For example, as Figures 18 to 21 As shown, the photogate signal SPGB corresponding to at least one second signal applied to the second tap TB can be activated during the second excitation period TS2, and the second sampling data SOB can correspond to the second sensitivity sampling data of the middle exposure.

[0121] The third sensitivity sampling data can be generated by activating at least one of the multiple sampling control signals during the third excitation period TS3 (S244). For example, as Figure 18 and Figure 19 As shown, the photogate signals SPGC and SPGD corresponding to at least one third signal applied to the third and fourth taps TC and TD can be activated during the third excitation period TS3, and the third and fourth sampling data SOC and SOD can correspond to the third sensitivity sampling data of long exposure.

[0122] According to an example embodiment, at least one of the plurality of sampling control signals can be deactivated during the integration period TINT to generate noise sampling data indicating the sensed noise of depth pixels. For example, as Figure 20 and Figure 21 As shown, the photogate signal SPGD applied to the fourth tap TD can be deactivated during the integration period TINT, and the fourth sampled data SOD can be used as noise sampled data or a noise signal. Sensing noise of depth pixels can include dark noise, light leakage noise, offset noise, etc.

[0123] In some example embodiments, such as Figure 18 and Figure 20 As shown, Figure 2 The light source 210 can be controlled to activate the emitted light TL during the integration period TINT to include a single pulse. In some example embodiments, such as Figure 19 and Figure 21 As shown, Figure 2The light source 210 can be controlled to deactivate the emitted light TL during the integration period TINT.

[0124] Figure 22 This is a circuit diagram illustrating a depth pixel PX2 with a four-tap structure according to an example embodiment. Figure 23 This is a timing diagram illustrating an example embodiment of the operation of depth pixels in distance detection mode. The above references may be omitted in the following text. Figure 5 and 6 The description provided is redundant.

[0125] refer to Figure 22 The depth pixel PX2 may include: transistors TMA, TS1, and TT1 corresponding to the first tap TA; transistors TMB, TS1, and TT1 corresponding to the second tap TB; transistors TMC, TS2, and TT2 corresponding to the third tap TC; transistors TMD, TS2, and TT2 corresponding to the fourth tap TD; transistors TRS1, TRS2, TSF1, TSF2, TSL1, and TSL2 corresponding to the readout circuit; and a common photogate CPG, overflow gates OG1 and OG2, and a photodiode PD corresponding to the shared circuit.

[0126] Each of transistors TMA, TMB, TMC, TMD, TS1, TS2, TT1, TT2, TRS1, and TRS2 may include a gate disposed above a semiconductor substrate and source and drain regions disposed on either side of the gate in the semiconductor substrate. The gates of transistors TMA, TMB, TMC, TMD, TS1, TS2, TT1, TT2, and TRS1 correspond to the first demodulation transfer gate TGA, the second demodulation transfer gate TGB, the third demodulation transfer gate TGC, the fourth demodulation transfer gate TGD, the memory gate SG, the FD transfer gates TG1 and TG2, and the reset gates RG1 and RG2, respectively.

[0127] The photogate voltage VPG is applied to the common photogate CPG, the overflow gate voltage VOG is applied to overflow gates OG1 and OG2, the storage control signals SSG1 and SSG2 are applied to storage gates SG1 and SG2, the FD transfer control signals STG1 and STG2 are applied to FD transfer gates TG1 and TG2, the reset signals SRG1 and SRG2 are applied to reset gates RG1 and RG2, and the select signals SEL1 and SEL2 are applied to the gates of select transistors TSL1 and TSL2. First to fourth demodulation transfer control signals STGA, STGB, STGC, and STGD, with different phases, are applied to first to fourth demodulation transfer gates TGA, TGB, TGC, and TGD, respectively.

[0128] As referenced above Figure 2Under the control of the controller 150, the row scanning circuit 130 can provide photogate voltage VPG, overflow gate voltage VOG, storage control signals SSG1 and SSG2, FD transmission control signals STG1 and STG2, reset signals SRG1 and SRG2, selection signals SEL1 and SEL2, and demodulation transmission control signals STGA, STGB, STGC and STGD.

[0129] refer to Figure 22 and 23 During the integration period TINT, the photogate voltage VPG applied to the common photogate CPG can have a DC voltage level VDC for collecting photocharge, and the overflow gate voltage VOG applied to the overflow gates OG1 and OG2 can have a turn-off voltage level VOFF to prevent photocharge discharge. Additionally, during the integration period TINT, first to fourth demodulation transmission control signals STGA, STGB, STGC, and STGD of different phases can be applied to the first to fourth demodulation transmission gates TGA, TGB, TGC, and TGD, respectively. The phase of the first demodulation transmission control signal STGA can be synchronized with the phase of the emitted light TL. In some example embodiments, the phase difference between the first and second demodulation transmission control signals STGA and STGB can be 90 degrees, the phase difference between the first and third demodulation transmission control signals STGA and STGC can be 180 degrees, and the phase difference between the first and fourth demodulation transmission control signals STGA and STGD can be 270 degrees. The distance measurement method using demodulated signals of different phases is the same as described above. Figures 2 to 4 The description is the same or similar.

[0130] In a CPG with a single common photogate Figure 22 In an example embodiment of the depth pixel PX2, the first to fourth demodulation transmission control signals STGA to STGD applied to the first to fourth demodulation transmission gates TGA to TGD correspond to the aforementioned sampling control signals. (See reference...) Figures 6 to 21 The example embodiment described above can be applied in the same manner by replacing the first to fourth photogate signals SPGA to SPGD with the first to fourth demodulation transmission control signals STGA to STGD. Figure 22 The depth of the pixels is PX2.

[0131] Figure 24 and Figure 25 This is a diagram illustrating the structure of a floating diffusion region of shared depth pixels according to an example embodiment.

[0132] Figure 24 The depth pixel PX2 is essentially the same as that with a four-tap structure and a common photogate CPG. Figure 22 The depth pixels are the same as those in PX2.

[0133] like Figure 24 As shown, the first floating diffusion region FDA corresponding to the first tap TA and the second floating diffusion region FDB corresponding to the second tap TB can be electrically connected to each other via conductive path LN1. Furthermore, the third floating diffusion region FDC corresponding to the third tap TC and the fourth floating diffusion region FDD corresponding to the fourth tap TD can be electrically connected to each other via conductive path LN2. Conductive paths LN1 and LN2 may include wires above the semiconductor substrate and vertical contacts such as vias.

[0134] exist Figure 24 In the case of a depth pixel PX2, a demodulated signal having a first phase can be applied to the first and second demodulation transmission gates TGA and TGB, and a demodulated signal having a second phase different from the first phase can be applied to the third and fourth demodulation transmission gates TGC and TGD. Therefore, the sensing sensitivity of the depth pixel can be enhanced by electrically connecting at least two of the plurality of floating diffusion regions included in each depth pixel.

[0135] Figure 25 Four adjacent depth pixels PXa, PXb, PXc, and PXd in the first horizontal direction X and the second horizontal direction Y are shown for illustration. It should be understood that... Figure 2 More pixels can be repeatedly arranged in the pixel array 110.

[0136] refer to Figure 25 Four adjacent depth pixels PXa, PXb, PXc, and PXd can share a floating diffusion region. For example, a second demodulated signal can be jointly applied to four taps adjacent to the floating diffusion region FDB and respectively included in the four adjacent depth pixels PXa, PXb, PXc, and PXd, and the photocharge collected by the four adjacent depth pixels PXa, PXb, PXc, and PXd can be added to the central floating diffusion region FDB. In this way, depending on the phase of the demodulated signal, the photocharge collected by each of the four adjacent depth pixels can be added to the floating diffusion regions FDA, FDB, FDC, and FDD respectively. Therefore, the sensing sensitivity of the ToF sensor can be enhanced by using a structure that shares a floating diffusion region.

[0137] Figure 26 This is a block diagram illustrating an electronic device 1000 according to an example embodiment. Figure 27 It shows that it is included Figure 26 A block diagram of the camera module 1100b in an electronic device.

[0138] refer to Figure 26The electronic device 1000 may include a camera module group 1100, an application processor 1200, a power management integrated circuit (PMIC) 1300, and an external memory 1400.

[0139] Camera module group 1100 may include multiple camera modules 1100a, 1100b, and 1100c. At least one of camera modules 1100a, 1100b, and 1100c may include the features described in the reference above. Figures 1 to 25 The aforementioned multifunctional ToF sensor. Figure 26 Three camera modules 1100a, 1100b, and 1100c are shown as examples, and it should be understood that the example embodiments are not limited to a specific number of camera modules. According to other example embodiments, the camera module group 1100 may include two camera modules and four or more camera modules.

[0140] In the following text, see references Figure 27 This describes an example configuration for camera module 1100b. The same description can be applied to one or more of the other camera modules 1100a and 1100c.

[0141] refer to Figure 27 The camera module 1100b may include a prism 1105, an optical pathfolding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage device 1150.

[0142] Prism 1105 may include a reflecting surface 1107 to alter the path of light L incident on prism 1105.

[0143] In some example embodiments, prism 1105 can change the path of light L incident in a first direction X to a path in a second direction Y perpendicular to the first direction X. Furthermore, prism 1105 can rotate the reflecting surface 1107 about a central axis 1106 and / or rotate the central axis 1106 in the B direction to align the path of the reflected light along the second direction Y. Additionally, OPFE 1110 can move upwards in a third direction perpendicular to both the first and second directions Y.

[0144] The rotation angle of prism 1105 can be less than 15 degrees in the positive (+)A direction and greater than 15 degrees in the negative (-)A direction, but it is understood that one or more other example embodiments are not limited thereto.

[0145] Furthermore, prism 1105 can rotate within 20 degrees in both the positive B direction and the negative B direction, but it should be understood that one or more other example embodiments are not limited thereto.

[0146] In addition, the prism 1105 can move the reflecting surface 1106 in a third direction Z parallel to the central axis 1106.

[0147] OPFE 1110 may include optical lenses divided into m groups, where m is a positive integer. The m lens groups can be moved in the second direction Y to change the optical zoom ratio of camera module 1100b. For example, when K is the basic optical zoom ratio of camera module 1100b, the optical zoom ratio can be changed in the range of 3K, 5K, etc., by moving the m lens groups.

[0148] Actuator 1130 can move OPFE 1110 or optical lens to a specific position. For example, actuator 1130 can adjust the position of optical lens so that image sensor 1142 can be located at a position corresponding to the focal point of optical lens for accurate sensing.

[0149] Image sensing device 1140 may include image sensor 1142, control logic 1144, and memory 1146. Image sensor 1142 can capture or sense images using light provided through an optical lens. Control logic 1144 can control the overall operation of camera module 1100b. For example, control logic 1144 can control the operation of camera module 1100b by providing control signals through control signal line CSLb.

[0150] The memory 1146 may store information such as calibration data 1147 for the operation of the camera module 1100b. For example, the calibration data 1147 may include information for generating image data based on the provided light, such as information about the aforementioned rotation angle, focal length, optical axis, etc. When the camera module is implemented as a multi-state camera with a variable focal length depending on the position of the optical lens, the calibration data 1147 may include multiple focal length values ​​and autofocus values ​​corresponding to multiple states.

[0151] Storage device 1150 can store image data sensed using image sensor 1142. Storage device 1150 can be configured as an output of image sensor 1140, and storage device 1150 can be attached to a sensor chip containing image sensor 1140. Storage device 1150 can be implemented using electrically erasable programmable read-only memory (EEPROM), although it will be understood that one or more other example embodiments are not limited thereto.

[0152] refer to Figure 26 and Figure 27Each of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. In this case, camera modules 1100a, 1100b, and 1100c may include the same or different calibration data 1147 depending on the operation of actuator 1130.

[0153] In some example embodiments, a camera module 1100b may have a folded lens structure including the aforementioned prism 1105 and OPFE 1110, and other camera modules 1100a and 1100b may have a vertical structure without prism 1105 and OPFE 1110.

[0154] In some example embodiments, a camera module 1100c may be a depth camera configured to measure distance information of an object using infrared light. In this case, the application processor 1200 may combine the distance information provided by the depth camera 1100c with image data provided by other camera modules 1100a and 1100b to generate a three-dimensional depth image.

[0155] In some example embodiments, at least two of the camera modules 1100a, 1100b and 1100c may have different fields of view, for example, through different optical lenses.

[0156] In some example embodiments, each of the camera modules 1100a, 1100b, and 1100c may be physically separate from each other. In other words, camera modules 1100a, 1100b, and 1100c may each include a dedicated image sensor 1142.

[0157] Application processor 1200 may include image processing device 1210, memory controller 1220, and internal memory 1230. Application processor 1200 may be separate from camera modules 1100a, 1100b, and 1100c. For example, application processor 1200 may be implemented as a single chip, and camera modules 1100a, 1100b, and 1100c may be implemented as another chip or other chips.

[0158] The image processing device 1210 may include a plurality of subprocessors 1212a, 1212b and 1212c, an image generator 1214 and a camera module controller 1216.

[0159] Image data generated by camera modules 1100a, 1100b, and 1100c can be provided to subprocessors 1212a, 1212b, and 1212c via different image signal lines ISLa, ISLb, and ISLc, respectively. For example, image data transmission can be performed using a camera serial interface (CSI) based on a mobile industry processor interface (MIPI), although it will be understood that one or more other example embodiments are not limited thereto.

[0160] In some example embodiments, a subprocessor can be shared among two or more camera modules. In this case, a multiplexer can be used to selectively transfer image data from one of the camera modules to the shared subprocessor.

[0161] Image data from subprocessors 1212a, 1212b, and 1212c can be provided to image generator 1214. Image generator 1214 can generate an output image using the image data from subprocessors 1212a, 1212b, and 1212c, based on image generation information or a mode signal. For example, image generator 1213 can combine at least a portion of image data from camera modules 1100a, 1100b, and 1100c with different fields of view, based on image generation information or a mode signal, to generate an output image. Additionally, image generator 1214 can select one of the image data from camera modules 1100a, 1100b, and 1100c as the output image, based on image generation information or a mode signal.

[0162] In some example embodiments, image generation information may include a zoom factor or zoom signal. Furthermore, the mode signal may be a user-selected signal.

[0163] When the image generation information is a zoom factor and camera modules 1100a, 1100b, and 1100c have different fields of view, image generator 1214 can perform different operations depending on the zoom signal. For example, when (or based on) the zoom signal is a first signal, image generator 1214 can combine image data from different camera modules to generate an output image. When the zoom signal is a second signal different from the first signal, image generator 1214 can select one of the image data from camera modules 1100a, 1100b, and 1100c as the output image.

[0164] In some example embodiments, image generator 1214 may receive image data with different exposure times from camera modules 1100a, 1100b, and 1100c. In this case, image generator 1214 may perform high dynamic range (HDR) processing on the image data from camera modules 1100a, 1100b, and 1100c to generate an output image with increased dynamic range.

[0165] The camera module controller 1216 can provide control signals to camera modules 1100a, 1100b, and 1100c. The control signals generated by the camera module controller 1216 can be provided to camera modules 1100a, 1100b, and 1100c respectively through different control signal lines CSLa, CSLb, and CSLc.

[0166] In some example embodiments, one of camera modules 1100a, 1100b, and 1100c may be designated as the master camera based on image generation information generated from the pattern signal, while the other camera modules may be designated as slave cameras.

[0167] The camera module acting as the main camera can be changed based on the zoom factor or operating mode signal. For example, when camera module 1100a (e.g., the first camera module) has a wider field of view than camera module 1100b (e.g., the second camera module) and the zoom factor indicates a lower zoom magnification, camera module 1100b can be designated as the main camera. Conversely, when the zoom factor indicates a higher zoom magnification, camera module 1100a can be designated as the main camera.

[0168] In some example embodiments, the control signals provided from the camera module controller 1216 may include a synchronization enable signal. For example, when camera module 1100b is the master camera and camera modules 1100a and 1100c are slave cameras, the camera module controller 1216 may provide a synchronization enable signal to camera module 1100b. Camera module 1100b may generate a synchronization signal based on the provided synchronization enable signal and provide the synchronization signal to camera modules 1100a, 1100b, and 1100c via the synchronization signal line SSL. In this way, camera modules 1100a, 1100b, and 1100c may transmit synchronized image data to the application processor 1200 based on the synchronization signal.

[0169] In some example embodiments, control signals provided from camera module controller 1216 may include information about operating modes. Camera modules 1100a, 1100b, and 1100c may operate in a first operating mode or a second operating mode based on information from camera module controller 1216.

[0170] In a first operating mode, camera modules 1100a, 1100b, and 1100c can generate image signals with a first speed (e.g., a first frame rate) and encode the image signals at a second speed (e.g., a second frame rate higher than the first frame rate) for transmission to application processor 1200. The second speed can be less than thirty times the first speed. Application processor 1200 can store the encoded image signals in internal memory 1230 or external memory 1400. Application processor 1200 can read and decode the encoded image signals to provide display data to a display device. For example, subprocessors 1212a, 1212b, and 1212c can perform decoding operations, and image generator 1214 can process the decoded image signals.

[0171] In the second operating mode, camera modules 1100a, 1100b, and 1100c can generate image signals with a third speed lower than the first speed (e.g., a third frame rate lower than the first frame rate) to transmit the generated image signals to application processor 1200. That is, unencoded image signals can be provided to application processor 1200. Application processor 1200 can process the received image signals or store them in internal memory 1230 or external memory 1400.

[0172] PMIC 1300 can provide power voltages to camera modules 1100a, 1100b, and 1100c respectively. For example, under the control of application processor 1200, PMIC 1300 can provide a first power to camera module 1100a via power line PSLa, a second power to camera module 1100b via power line PSLb, and a third power to camera module 1100c via power line PSLc.

[0173] In response to a power control signal PCON from application processor 1200, PMIC 1300 can generate and control the power levels corresponding to camera modules 1100a, 1100b, and 1100c, respectively. The power control signal PCON may include power information depending on the operating mode of camera modules 1100a, 1100b, and 1100c. For example, the operating mode may include a low-power mode where camera modules 1100a, 1100b, and 1100c operate with low power. The power levels of camera modules 1100a, 1100b, and 1100c may be the same or different from each other. Furthermore, the power levels may change dynamically or adaptively.

[0174] This invention can be applied to any electronic device and system including ToF sensors. For example, it can be applied to mobile phones, smartphones, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, camcorders, personal computers (PCs), server computers, workstations, laptops, digital televisions, set-top boxes, portable game consoles, navigation systems, wearable devices, Internet of Things (IoT) devices, Internet of Everything (IoE) devices, e-books, virtual reality (VR) devices, augmented reality (AR) devices, vehicle navigation devices, videophones, monitoring systems, autofocus systems, tracking systems, motion detection systems, etc.

[0175] The foregoing description is an illustration of exemplary embodiments and should not be construed as limiting them. Although some exemplary embodiments have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without substantially departing from the inventive concept.

Claims

1. A method of operating a time-of-flight (ToF) sensor including at least one depth pixel having a multi-tap structure and a light source that irradiates emission light to an object, the method comprising: determining an operation mode of the ToF sensor from among a distance detection mode for sensing a distance to the object and a plurality of additional operation modes; controlling a plurality of taps of a depth pixel among the at least one depth pixel and the light source based on the determined operation mode, such that the plurality of taps generate a plurality of sampling data corresponding to the determined operation mode; and determining a sensing result corresponding to the determined operation mode based on the plurality of sampling data, wherein the controlling of the plurality of taps and the light source includes, based on the determined operation mode being one of the plurality of additional operation modes, dividing an integration period for collecting photocharges generated by incident light into a plurality of firing periods.

2. The method of claim 1, wherein, The plurality of additional operation modes includes an object detection mode for sensing a presence of the object, a motion detection mode for sensing a motion of the object, a combined detection mode for simultaneously sensing the presence of the object and the motion of the object, and a wide dynamic range (WDR) mode for sensing the object at a plurality of sensing sensitivities.

3. The method of claim 1, wherein, The controlling of the plurality of taps and the light source further includes: based on the determined operation mode, selectively activating the emission light and a plurality of sampling control signals for controlling the plurality of taps during the plurality of firing periods.

4. The method of claim 1, wherein, At least one of the plurality of sampling control signals for controlling the plurality of taps is deactivated during an integration period for collecting photocharges generated by incident light to generate noise sampling data indicating sensing noise of the depth pixel.

5. The method of claim 1, wherein, The controlling of the plurality of taps and the light source further includes: based on the determined operation mode being the distance detection mode, applying a plurality of sampling control signals of different phases to the plurality of taps during an integration period for collecting photocharges generated by incident light; and based on the determined operation mode being one of the plurality of additional operation modes, selectively activating the plurality of sampling control signals during the plurality of firing periods based on the determined operation mode and applying the plurality of sampling control signals to the plurality of taps.

6. The method of claim 1, wherein, The controlling of the plurality of taps and the light source further includes: based on the determined operation mode being the distance detection mode, controlling the light source to generate emission light modulated with a modulation frequency during an integration period for collecting photocharges generated by incident light; and based on the determined operation mode being one of the plurality of additional operation modes, controlling the light source to generate the emission light that is selectively activated during the plurality of firing periods based on the determined operation mode. 7.The method of claim 1, further comprising: changing the determined operation mode based on the sensing result.

8. The method of claim 1, wherein, The number of the plurality of taps is greater than two.

9. The method of claim 1, wherein, The controlling of the plurality of taps and the light source further includes, based on the determined operation mode being an object detection mode for sensing a presence of the object: dividing an integration period for collecting photo charges generated by incident light into a first excitation period and a second excitation period; deactivating the emission light during the first excitation period; generating ambient light sampling data corresponding to ambient light by activating at least one first signal of a plurality of sampling control signals for controlling the plurality of taps during the first excitation period; activating the emission light during the second excitation period; and generating object sampling data corresponding to the object by activating at least one second signal of the plurality of sampling control signals during the second excitation period. determining the sensing result includes, based on the determined operation mode being the object detection mode:

10. The method of claim 9, wherein, determining whether the object exists within a reference distance based on a value of the object sampling data minus the ambient light sampling data. controlling the plurality of taps and the light source further includes, based on the determined operation mode being a motion detection mode for sensing motion of an object:

11. The method of claim 1, wherein, dividing an integration period for collecting photo charges generated by incident light into a first excitation period and a second excitation period; generating first object sampling data corresponding to the object by activating at least one first signal of a plurality of sampling control signals for controlling the plurality of taps during the first excitation period; and generating second object sampling data corresponding to the object by activating at least one second signal of the plurality of sampling control signals during the second excitation period. determining the sensing result includes, based on the determined operation mode being the motion detection mode: determining motion of the object based on a difference between the first object sampling data and the second object sampling data.

12. The method of claim 11, wherein, controlling the plurality of taps and the light source further includes, based on the determined operation mode being a motion detection mode for sensing motion of an object: dividing an integration period for collecting photo charges generated by incident light into a plurality of excitation periods; and 13. The method of claim 1, wherein, generating a plurality of object sampling data corresponding to the object by sequentially activating a plurality of sampling control signals for controlling the plurality of taps during the plurality of excitation periods. controlling the plurality of taps and the light source includes, based on the determined operation mode being a combined detection mode for sensing both presence of the object and motion of the object: dividing an integration period for collecting photo charges generated by incident light into a first excitation period, a second excitation period, and a third excitation period; deactivating the emission light during the first excitation period; 14. The method of claim 1, wherein, generating ambient light sampling data corresponding to ambient light by activating at least one first signal of the plurality of sampling control signals for controlling the plurality of taps during the first excitation period; activating the emission light during the second excitation period and the third excitation period; generating first object sampling data corresponding to the object by activating at least one second signal of a plurality of sampling control signals during the second excitation period; and generating second object sampling data corresponding to the object by activating at least one third signal of the plurality of sampling control signals during the third excitation period. ​ ​ ​ generate second object sampling data corresponding to the object by activating at least one third signal of the plurality of sampling control signals during the third excitation period.

15. The method of claim 14, wherein, determining the sensing result comprises, based on the determined operation mode being the combined detection mode: determining whether the object exists within a reference distance based on a value after the first object sampling data is subtracted by the ambient light sampling data and a value after the second object sampling data is subtracted by the ambient light sampling data; and determining a motion of the object based on a difference between the first object sampling data and the second object sampling data.

16. The method of claim 1, wherein, controlling the plurality of taps and the light source further comprises, based on the determined operation mode being a WDR mode for sensing the object with a plurality of sensing sensitivities: dividing an integration period for collecting photo charges generated by incident light into a first excitation period, a second excitation period longer than the first excitation period, and a third excitation period longer than the second excitation period; generating first sensitivity sampling data by activating at least one first signal of a plurality of sampling control signals for controlling the plurality of taps during the first excitation period; generating second sensitivity sampling data by activating at least one second signal of the plurality of sampling control signals during the second excitation period; and generating third sensitivity sampling data by activating at least one third signal of the plurality of sampling control signals during the third excitation period.

17. A time-of-flight (ToF) sensor, comprising: a light source configured to irradiate emission light to an object; a pixel array including a depth pixel having a multi-tap structure; a row scanning circuit configured to generate a plurality of sampling control signals applied to a plurality of taps of the depth pixel; and a controller configured to control the light source, the pixel array, and the row scanning circuit based on a mode signal indicating a selected operation mode of the ToF sensor, wherein the selected operation mode is selected from among a distance detection mode for sensing a distance to the object and a plurality of additional operation modes, wherein the row scanning circuit is further configured to divide an integration period for collecting photo charges generated by incident light into a plurality of excitation periods based on the selected operation mode being one of the plurality of additional operation modes.

18. The ToF sensor of claim 17, wherein, the row scanning circuit is further configured to: based on the selected operation mode being the distance detection mode, apply a plurality of sampling control signals of different phases to the plurality of taps during the integration period for collecting photo charges generated by incident light; and based on the selected operation mode being one of the plurality of additional operation modes, selectively activate the plurality of sampling control signals during the plurality of excitation periods based on the selected operation mode and apply the plurality of sampling control signals to the plurality of taps.

19. The ToF sensor of claim 17, wherein, the light source is further configured to: based on the selected operation mode being the distance detection mode, generate the emission light modulated with a modulation frequency during the integration period for collecting photo charges generated by incident light; and based on the selected operating mode being one of the plurality of additional operating modes, dividing the integration period into a plurality of excitation periods and generating the emission light being selectively activated during the plurality of excitation periods based on the selected operating mode.

20. A method of operating a time-of-flight (ToF) sensor, the ToF sensor comprising at least one depth pixel having a multi-tap structure and a light source to illuminate emission light to an object, the method comprising: determining an operating mode of the ToF sensor from among a distance detection mode to sense a distance to the object and a plurality of additional operating modes; based on the determined operating mode being the distance detection mode, applying a plurality of sampling control signals of different phases to a plurality of taps of the depth pixel during an integration period to collect photo charges generated by incident light; and based on the determined operating mode being one of the plurality of additional operating modes, dividing the integration period into a plurality of excitation periods and selectively activating the emission light and the plurality of sampling control signals during the plurality of excitation periods based on the determined operating mode.

Images