Tof sensor operable in various pixel driving methods and operating method thereof

Abstract

Disclosed is a ToF sensor having a pixel structure including a shared area. A ToF sensor according to at least one of various embodiments disclosed herein comprises a processor for sensing a phase difference in a received image signal using a pixel structure including a plurality of pixels, wherein the processor can calculate a standard deviation value of a distance from the phase difference, and determine the driving method of the pixel structure according to the calculated standard deviation value of the distance.

Description

TOF sensor capable of operating in various pixel driving methods and its operating method

[0001] The present disclosure relates to a ToF sensor capable of operating in various pixel driving methods and an operating method thereof.

[0002] ToF (Time of Flight) sensor is a technology used to measure the distance between objects.

[0003] These ToF sensors can calculate the distance to an object by measuring the time it takes for light to reflect from the object and return using optical methods.

[0004] In the past, when sensing the phase difference in the image signal received by the ToF sensor, the sensitivity was reduced due to noise, and methods designed for noise removal, etc. had the problem of difficulty in high-speed operation.

[0005] Accordingly, there is a need for a pixel structure that can operate at high speed while removing noise and increasing sensitivity.

[0006] The present disclosure aims to provide a ToF (Time of Flight) sensor and an operating method thereof that can operate in various pixel driving methods while improving pixel sensitivity and minimizing noise of the ToF sensor.

[0007] Another object of the present disclosure is to provide a ToF sensor having a pixel structure with a phase swap function in column units and an operating method thereof.

[0008] The technical problems of the present disclosure are not limited to those described in this article, but include those that can be understood through the description of the invention.

[0009] A ToF sensor according to at least one of various embodiments of the present disclosure includes a processor that senses a phase difference in a received image signal using a pixel structure including a plurality of pixels, wherein the processor can calculate a standard deviation value of a distance from the phase difference, and determine a driving method of the pixel structure according to the calculated standard deviation value of the distance.

[0010] According to a ToF sensor according to at least one of the various embodiments of the present disclosure, if the standard deviation value of the calculated distance is greater than or equal to a threshold value, the processor may determine a driving method of the pixel structure as a first driving method so that a specific pixel among the plurality of pixels is driven as a shared pixel for adjacent pixels.

[0011] According to a ToF sensor according to at least one of the various embodiments of the present disclosure, the processor can obtain a virtual 4-tap phase difference value from each adjacent pixel according to the determined first driving method.

[0012] According to a ToF sensor according to at least one of the various embodiments of the present disclosure, when calculating depth using the acquired virtual 4-tap phase difference value, the processor may ignore the first pixel data of the first row and the last pixel data of the second row without using them.

[0013] According to a ToF sensor according to at least one of the various embodiments of the present disclosure, if the standard deviation value of the calculated distance is less than a threshold value, the processor may determine a driving method of the pixel structure as a second driving method so that a specific pixel among the plurality of pixels is not driven as a shared pixel for adjacent pixels.

[0014] According to a ToF sensor according to at least one of the various embodiments of the present disclosure, the processor can obtain a virtual two-tap phase difference value from each pixel according to the determined second driving method.

[0015] According to the ToF sensor according to at least one of the various embodiments of the present disclosure, the processor can apply the driving method of the determined pixel structure to the entire screen unit.

[0016] According to a ToF sensor according to at least one of the various embodiments of the present disclosure, the processor can apply the driving method of the determined pixel structure to at least one area unit among a plurality of areas that are preset and distinguished across the entire screen.

[0017] A method of operating a ToF sensor according to at least one of various embodiments of the present disclosure may include the steps of: sensing a phase difference in a received image signal using a pixel structure including a plurality of pixels; calculating a standard deviation value of a distance from the phase difference; and determining a driving method of the pixel structure according to the calculated standard deviation value of the distance.

[0018] According to a method of operating a ToF sensor according to at least one of the various embodiments of the present disclosure, if the standard deviation value of the calculated distance is greater than or equal to a threshold value, the processor may determine a driving method of the pixel structure as a first driving method so that a specific pixel among the plurality of pixels is driven as a shared pixel for adjacent pixels.

[0019] According to a method of operating a ToF sensor according to at least one of the various embodiments of the present disclosure, the processor can obtain a virtual 4-tap phase difference value from each adjacent pixel according to the determined first driving method.

[0020] According to a method of operating a ToF sensor according to at least one of the various embodiments of the present disclosure, the processor may ignore the first pixel data of the first row and the last pixel data of the second row when performing depth calculation using the acquired virtual 4-tap phase difference value.

[0021] According to a method of operating a ToF sensor according to at least one of the various embodiments of the present disclosure, if the standard deviation value of the calculated distance is less than a threshold value, the processor may determine a driving method of the pixel structure as a second driving method so that a specific pixel among the plurality of pixels is not driven as a shared pixel for adjacent pixels.

[0022] According to a method of operating a ToF sensor according to at least one of the various embodiments of the present disclosure, the processor can obtain a virtual 2-tap phase difference value from each pixel according to the determined second driving method.

[0023] According to a method of operating a ToF sensor according to at least one of the various embodiments of the present disclosure, the processor can apply the driving method of the determined pixel structure to the entire screen unit or to at least one area unit among a plurality of areas pre-set and distinguished from the entire screen.

[0024] According to at least one of the various embodiments of the present disclosure, the following effects are achieved.

[0025] First, the power consumption of the sensor can be reduced by reducing the amount of calculation required to obtain the standard deviation of the distance and shortening the calculation time.

[0026] Second, there is an advantage in that the sensitivity of the pixels can be improved by applying the optimal driving method to the acquired data.

[0027] Second, it has the advantage of minimizing sensor noise.

[0028] Third, there is an advantage in being able to propose a pixel structure with page swap functionality.

[0029] The technical effects of the embodiments are not limited to those described in this article, but include those that can be understood through the description of the invention.

[0030] Figure 1 is a diagram illustrating the transmission, reflection, and reception principles in a ToF system.

[0031] Figure 2 is a drawing illustrating the distance, phase distance, and FPPN of the target plane.

[0032] FIG. 3 is a drawing illustrating a ToF pixel structure according to an embodiment of the present disclosure.

[0033] FIG. 4 is a drawing illustrating a ToF pixel structure according to another embodiment of the present disclosure.

[0034] FIGS. 5 and 6 are drawings illustrating a circuit diagram type according to one embodiment of the present disclosure.

[0035] FIGS. 7 to 10 are drawings illustrating a physical pixel structure according to an embodiment of the present disclosure.

[0036] FIG. 11 is a schematic diagram illustrating signal processing in a ToF pixel according to an embodiment of the present disclosure.

[0037] FIG. 12 is a diagram illustrating an inference model for calculating the standard deviation (Std.Dev.) value of distance according to the present disclosure.

[0038] FIG. 13 is a diagram illustrating a data processing method in a ToF sensor according to the present disclosure.

[0039] FIG. 14 illustrates a block diagram of a ToF sensor according to an embodiment of the present disclosure.

[0040] FIG. 15 is a flowchart illustrating a pixel driving method determination and data processing method according to the present disclosure.

[0041] Hereinafter, an invention according to an embodiment for solving the above problem will be described in more detail with reference to the drawings.

[0042] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0043] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0044] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0045] In this application, it should be understood that terms such as “include,” “have,” or “comprising” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0046] This specification discloses a pixel structure of a sensor according to at least one of various embodiments of the present disclosure. Here, the sensor may include, for example, a Time of Flight (ToF) sensor or a ToF-based image sensor.

[0047] In the following specification, for convenience of explanation, a ToF sensor is used as an example, and the pixel structure of the ToF sensor is disclosed.

[0048] Fig. 1 (a) is illustrated to explain the principles of emission, reflection, and reception in a ToF system (or ToF sensor) in relation to the present disclosure, and Fig. 1 (b) is illustrated to explain four phase step samples.

[0049] As illustrated in (a) of Fig. 1, the ToF system may include an optical transmitter (Tx) and an optical receiver (Rx).

[0050] Here, an optical transmitter (Tx) can transmit a modulated signal that is reflected by an object, while an optical receiver (Rx) can detect the reflected signal.

[0051] At this time, the distance to the target can be measured based on the round-trip time of light from the optical transmitter (Tx) to the optical receiver (Rx). Meanwhile, if the signal is periodic, the phase change between the transmitted signal and the signal reflected from the object and received can be used as an indicator of the round-trip time.

[0052] Figure 1 illustrates the process by which an optical signal travels back and forth through the environment to a target. The distance can be calculated as in Equation 1 based on the time delay Δt and the speed of light c.

[0053]

[0054] An integrated system utilizing the TOF principle can measure distance using either pulse mode control or continuous wave (CW) mode control.

[0055] The continuous wave (CW) mode control method can estimate the phase between the two signals by using the cross-correlation operation between the transmitting and receiving signals.

[0056] As one method of phase measurement, four samples of the transmitted signal, each 90° in phase, can be considered. Electrical accumulation from the reflected signal during the four samples is performed, and the amount of photons can be detected at Q1, Q2, Q3, and Q4, as illustrated in (b) of Fig. 1.

[0057]

[0058] Looking more closely at the continuous wave (CW) phase φ, the difference between (Q3- Q4) and (Q1- Q2) can normalize the constant offset of the returned signal. The offset can occur when ambient light interferes with the transmitted signal. Additionally, the ratio between (Q3- Q4) and (Q1- Q2) can provide amplitude normalization. In reality, the amount of energy received is reduced compared to the transmitted energy due to dispersion, which results in a reduced amplitude. Independence from signal offset and attenuation is necessary for robust phase estimation. The amplitude (A) and offset (B) of the returned signal can be estimated according to Equation 3.

[0059]

[0060]

[0061] The amplitude A and offset B of the reflected signal affect the depth measurement accuracy σ. The measurement variance can be approximated by Equation 4, where the modulation contrast ratio c d This indicates how well the ToF sensor separates and collects photoelectrons. Higher amplitude, higher modulation frequency (up to physical limits), and higher modulation contrast can actually improve accuracy. Large offsets can lead to saturation and inaccuracy.

[0062]

[0063] A pixel structure of a ToF sensor according to an embodiment of the present disclosure is disclosed.

[0064] There are methods such as making the phase between rows different (0-180 row, 90-270 row), making the phase different for each pixel, and inverting the phase of the gate switch. This is because four phases (0-180, 90-270) are required to extract the distance in ToF, and since FPN (Fixed Phase Noise) occurs due to the offset effect between each switch, the phase must be inverted again to read out. In order to improve this, two phases are arranged for each pixel and the phase inversion between switches (e.g., 0-180 -> 180-0) is performed, and these are synthesized during subsequent data processing.

[0065] However, this method drives the photogates with a 90-degree phase difference per row, and adjacent pixels must be arranged so that their phases are inverted by 180 degrees. For example, in a 2-tap (e.g., photogate A, B) / pixel structure, when implementing a 4-tap structure, the phases are inverted to reduce the FPN caused by the mismatch of the left and right elements of the pixels.

[0066] However, this method has the disadvantage that mismatches between the output within the pixel, i.e. column to column, are output (V 0,90 and V 180,270 ) has a fixed location, making it difficult to solve.

[0067] To solve this, the metal routing is crossed once to invert the phase between adjacent pixels, which breaks the routing symmetry and causes FPN.

[0068] Accordingly, in the present disclosure, a pixel structure having a phase swap function between columns is disclosed so that the sensitivity of pixels can be improved by overlapping adjacent pixels while minimizing the fixed phase pattern noise (FPPN) of the ToF sensor, as low-noise pixel readout is required for sensing a phase difference in an image signal received by a ToF sensor using a modulated light source, and high sensitivity, high-speed operation, etc. are required.

[0069] Figure 2 is a drawing illustrating the distance, phase distance, and FPPN of the target plane.

[0070] FPPN, which corresponds to FPN within the depth frame, is a defect caused by a mismatch between pixels and readout.

[0071] Referring to (a) of FIG. 2, a ToF pixel may have two switches in one photodiode, each storing a photodiode signal with a phase difference of 180 degrees, and may have two output lines for each pixel.

[0072] The offset and column-to-column mismatch characteristics between transistors that constitute a switch can be expressed as the FPN of the phase signal, as shown in (c) of Fig. 2. In (c) of Fig. 2, the uppermost graph represents the target plane distance, the middle graph represents the distance, and the lowermost graph represents the FPPN.

[0073] Therefore, in this case, to minimize FPPN, it is necessary to use two frame data as shown in (b) of Fig. 2, i.e., to additionally read out a frame that reverses the phase difference between taps A and B (0-180, 180-0). Therefore, referring to Equation 5 of the frame rate due to reading out an additional frame, the speed inevitably decreases by half.

[0074]

[0075]

[0076]

[0077] FIG. 3 is a drawing illustrating a ToF pixel structure according to an embodiment of the present disclosure.

[0078] FIG. 3 illustrates an example of a pixel structure to which an adjacent pixel overlapping method according to an embodiment of the present disclosure is applied.

[0079] In the case of the pixel structure described above, the phase difference between adjacent rows must be inverted, but to invert the phase difference between adjacent rows like this, the metal routing must be crossed.

[0080]

[0081] On the other hand, in a pixel structure according to an embodiment of the present disclosure, a method of overlapping adjacent pixels is proposed. This is a method of simultaneously reading out the left and right signals of a photogate switch (PG) according to the overlapping of adjacent pixels. In other words, the phase difference between rows can be inverted so that the mismatch phenomenon between columns in the readout is eliminated. When depth is calculated in a ToF sensor according to this method, the offset between transistors (TR) can be eliminated.

[0082] The pixel structure can be, for example, a 2x2 array as a basic unit. Therefore, typically, one pixel may include one storage diode (SD) and two photodiodes (PD). Additionally, each pixel may have one tap, to which a photogate (PG) switch may be connected. However, this is not a limitation.

[0083] In Fig. 3, a two-tap structure is disclosed.

[0084] In the pixel structure according to the present disclosure, an overlapping area (350) (hereinafter referred to as a “shared area”) exists between the first pixel (310) and the second pixel (320). However, the present disclosure is not limited to this term.

[0085] Meanwhile, in the present disclosure, such shared area may be a pixel or may not be a pixel (e.g., non-pixel).

[0086] In FIG. 3, a photodiode (PD) exists in the shared area (350). Therefore, the photodiode (PD) included in the shared area (330) can be involved in both the first pixel (310) and the second pixel (320).

[0087] Meanwhile, referring to FIG. 3, two tabs (tab A and tab B) are configured based on the shared area (350).

[0088] In FIG. 3, the phases of the first pixel (310) and the second pixel (320) and the phases of the third pixel (330) and the fourth pixel (340) can be inverted in row units. That is, data with a phase of 0 degrees can be acquired in tap A, and data with a phase of 180 degrees can be acquired in tap B.

[0089] On the other hand, the phases of the first pixel (310) and the third pixel (330) and the phases of the second pixel (320) and the fourth pixel (340) in the column unit may be the same. That is, only data with a phase of 0 degrees can be acquired through the photogate (PG_A) switch of tab A, and only data with a phase of 180 degrees can be acquired through the photogate (PG_B) switch of tab B.

[0090] In summary, in accordance with a pixel structure implemented so that a shared area (350) exists between adjacent pixels according to an embodiment of the present disclosure, a signal with an inverted phase can be obtained in a row unit, and a signal with the same phase can be obtained in a column unit.

[0091]

[0092]

[0093] Referring to Equation 6, we can see that we can obtain an inverted output for each row. On the other hand, we can obtain a fixed output for each column.

[0094] FIG. 4 is a drawing illustrating a ToF pixel structure according to another embodiment of the present disclosure.

[0095] In Fig. 3, a 2-tap pixel structure based on a shared area is disclosed, while in Fig. 4, a 4-tap pixel structure based on a shared area is disclosed.

[0096] Figure 4 (a) shows a structure in which two tabs are added to the pixel structure of Figure 3 described above.

[0097] In (a) of Fig. 4, based on the basic unit, four tabs are sequentially arranged: tab B (first tab), tab A (second tab), tab B (third tab), and tab A (fourth tab).

[0098] Based on the plan view of the pixel basic unit shown in (a) of Fig. 4, the first node OD ) Tab A is placed on the left, and Tab B is placed on the right.

[0099] Meanwhile, in the present disclosure, the second node (Node A ), tab A is placed on the right side as described above, and tab B is placed on the left side. The tab B added in this way is different from the aforementioned Fig. 3.

[0100] In addition, the present disclosure provides a third node (Node B ), tab B is placed on the left as above, and tab A is placed on the right. The added tab A is different from the aforementioned Fig. 3.

[0101] Meanwhile, the first tap (tap B) can be connected to the third pixel via the photogate switch (PG_B). The second tap (tap A) can be connected to the first pixel via the photogate switch (PG_A). The third tap (tap B) can be connected to the second pixel via the photogate switch (PG_B). Finally, the fourth tap (tap A) can be connected to the fourth pixel via the photogate switch (PG_A).

[0102] Based on this tab structure and the arrangement relationship of the photogate switch, the pixel structure of Fig. 4 (a) can obtain a signal of inverted phase in the row unit, as in Fig. 3.

[0103] Meanwhile, based on the arrangement relationship of the tab structure and the photogate switch, the pixel structure of Fig. 4 (a) can obtain a signal of inverted phase even in column units, unlike Fig. 3.

[0104]

[0105]

[0106]

[0107]

[0108] Referring to mathematical equations 7 and 8 of Fig. 4 (a), when readout is performed simultaneously from tabs A and B, mismatch can be eliminated at the column level. That is, by changing the positions of tabs A and B, outputs with a 0-degree phase and a 180-degree phase can be output alternately in one column.

[0109] In this respect, Figures 3 and 4 differ. Specifically, while Figure 3 features pixel overlap, Figure 4 also enables column page swaps in addition to pixel overlap. This eliminates or minimizes not only row-level deviations but also column-level deviations.

[0110] Meanwhile, when applying the arrangement relationship of the photogate switch as shown in FIG. 4 of the present disclosure, there is no need for additional metal routing or cross-routing of metal when designing the pixel structure as in the past. Therefore, problems arising from including metal routing or applying cross-routing of metal routing can be prevented.

[0111] In addition, (b) to (d) of FIG. 4 illustrate examples of output terminal configurations at each node illustrated in (a) of FIG. 4.

[0112] Figure 4 (b) shows the first node (Node OD ) shows an example of the configuration of the output terminal. Fig. 4 (c) shows the second node (Node A ) shows an example of the configuration of the output terminal. Fig. 4 (d) shows the third node (Node B ) is an example of the configuration of the output terminal. However, this is only an example of the circuit configuration of the output terminal at the corresponding node and is not limited thereto.

[0113] FIGS. 5 and 6 are drawings illustrating a schematic type according to an embodiment of the present disclosure.

[0114] Figures 5 and 6 illustrate the layout of the pixel structure for each circuit type of the present disclosure.

[0115] Here, for convenience of explanation, FIGS. 5 and 6 are described based on the pixel structure of FIG. 4, but are not limited thereto.

[0116] First, Fig. 5 (a) shows the first circuit diagram type. Fig. 5 (b) shows the first circuit diagram type part of Fig. 5 (a) on the pixel structure layout.

[0117] Referring to (b) of Fig. 5, the first circuit diagram type can be defined as a polygonal shape. The polygonal shape can include a rectangle, a square, etc.

[0118] If a ToF sensor is implemented with the pixel layout illustrated in (b) of Fig. 5, the regularity of signal output of PG_A and PG_B within the basic pixel unit, i.e., the 2x2 array, can be guaranteed. However, the pixel layout according to the type of the first circuit may have relatively lower layout efficiency compared to (b) of Fig. 6 described later.

[0119] However, Fig. 6 (a) shows the second circuit diagram type. Fig. 6 (b) shows the second circuit diagram type part of Fig. 6 (a) on the pixel structure layout.

[0120] Meanwhile, referring to (b) of Fig. 6, the second circuit diagram type can be defined as a shape other than a polygon, for example, a trapezoidal shape.

[0121] If a ToF sensor is implemented with the pixel layout shown in (b) of Fig. 6, the efficiency of the layout of PG_A and PG_B within the basic pixel unit, i.e., the 2x2 array, can be improved, for example, the driving route of PG_A and PG_B can be simplified (e.g., using 3 lines instead of 4 lines). However, depending on the pixel layout type in the first circuit, the output of the signal to be processed by the switch through the photogate switch can be complicated. This is due to the zigzag connection, which can complicate the operation.

[0122] Meanwhile, as illustrated in (c) of FIG. 6, when a mirror type photogate (PG) switch (620) is used, the number of photogates can be reduced. For example, referring to (b) of FIG. 6, one photogate (PG) switch is placed between two photodiodes, but if the mirror type is utilized, only one photogate (PG) switch (620) can be placed between four photodiodes (610). In this way, by reducing the number of photogate (PG) switches, it is advantageous in securing space, and thus the degree of freedom in design can be increased. In addition, the efficiency of the circuit can be improved.

[0123] FIGS. 7 to 10 are drawings illustrating a physical pixel structure according to an embodiment of the present disclosure.

[0124] In conventional pixel structures, pixels are physically separated from each other by using Shallow Trench Isolation (STI) between adjacent pixels.

[0125] In Fig. 7, a pixel structure according to a first embodiment is disclosed. At this time, Fig. 7 (a) may represent a plan view, and Fig. 7 (b) may represent a cross-sectional view.

[0126] Fig. 7 is basically the same as the conventional pixel structure described above. That is, referring to (a) and (b) of Fig. 7, it can be seen that adjacent pixels are physically separated based on STI.

[0127] In the present disclosure, the sensitivity against the same pitch can be improved by using an adjacent pixel overlapping structure, i.e., a shared area between adjacent pixels, similar to pixel binning, which groups or combines multiple pixels and processes them as a single large pixel.

[0128] In addition, according to the adjacent pixel overlapping structure of the present disclosure, the directional signals (left, right) of the photogates (PG_A, PG_B) can be simultaneously stored in a storage node. Here, the directional signal can refer to a signal having a slight difference in current and voltage depending on the left and right arrangement of the switch in a semiconductor process, for example.

[0129] Even in the first embodiment, although adjacent pixels are separated from each other by the STI process, the physical separation through STI can be artificially connected by connecting adjacent gates (711, 712) with metal.

[0130] In Fig. 8, a pixel structure according to a second embodiment is disclosed. Similarly, Fig. 8 (a) may represent a plan view, and Fig. 8 (b) may represent a cross-sectional view.

[0131] Referring to (a) and (b) of FIG. 8, a shared area (813) may exist between adjacent pixels, i.e., a first pixel (811) and a second pixel (812). In this case, the shared area (813) may be a pixel.

[0132] A gate called a photogate (PG) may be connected between the first pixel (811) and the shared pixel (813) through metal routing. Similarly, a gate called a photogate (PG) may be connected (i.e., merged) between the second pixel (812) and the shared pixel through metal routing.

[0133] Therefore, in the second embodiment, unlike the first embodiment, a shared pixel (813) exists between adjacent pixels (811, 812), and since this shared pixel (813) plays the same role as an adjacent pixel of the target pixel, it is expressed as overlapping. In this way, since the shared pixel (813) exists, STI processing between adjacent pixels through STI may not be necessary, as in the first embodiment.

[0134] Additionally, the second embodiment is characterized by having a smaller number of photodiodes (PDs) than the first embodiment. While four photodiodes (PDs) were required to connect two gates in the first embodiment, only three photodiodes (PDs) may be sufficient to connect two gates in the second embodiment.

[0135] A third embodiment is illustrated in Fig. 9. The basic concept of this third embodiment is similar to that of the second embodiment of Fig. 8.

[0136] In the second embodiment, for example, as illustrated in FIG. 8, an N+ doping layer may be formed in the active region below the storage gate (SG) (821) and / or around the photodiode (PD) between the first pixel (811) and the shared pixel (813). However, the N+ doping layer formed in this manner may induce, for example, a dark current.

[0137] Therefore, in Fig. 9, unlike Fig. 8, a layout structure is proposed that minimizes the aforementioned dark current.

[0138] In the second embodiment described above, the photogate (PG) is connected by metal routing, but in the third embodiment, the gates are merged and the N-type junction diode under the storage gate can be extended to be connected to the photogate (PG). Through this, in the third embodiment, the formation of the N+ doping layer can be prevented, thereby minimizing the induction of dark current.

[0139] Meanwhile, in the second or third embodiment, since the structure of the pixels is an overlapping structure including a shared area, a shared pixel (A, B, C) is placed between each pixel, such as 1-A-2-B-3-4-C-5, so there is no need for a physical separation process of adjacent pixels through an STI process.

[0140] Meanwhile, Fig. 10 shows a cross-sectional view between A-A' in (a) of Fig. 8 and between B-B' in (a) of Fig. 9.

[0141] FIG. 11 is a waveform diagram illustrating signal processing in a ToF pixel according to an embodiment of the present disclosure.

[0142] The signal waveform of the ToF pixel shown in (a) of Fig. 11 is divided into a single first signal period (global period) and a repetitive second signal period (iteration period).

[0143] In the above, the first signal section may be, for example, a global reset section.

[0144] Referring to (a) of Fig. 11, the first signal section may include a global reset, modulation & integration, and anti-blooming section.

[0145] On the other hand, the repetitive second signal period may include reset sampling, signal transfer, signal sampling / analog-to-digital conversion, and blank periods.

[0146] Meanwhile, in relation to the present disclosure, the output (PGA / B) (1110) from the A tap and the B tap can be obtained in the first signal section.

[0147] Figure 11 (b) is an enlarged waveform diagram of the output (PGA / B) (1110) from the A and B taps.

[0148] Referring to (b) of Fig. 11, the outputs of PG_A and PG_B may have waveforms with phases inverted from each other.

[0149] Hereinafter, in a processor according to the present disclosure, a pixel structure including a plurality of pixels is used to sense a phase difference in a received image signal, and then a standard deviation (Std.Dev.: Standard Deviation) value of a distance (i.e., phase) is calculated from the phase difference, and a driving method of the pixel structure is determined based on the standard deviation value (Std. Dev. value) of the distance thus calculated.

[0150] At this time, according to the present disclosure, if the standard deviation value of the calculated distance is a first value or a first range, a specific pixel among a plurality of pixels can be driven as a shared pixel for adjacent pixels. This will be conveniently described as a "first driving method." In the above, the first value or the first range may represent, for example, a predefined threshold value or a value greater than or equal to the threshold value.

[0151] According to this first driving method, the processor can obtain a virtual 4-tap phase difference value from each of the adjacent pixels of the shared pixel.

[0152] On the other hand, according to the present disclosure, if the standard deviation value of the calculated distance is a second value or a second range, a specific pixel among a plurality of pixels can be prevented from being driven as a shared pixel for adjacent pixels. This is described and named as a "second driving method" to distinguish it from the first driving method, but is not limited to this name. In the above, the second value or the second range may represent, for example, a predefined threshold value or a value less than the threshold value.

[0153] In this second driving method, the processor can obtain virtual two-tap phase difference values ​​from each pixel because there are no shared pixels. Therefore, in this case, four phase difference values ​​can be obtained from two pixels compared to the first driving method, so the frame rate is bound to be reduced.

[0154] Meanwhile, the first or second driving method described above can be applied to the entire screen unit, for example, in the pixel structure according to the present disclosure.

[0155] Alternatively, the driving method may be applied to at least one area (or area unit) among a plurality of areas that are set to be distinguished in advance for the entire screen.

[0156] That is, the embodiments below are applicable to both a conventional pixel structure (second driving method) and a pixel structure (first driving method) in which a specific area can be shared by adjacent pixels as illustrated in FIGS. 3 to 9 described above.

[0157] In other words, the following embodiment relates to determining whether to drive a pixel structure in a first driving manner or a second driving manner when processing a received image signal, and processing the image signal accordingly.

[0158] Therefore, in the above-described embodiment, the shared area may be defined as a pixel hereinafter.

[0159] In relation to this, in determining the pixel driving method below, the standard deviation value of the distance is used, and in order to obtain the standard deviation value, the original data of the previous frame must be available, but in the present disclosure, this can be used by defining an estimation model method according to the present disclosure.

[0160] Therefore, according to the present disclosure, even without the raw data of the previous frame, the standard deviation estimation model can be used to reduce the amount of computation and thus the computation time. This reduction in computational amount and computation time ultimately results in the benefit of lowering the power consumption of the device.

[0161] Meanwhile, in the case of following the standard deviation estimation modeling method according to the above-described disclosure, the quality of the distance calculation result of the acquired data can be known, and thus, it is possible to determine whether to apply the pixel driving method, i.e., the binning method, and thereby increase the sensitivity and / or resolution of the sensor device.

[0162] Therefore, by actively determining the driving method and referring to the pixel driving method and the modeled results according to the present disclosure, the obtained result value can also contribute to improving the reliability of the distance calculation result.

[0163] In relation to the present disclosure, as described above, in the case of the first driving method, since pixels are used as shared areas for adjacent pixels, it can be viewed as a driving method that relatively improves the sensitivity of data by sacrificing loss of data resolution.

[0164] In other words, in the first driving method, in order to increase the depth frame rate, AB and BA frames are not extracted separately, but both AB and BA are obtained from one frame, so that compensation values ​​are obtained from up, down, left, and right, and ultimately 180-degree phase difference information can be secured.

[0165] However, in the present disclosure, if the data sensitivity is sufficient, i.e., if the data sensitivity is greater than a preset value, it may be more efficient to apply the second driving method rather than the first driving method described above. This is because, if the first driving method described above is followed, the dynamic range may be reduced.

[0166] In this way, the present disclosure requires a pixel driving method to be determined, and the criteria for such determination may be, for example, whether data sensitivity is sufficient, or whether improvement in data sensitivity is required. However, the present disclosure is not limited thereto.

[0167] For example, if the amount of reflected light received is insufficient, the sampled signal may be weakened. A weakened sampled signal can also result in a lower signal amplitude. Furthermore, a lower signal amplitude ultimately increases the standard deviation (Std.Dev.) of the acquired distance (i.e., phase).

[0168] Meanwhile, the standard deviation (Std.Dev.) value of distance may be closely related to, for example, the signal-noise ratio (SNR) and the reliability of the distance calculation result.

[0169] Typically, the standard deviation (Std.Dev.) of distance must be calculated based on accumulated data from at least several frames (e.g., several to several dozen) to obtain accurate data values. This can be a burden on the sensor or processor, requiring not only computational effort and time, but also storage space.

[0170] In this specification, a standard deviation (Std.Dev.) inference model of distance according to the present disclosure is disclosed.

[0171] Such a standard deviation (Std.Dev.) inference model of distance can be, for example, a model that can infer the standard deviation (Std.Dev.) of distance using only information from a minimum number of frames. Here, the minimum can mean, for example, one frame or at least a number of frames significantly less than the number of frames used in conventional methods.

[0172] FIG. 12 is a diagram illustrating an inference model for calculating the standard deviation (Std.Dev.) value of distance according to the present disclosure.

[0173] Referring to Figure 12, a standard deviation (Std.Dev.) frame of distance can first be generated, and the σ value can be calculated for each pixel. Next, a target value can be set, and the mean value of the σ data (mean of temporal Std.Dev.) can be calculated.

[0174] In this way, in the standard deviation (Std.Dev.) inference model of distance according to the present disclosure, the standard deviation (Std.Dev.) of distance can be inferred and a calibration process can be performed using only information from a minimum number of frames (e.g., one frame). In this case, the calibration process can extract, for example, the corresponding parameter.

[0175] The parameters extracted in this way can be used to determine the standard deviation (Std.Dev.) of distance with a simple calculation operation such as the one below.

[0176]

[0177] In the above, d i can represent the distance value of pixel i, and a i can represent the value of the intensity (amplitude component) of pixel i. And f can represent a modeling function. σ d The value can be set to 1 for computational problems.

[0178]

[0179] Accordingly, in the sensor according to the present disclosure, the pixel driving method can be determined based on the standard deviation (Std.Dev.) of the distance calculated through the above-described inference model.

[0180] Referring again to FIG. 4 to explain the first driving method, in order to increase sensitivity, the first driving method may be designed so that two photodiodes (PDs) are turned on simultaneously for each phase by overlapping or superimposing adjacent photodiodes (PDs) with photogates. For example, in the case of a 3.5um pitch, sensing can be performed at 3.5 x 7.0um, but is not limited thereto.

[0181] However, the taps within the pixel are configured as two taps as before, but by designing them to be shared with adjacent photodiodes (PDs), the driving and lead-out of the photogate driver can be used as is without changing the existing structure.

[0182] However, as described above, in order to increase the depth frame rate, in FIG. 4, rather than extracting AB and BA frames separately, it is possible to arrange for a 180-degree phase difference operation in all directions, i.e., page swap, to be possible so that both AB and BA can be obtained in one frame.

[0183] Referring to FIGS. 4 and 13, the first driving method is described in more detail.

[0184] Fig. 13 (a) may be a top-view illustration of the first driving method and the related pixel structure, Fig. 13 (b) may be an illustration to explain data output in the first driving method, and Fig. 13 (c) may be a cross-sectional illustration of Fig. 13 (a).

[0185] Referring to (a) and (b) of FIG. 4 and FIG. 13, the processor can obtain the phase required for one pixel from the pixel adjacent to the pixel on the left or right (odd, even with different direction between lines).

[0186] In this case, the processor can use upper frame / lower frame data, such as AB and BA, where page swap data is output simultaneously.

[0187] Referring to (b) of Fig. 13, it can be seen that two rows (r01, r02) can be read out simultaneously.

[0188] That is, in a 2-way read-out (upper / lower) structure, the left (Odd) row can obtain the phase value required for the corresponding pixel from the adjacent left pixel.

[0189] On the other hand, in a 2-way read-out structure, the even row can obtain the phase value required for that pixel from the adjacent right pixel.

[0190] In (b) of Fig. 13, in the case of the r01(0,0) pixel, the B phase signal is read out from the (-1,0) pixel, and the (0,0) pixel operates normally with A, B 2 taps, and in the case of the r02(639,0) pixel, the B phase signal can also be read out from the (640,0) pixel.

[0191] In other words, by adding 2 x 2 units to the left and right of the 640 x 480 array, the actual operation (PG B ,Vout B ) can be arranged so that it can be placed.

[0192] In relation to this, in the present disclosure, the first pixel data of the left row (e.g., odd row) and the last pixel data of the right row (e.g., even row) can be ignored or filtered during depth calculation.

[0193] FIG. 14 illustrates a block diagram of a ToF sensor according to an embodiment of the present disclosure.

[0194] Referring to Fig. 14, the ToF sensor can be divided into a data acquisition configuration block (left of AA') and a depth data acquisition configuration block (right of AA'), for example, based on A-A'. However, this is only an example and is not limited thereto.

[0195] The data acquisition configuration block can be configured to include a device unit (1410), a driving circuit (1421, 1422), a driving method determination unit (1430), etc.

[0196] The data acquisition configuration block generates a modulation signal of a light source of a light emitting unit, and generates a control signal to perform demodulation in synchronization with the generated modulation signal, so that a signal can be received according to each phase delay within the light receiving sensor.

[0197] The element portion (1410) includes a pixel structure including a plurality of pixels.

[0198] In the present disclosure, there may be a plurality of driving circuits (1421, 1422). That is, the number of driving circuits may be determined according to the number of driving methods determined in the driving method determination unit (1430) described later.

[0199] For convenience, in this disclosure, as described above, the first driving method and the second driving method are disclosed, and in Fig. 14, only two driving circuits, i.e., the first driving circuit (1421) and the second driving circuit (1422), are disclosed. However, the present disclosure is not limited thereto.

[0200] Each driving circuit (1421, 1422) can acquire data by driving the sofa unit (1410) according to the corresponding driving method. The data acquired in this way can be provided to, for example, a depth data acquisition configuration block.

[0201] The driving method determining unit (1430) can determine the driving method of the element unit (1410) based on the depth noise estimation value transmitted or fed back from the depth data acquisition configuration block. The driving method determining unit (14300) can control the operation of the element unit (1410) by controlling the corresponding driving circuit (1421 or 1422) according to the determined driving method.

[0202] The driving method determination unit (1430) can determine whether to perform binning, i.e., the first or second driving method, based on the degree of depth noise estimated by the depth noise estimation unit (1445) described later.

[0203] The driving method determination unit (1430) may determine to operate in the first driving method (or shared pixel mode), for example, when the estimated amount of noise is greater than or equal to a threshold. On the other hand, the driving method determination unit (1430) may determine to operate in the second driving method (or individual pixel mode), for example, when the estimated amount of noise is less than a threshold (e.g., when resolution is important).

[0204] The driving mode determination unit (1430) can determine the driving mode based on the extracted σ.

[0205] The driving method decision unit (1430) can select, for example, the first driving method (i.e., shared pixel) if the σ value is greater than T (a certain standard), and otherwise the second driving method (i.e., individual pixel) driving.

[0206] If the results differ significantly or are frequently repeated (when σ exists near T (boundary)) due to the difference between the first and second driving methods, the sensor output may appear to be shaky. Therefore, hysteresis can be applied to resolve this problem. For example, when hysteresis is applied, the current value or setting can be maintained for at least n frames (where n is a natural number). Here, n can be set arbitrarily and may be determined by an artificial intelligence engine (not shown) through learning, or may be determined by scene unit, object unit, etc.

[0207] Next, the depth data acquisition configuration block includes a data operation unit (1440), a depth noise estimation unit (1445), a distance correction unit (1450), a lens correction unit (1460), a conversion unit (1470), etc., and can acquire depth data based on the pixel data transmitted from the aforementioned data acquisition configuration block.

[0208] The data calculation unit (1440) can calculate the values ​​of the amplitude, phase, and intensity of the reflection from the acquired data. The data calculation unit (1440) can use the aforementioned mathematical expressions 2 and 3, etc., for the calculation.

[0209] The data operation unit (1440) is a component that performs the operation operations that are basically performed in ToF signal processing, and can perform a 4-phase sampling or 3-phase sampling operation with reference to the aforementioned mathematical expressions 2 and 3.

[0210] The element unit (1410) or data operation unit (1440) can convert the accumulated light quantity information from the ROIC (Read Out Integration Circuit) into a digital number (DN) and receive it.

[0211] The depth noise estimation unit (1445) can estimate depth noise by receiving the result calculated by the data calculation unit (1440).

[0212] The depth noise estimation unit (1445) is a configuration that estimates noise of depth, i.e., distance and phase, and can estimate the level of noise component of the final depth (distance) based on the depth noise model based on given information (e.g., amplitude, phase, intensity, etc. calculated by the data calculation unit (1440).

[0213] The distance correction unit (1450) can correct the distance value by receiving the result calculated by the data calculation unit (1440).

[0214] In relation to the operation of the distance correction unit (1450), distance correction can perform correction for errors occurring due to various causes.

[0215] At this time, the error between each pixel to be corrected may include FPPN (Fixed Phase Pattern Noise), FPN (Fixed Pattern Noise), etc.

[0216] Meanwhile, the error due to distance may include wiggling error, i.e. periodic function component error due to harmonic signal by modulation / demodulation.

[0217] In addition, errors due to the temperature of the light emitting / receiving unit may include or cause additional errors, such as delays due to the temperature of the light emitting element and the light receiving element.

[0218] The lens correction unit (1460) can correct the lens value by receiving the result calculated by the data calculation unit (1440).

[0219] The lens correction unit (1460) can perform correction by considering lens distortion parameters (barrel-pincushion, tangential distortion).

[0220] The conversion unit (1470) can convert data that has passed through the data operation unit (1440), distance correction unit (1450), and lens correction unit (1460) into three-dimensional coordinates.

[0221] In this way, depth data can be generated by converting into three-dimensional coordinates through the conversion unit (1470).

[0222] In a vision camera system, correction can be performed based on precisely extracted parameters to convert 2D data into 3D data.

[0223] These extracted parameters may include camera intrinsic parameters, lens distortion parameters, and camera extrinsic parameters.

[0224] For example, among the extracted parameters, the camera internal parameters may include parameters that are present in the camera model (e.g., pinhole camera model) when projected from 3D coordinates to 2D, such as focal length, principal point, skew coefficient, etc.

[0225] Among the extracted parameters, lens distortion parameters may include, for example, barrel-pincushion and tangential distortion.

[0226] Among the extracted parameters, the camera external parameters may include parameters that move from the 3D world coordinate system to the 3D camera coordinate system, such as rotation and translation.

[0227] Meanwhile, the conversion unit (1470) can convert 2D data into 3D data based on a camera model (e.g., a pinhole camera) in the 3D coordinate conversion process. That is, distance data (d) can be added to the u, v positions corresponding to pixels to convert them into X, Y, and Z.

[0228] In Fig. 14, other components except for the component part may be components included in the processor. Such a processor may also be implemented in the form of a single chip.

[0229] FIG. 15 is a flowchart illustrating a pixel driving method determination and data processing method according to the present disclosure.

[0230] A ToF sensor can sense a phase difference in a received image signal using a pixel structure including multiple pixels. The ToF sensor can calculate a standard deviation value of a distance based on the sensed phase difference. The ToF sensor can determine an operating method of the pixel structure based on the calculated standard deviation value of the distance.

[0231] In relation to this, as illustrated in FIG. 15, the ToF sensor can acquire raw data (S20) when pixel driving starts (S10).

[0232] The ToF sensor can calculate values ​​such as amplitude, phase, and intensity of reflected light based on the acquired raw data (S30).

[0233] The ToF sensor can estimate depth noise by simulating it with a depth noise inference model using values ​​calculated in the S30 operation (S40).

[0234] The ToF sensor can determine whether the σ value is less than T (a certain standard) based on the depth noise estimated from the S40 operation (S50).

[0235] The ToF sensor can apply hysteresis if the σ value is smaller than T (a certain standard) as a result of the S50 operation judgment (S60).

[0236] The ToF sensor can perform up to S60 operation and then return to S10 operation and repeat the operation.

[0237] The ToF sensor determines the S50 operation, and if the σ value is greater than T (a certain standard), the driving method can be changed and the S10 operation can be performed according to the changed driving method.

[0238] Although the present invention has been described above with reference to embodiments thereof, it will be readily understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0239] The present disclosure can process a ToF signal as described above, and the ToF sensor and its operating method according to the present disclosure can be applied or utilized in devices used in various fields such as automobiles, robots, industrial automation, eXtended Reality, and environmental monitoring, and thus has industrial applicability.

Claims

1. A processor that senses a phase difference in a received image signal using a pixel structure including a plurality of pixels, The above processor, Calculate the standard deviation value of the distance from the above phase difference, The driving method of the pixel structure is determined based on the standard deviation value of the calculated distance. ToF (Time of Flight) sensor.

2. In claim 1, The above processor, If the standard deviation value of the distance calculated above is greater than or equal to a threshold value, the driving method of the pixel structure is determined as the first driving method so that a specific pixel among the plurality of pixels is driven as a shared pixel for adjacent pixels. ToF sensor.

3. In claim 2, The above processor, Obtaining a virtual 4-tap phase difference value from each adjacent pixel according to the first driving method determined above, ToF sensor.

4. In claim 3, The above processor, When calculating depth using the virtual 4-tap phase difference value obtained above, The first pixel data of the first row and the last pixel data of the second row are ignored and not used. ToF sensor.

5. In claim 1, The above processor, If the standard deviation value of the distance calculated above is less than a threshold value, the driving method of the pixel structure is determined as the second driving method so that a specific pixel among the plurality of pixels is not driven as a shared pixel for adjacent pixels. ToF sensor.

6. In claim 5, The above processor, Obtaining a virtual 2-tap phase difference value from each pixel according to the second driving method determined above, ToF sensor.

7. In claim 1, The above processor, Applying the driving method of the pixel structure determined above to the entire screen unit, ToF sensor.

8. In claim 1, The above processor, The driving method of the above-determined pixel structure is set in advance across the entire screen and applied to at least one area unit among a plurality of areas divided, ToF sensor.

9. A step of sensing a phase difference in a received image signal using a pixel structure including a plurality of pixels; A step of calculating a standard deviation value of distance from the phase difference; and Including a step of determining the driving method of the pixel structure according to the standard deviation value of the calculated distance, How the ToF sensor works.

10. In claim 9, The above processor, If the standard deviation value of the distance calculated above is greater than or equal to a threshold value, the driving method of the pixel structure is determined as the first driving method so that a specific pixel among the plurality of pixels is driven as a shared pixel for adjacent pixels. How the ToF sensor works.

11. In claim 10, The above processor, Obtaining a virtual 4-tap phase difference value from each adjacent pixel according to the first driving method determined above, How the ToF sensor works.

12. In claim 11, The above processor, When calculating depth using the virtual 4-tap phase difference value obtained above, The first pixel data of the first row and the last pixel data of the second row are ignored and not used. How the ToF sensor works.

13. In claim 9, The above processor, If the standard deviation value of the distance calculated above is less than a threshold value, the driving method of the pixel structure is determined as the second driving method so that a specific pixel among the plurality of pixels is not driven as a shared pixel for adjacent pixels. How the ToF sensor works.

14. In claim 13, The above processor, Obtaining a virtual 2-tap phase difference value from each pixel according to the second driving method determined above, How the ToF sensor works.

15. In claim 9, The above processor, The driving method of the above-determined pixel structure is applied to at least one area unit among a plurality of areas preset on the entire screen or the entire screen. How the ToF sensor works.

Images