TOF sensor and distance calculation method thereof

Abstract

A time-of-flight (TOF) sensor and a distance calculation method thereof are disclosed. A TOF sensor according to at least one of various embodiments of the present disclosure may comprise: a light-emitting unit for emitting light toward an object; a light reception unit for receiving reflected light having a delayed phase from the object; and a processor for calculating a distance to the object on the basis of the reflected light, wherein the processor estimates a delay phase value at a second pixel on the basis of a first pixel among multiple pixels included in the light reception unit, and calculates a distance value to the object at the second pixel on the basis of the estimated delay phase value at the second pixel.

Description

TOF sensor and its distance calculation method

[0001] The present disclosure relates to a ToF sensor, and more particularly, to a distance calculation (or measurement) method and a ToF sensor when at least one pixel is saturated.

[0002] A Time of Flight (ToF) sensor is a technology used to measure the distance between objects. These ToF sensors use optical methods to measure the time it takes for light to reflect from an object and return, thereby calculating the distance to that object.

[0003] In this way, when measuring the distance to an object using an optical method in a ToF sensor, if the target object is close by, or if the light is reflected from an object with high reflectivity even though the distance is somewhat far, there are cases where the pixels become saturated by the received light.

[0004] When pixels become saturated, the distance values ​​calculated from those pixels become distorted. Therefore, the distance measurement values ​​between the ToF sensor and the target object, including the distance calculation values ​​for saturated pixels, are problematic due to their low accuracy.

[0005] In this way, a method for calculating or measuring the distance to a target object more accurately is required even when saturation occurs in the pixels of the ToF sensor.

[0006] One embodiment of the present disclosure is to provide a method and device for detecting saturated pixel(s) in a sensor.

[0007] An embodiment of the present disclosure provides a method and device for obtaining a distance operation value in a saturated pixel.

[0008] One embodiment of the present disclosure provides a method and device for correcting and outputting at least one of a phase and a depth from a sensor when at least one pixel is saturated.

[0009] The technical tasks of the embodiment are not limited to those described in this article, but include those that can be understood through the description of the invention.

[0010] A Time of Flight (ToF) sensor according to at least one of various embodiments of the present disclosure comprises: a light emitting unit that irradiates light toward an object; a light receiving unit that receives reflected light having a delayed phase from the object; and a processor that calculates a distance to the object based on the reflected light; wherein the processor can estimate a delayed phase value in a second pixel based on a first pixel among a plurality of pixels included in the light receiving unit.

[0011] According to at least one of the various embodiments of the present disclosure, the processor can calculate a distance value for the object at the second pixel based on the estimated delay phase value at the second pixel.

[0012] According to at least one of the various embodiments of the present disclosure, the processor can detect a saturated pixel area when it is determined that at least one pixel included in the light receiving unit is saturated from the reflected light.

[0013] According to at least one of the various embodiments of the present disclosure, the processor may obtain phase image data from the reflected light, and calculate intensity image data, and if the intensity is greater than a threshold, determine that the at least one pixel is saturated.

[0014] According to at least one of the various embodiments of the present disclosure, when the saturated pixel area is detected, the processor selects a first pixel among a plurality of pixels included in the light-receiving unit, wherein the first pixel may be a pixel that was not used for distance calculation in the detected saturated pixel area.

[0015] According to at least one of the various embodiments of the present disclosure, when the saturated pixel area is detected, the processor selects a first pixel among a plurality of pixels included in the light-receiving unit, wherein the first pixel may be selected based on predefined ratio information of the second pixel.

[0016] According to at least one of the various embodiments of the present disclosure, the second pixel may be a center pixel in the detected saturated pixel area.

[0017] According to at least one of the various embodiments of the present disclosure, the processor can correct and output a phase image of the reflected light based on the estimated delay phase value at the second pixel.

[0018] According to at least one of the various embodiments of the present disclosure, the processor may output a depth image from the reflected light, and correct the derived depth image based on a delay phase value at the estimated second pixel.

[0019] According to at least one of the various embodiments of the present disclosure, the light irradiated from the light emitting unit toward the object may include a point light source or a line light source.

[0020] A distance calculation method in a ToF sensor according to at least one of various embodiments of the present disclosure includes the steps of: irradiating light toward an object; receiving reflected light having a delay phase from the object; and calculating a distance to the object based on the reflected light; wherein the distance calculation step may include estimating a delay phase value in a second pixel based on a first pixel among a plurality of pixels receiving the reflected light.

[0021] According to at least one of the various embodiments of the present disclosure, the distance calculation step may further include a step of calculating a distance value for the object at the second pixel based on the estimated delay phase value at the second pixel.

[0022] According to at least one of the various embodiments of the present disclosure, the distance calculation step may further include a step of detecting a saturated pixel area when the at least one pixel is determined to be saturated by the reflected light.

[0023] According to at least one of the various embodiments of the present disclosure, the distance calculation step may obtain phase image data from the reflected light, and calculate intensity image data, and if the intensity is greater than a threshold, determine that at least one pixel is saturated.

[0024] According to at least one of the various embodiments of the present disclosure, the distance calculation step may include, when the saturated pixel area is detected, selecting a first pixel among a plurality of pixels included in the light-receiving unit, wherein the first pixel may be a pixel that was not used for distance calculation in the detected saturated pixel area.

[0025] According to at least one of the various embodiments of the present disclosure, the distance calculation step may, when the saturated pixel area is detected, select a first pixel among a plurality of pixels included in the light-receiving unit, wherein the first pixel may be selected based on predefined ratio information of the second pixel.

[0026] According to at least one of the various embodiments of the present disclosure, the second pixel may be a center pixel in the detected saturated pixel area.

[0027] According to at least one of the various embodiments of the present disclosure, the distance calculation step may further include a step of correcting and outputting a phase image of the reflected light based on the delay phase value at the estimated second pixel.

[0028] According to at least one of the various embodiments of the present disclosure, the distance calculation step may further include a step of calculating a depth image from the reflected light, and correcting and outputting the calculated depth image based on a delay phase value at the estimated second pixel.

[0029] According to at least one of the various embodiments of the present disclosure, the light irradiated from the light emitting unit toward the object may include a point light source or a line light source.

[0030] According to at least one of the various embodiments of the present disclosure, there is an effect of being able to detect saturated pixel(s) in an image sensor.

[0031] According to at least one of the various embodiments of the present disclosure, there is an effect that distance values ​​can be calculated more accurately even in saturated pixels.

[0032] According to at least one of the various embodiments of the present disclosure, there is an effect that the sensor can more accurately correct and output phase, depth, etc. even when at least one pixel is saturated.

[0033] The technical effects of the embodiments are not limited to those described in this article, but include those that can be understood through the description of the invention.

[0034] Figure 1 is a simplified diagram illustrating a sensor system and an object.

[0035] Figure 2 is a detailed configuration diagram of the sensor system of Figure 1.

[0036] FIG. 3 is a waveform diagram illustrating signal processing in the sensor system of FIGS. 1 and 2.

[0037] Figure 4 is a diagram illustrating the input of an appropriate level of light and the input of reflected light of a saturation level.

[0038] Figure 5 is a diagram illustrating a distance calculation method for saturated pixels.

[0039] Figure 6 is a diagram illustrating weight calculation.

[0040] Figure 7 is a block diagram of the configuration of a ToF sensor.

[0041] Figure 8 is a flowchart illustrating a distance calculation method in saturated pixels.

[0042] FIG. 9 is a flowchart illustrating a method for correcting phase information in a sensor when at least one pixel is saturated.

[0043] FIG. 10 is a flowchart illustrating a method for compensating depth information in a sensor when at least one pixel is saturated.

[0044] Hereinafter, an invention according to an embodiment for solving the above problem will be described in more detail with reference to the drawings.

[0045] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0046] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0047] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0048] In this application, it should be understood that terms such as “include,” “have,” or “comprising” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] In the present disclosure, it is intended to provide a method and device for detecting saturated pixels in a sensor, a method and device for calculating the distance of saturated pixels in a sensor, and a method and device for correcting and outputting at least one of phase and depth including saturated pixels in a sensor.

[0050] Here, the sensor according to the present disclosure is exemplified by a ToF (Time of Flight) sensor, but is not necessarily limited thereto.

[0051] Hereinafter, a sensor and a distance calculation method in the sensor according to the present disclosure will be described.

[0052] Figure 1 is a simplified diagram illustrating a sensor system (1) and an object (30).

[0053] Referring to Fig. 1, the sensor system (1) may be configured to include a light emitting unit (10) and a light receiving unit (20). At this time, the sensor system (1) may also be referred to as a ToF sensor system.

[0054] The light emitting unit (10) can be configured to include a light emitting sensor unit (11) and a light emitting lens (12).

[0055] At this time, the light-emitting sensor unit (11) may include multiple light-emitting sources. Each light-emitting source according to the present disclosure may be in the form of a point light source or a line light source. However, the present invention is not limited thereto.

[0056] The light emitting unit (10) can irradiate light from at least one light source included in the light emitting sensor unit (11) to an object (30) through a light emitting lens (12).

[0057] The light receiving unit (20) may be configured to include a light receiving sensor unit (21) and a light receiving lens (22). At this time, the light receiving unit (20) may be described as a ToF sensor.

[0058] The light receiving unit (20) can receive the light that is reflected from the object (30) by the light emitting unit (10) and delayed by the object (30) through the light receiving lens (22) at the light receiving sensor unit (21).

[0059] At this time, the light receiving sensor unit (21) may include a plurality of pixels that receive delayed light.

[0060] The configuration of this ToF sensor system (1) is described in more detail as follows.

[0061] Fig. 2 is a detailed configuration diagram of the sensor system (1) of Fig. 1.

[0062] Referring to FIG. 2, as described above, the sensor system (or ToF sensor system) (1) is composed of a light emitting unit (10) and a light receiving unit (20), and light irradiated from the light emitting unit (10) to an object (30) and reflected can be delayed through an environment such as air, i.e., a medium, and then received by the light receiving unit (20).

[0063] As described above, the light emitting unit (10) may be composed of a light emitting sensor unit (11) and a light emitting lens (12).

[0064] At this time, the light-emitting sensor unit (11) is configured to generate light to be irradiated to the object (30).

[0065] The light emitting sensor part (11) may represent, for example, the remaining part of the light emitting part (10) shown in Fig. 2, excluding the light emitting lens (12).

[0066] Referring to FIG. 2, the light emitting sensor unit (11) may include a phase signal generator (210), a light driver (220), and a light source (230).

[0067] The phase signal generator (210) can generate a signal with a predetermined phase and period and transmit it to the optical driver (220).

[0068] At this time, the phase signal generator (210) can generate signals corresponding to four phases, for example, 0 degrees, 90 degrees, 180 degrees, and 270 degrees. Accordingly, the phase signal generator (210) can generate signals corresponding to the four phases as signals corresponding to the phases of 0 degrees, 180 degrees, 90 degrees, and 270 degrees and sequentially transmit them to the optical driver (220).

[0069] In order to transmit four phase signals to the optical driver (220), the phase signal generator (210) may include a configuration such as a multiplexer (MUX) (not shown) or a switch (not shown). The multiplexer may be placed between the phase signal generator (210) and the optical driver (220).

[0070] The optical driver (220) can process (e.g., modulate) a phase signal sequentially generated and received by the phase signal generator (210) and transmit it to the light source (230).

[0071] At this time, the light driver (220) may be a driver configured according to the light source (230). That is, if the light source (230) is a laser light source, it may be a laser driver, and if the light source (230) is an LED light source, it may be an LED driver. However, the present invention is not limited thereto.

[0072] The light source (230) can irradiate light to the object (30) according to a signal processed by the light driver (220).

[0073] The light irradiated to the object (30) through the light emitting unit (10) is reflected through a medium such as air as shown in FIG. 2, and the reflected and delayed light can be received by the light receiving unit (20).

[0074] Therefore, the distance (d) between the ToF sensor (1) and the object (30) may be proportional to the phase (i.e., delayed phase).

[0075] The delayed phase signal can be transmitted to the light receiving sensor (21) through the light receiving lens (22).

[0076] Referring to FIG. 2, the light receiving sensor (21) can be configured to include a demodulation unit (240) and a pixel unit (250).

[0077] The demodulator (240) can demodulate a phase-delayed signal input through the light-receiving lens (22) based on a signal generated from the phase signal generator (210) of the light-emitting unit.

[0078] The pixel unit (250) can receive, convert, filter, and output a phase delay signal demodulated by the demodulator unit (240).

[0079] FIG. 3 is a waveform diagram illustrating signal processing in the sensor system (1) of FIGS. 1 and 2.

[0080] In (a) of Fig. 3, the signal emitted from the light emitting unit (10) (emitted signal: g(t)) and the reflected signal received with a phase delay (phase shift) (reflected signal: s(t)) are shown.

[0081] Referring to (a) of Fig. 3, it can be seen that the reflected signal (s(t)) is not only delayed in phase but also has lower power compared to the investigated signal (g(t)).

[0082] Figure 3 (b) shows the configuration of a depth frame.

[0083] Referring to (b) of FIG. 3, one depth frame may include signals of different phases generated from a phase signal generator (210).

[0084] At this time, based on the pixel timing of the horizontal axis, one depth frame may be composed of, for example, four sub-frames. However, this is not limited to this. For example, one depth frame may be composed of two or eight sub-frames.

[0085] Referring to (b) of Fig. 3, the first subframe may represent data with a phase of 0 degrees, i.e., raw phase, the second subframe may represent data with a phase of 180 degrees, the third subframe may represent data with a phase of 90 degrees, and the fourth subframe may represent data with a phase of 270 degrees. In this case, the phase of each subframe may be based on the raw phase.

[0086] In (b) of FIG. 3, each subframe may include a header part and a body part, the header part may include an identifier of each subframe, and the body part may include actual readable data of the corresponding subframe.

[0087] Meanwhile, a depth frame may sequentially have first to fourth sub-frames containing data for each phase, but this is not limited thereto.

[0088] Figure 3 (c) shows the illumination waveform and the reflection waveform.

[0089] Referring to (c) of Fig. 3, the delayed phase (φ) can be a value between the high portion of the investigated signal waveform and the high portion of the reflected signal waveform. This delayed phase can be calculated using the following mathematical expression 1.

[0090]

[0091] Q1 to Q4 can be calculated through measurements of C1 to C4 corresponding to the phase of each subframe in (c) of Fig. 3. In this case, C1 to C4 can represent waveforms whose phases are shifted based on the waveform of the investigated signal.

[0092] Q1 can be derived from the C1 waveform, which is low phase, i.e., has a phase of 0 degrees. Q1 can represent the overlapping portion of the reflected signal and the C1 waveform.

[0093] Alternatively, depending on the definition point of view, Q1 may represent the portion of the C1 waveform excluding the portion corresponding to the phase (Φ).

[0094] Q2 can be derived from the C2 waveform with a phase shift of 180 degrees. Q2 can represent the overlapping portion of the reflected signal and the C2 waveform.

[0095] Q3 can be derived from the C3 waveform, which is 90 degrees out of phase. Q3 can likewise represent the overlapping portion of the reflected signal and the C3 waveform.

[0096] Q4 can be derived from the C4 waveform, which has a phase shift of 270 degrees. Q4 can likewise represent the overlapping portion of the reflected signal and the C4 waveform.

[0097] As illustrated in (c) of Fig. 3, Q1 to Q4 may have different values. However, the values ​​of Q1 to Q4 may be at least less than half of one period of the investigated or reflected signal. In addition, the values ​​of each of Q1 to Q4 may be less than or equal to the delay phase (φ) value between the investigated signal and the reflected signal.

[0098] Meanwhile, the intensity (or amplitude) of the reflected signal can be calculated using the following mathematical expression 2.

[0099]

[0100] Additionally, the intensity of the reflected signal can be calculated using the following mathematical expression 3.

[0101]

[0102] Meanwhile, the distance (d: distance) from the reflected signal to the object (30) can be calculated using the following mathematical expression 4.

[0103]

[0104] In relation to a method and device for detecting saturated pixels in a sensor according to an embodiment of the present disclosure, saturated pixels are described.

[0105] Stray light is one cause of pixel saturation within a ToF sensor. This stray light can be caused by interference in distance calculations due to multiple paths within the ToF sensor.

[0106] In relation to this, FIG. 4 is a diagram illustrating the input of an appropriate level and the input of reflected light of a saturation level.

[0107] Figure 4 (a) shows a case where an appropriate level of reflected light is input to pixels, and a graph according to the intensity of reflected light in a normal state is shown below.

[0108] In (a) of Figure 4, if pixel 1 is referred to as the center pixel for convenience, it can be seen that the intensity of reflected light received from the pixels gradually weakens as one moves toward the surrounding pixels (2 to 5) based on the center point.

[0109] In addition, the graph of Fig. 4 (a) shows that the intensity of the reflected light at the center pixel is less than the maximum value (max_limit), so it can be called a non-saturated pixel (or normal pixel). Meanwhile, the intensity of the reflected light at the surrounding pixels on the left and right with respect to the center pixel may be symmetrical. The maximum value may be, for example, a reference value for determining saturation. However, the present invention is not limited thereto and may vary depending on the sensor system.

[0110] Meanwhile, Fig. 4 (b) shows a case where reflected light at a level other than the appropriate level, that is, a saturated level, is input to the pixels, and a graph according to the intensity of the saturated reflected light is shown below.

[0111] In Figure 4 (b), it can be seen that the first surrounding pixels also have values ​​greater than the maximum based on the center pixel and are saturated pixels.

[0112] Referring to (b) of Fig. 4, the second surrounding pixels based on the center pixel may be a boundary portion (410, 420) even if they are not the maximum. Here, the boundary portion (410, 420) may refer to a pixel corresponding to the boundary with pixels that receive reflected light but do not receive reflected light above a threshold value. This boundary portion (or boundary pixel) (410, 420) may, for example, receive only reflected light of the level of background brightness.

[0113] In the above, the reflected light input corresponding to the background brightness may act as noise. In these boundary portions (or boundary pixels) (410, 420), the reflected light input may not be sufficient to be used for distance calculation.

[0114] In other words, referring to (b) of Fig. 4, the center pixel and the first surrounding pixels are saturated pixels, and the second surrounding pixels are non-saturated pixels.

[0115] Meanwhile, the first surrounding pixels above may represent pixels positioned closer to the center pixel than the second surrounding pixels.

[0116] Figure 5 is a diagram illustrating a distance calculation method for saturated pixels.

[0117] S SL represents the center pixel among the saturated pixels, S1, S5, S6, S7, S 10 , S 11 The back represents the first surrounding pixels, and S x , S2, S3, S4, S8, S9, etc. can represent second surrounding pixels.

[0118] A typical stray light situation is an unsaturated S SL The affected part (S) a ) to remove S x We want to calculate the exact distance value.

[0119] As mentioned above, S SL In this unsaturated situation, S x may not receive the desired level of reflected light. S x is just a value of the level of background brightness (noise) (S b ) can be entered.

[0120] On the other hand, S SL In this saturation situation, that is, when the light received by the pixel is at a level where saturation occurs, S SL The impact on these adjacent pixels must be taken into account.

[0121] In other words, S SL S at a certain level of ratio x can affect. At this time, the value that affected is, for example, S a If named as S, this value is used in ToF distance calculation signal processing. x In S a It is desirable to remove the component.

[0122]

[0123]

[0124] In this situation, in the present disclosure, S in the above mathematical expressions 5 and 6 a Using the saturated position, i.e. the original signal S of the pixel SL can be estimated.

[0125] ω SL is S SL This S x It can be determined based on the information that has influenced the pixel. At this time, the information may include, for example, information on a predefined ratio of the pixel in question to the surrounding pixels. Such predefined ratios may include, for example, a PSF (Point Spread Function), a LUT (Look-Up Table), etc.

[0126] Sx S in position SL Based on information influenced by (e.g., PSF), ω SL S in proportion to SL You can have a signal.

[0127] Meanwhile, S x S in location b Since it has a level of light, it can be defined as in mathematical formula 7.

[0128]

[0129] As shown in the above mathematical formula 7, S b may be a negligible value. Therefore, S x In S b The value after subtracting is S a can be approximated as

[0130] Meanwhile, in Fig. 5 and above, S x can represent an area where no valid light enters from the existing pixels. S2, S3, S4, S8, S9, etc. are also S x In the same way, it can represent an area where no valid light comes in from an existing pixel.

[0131] In other words, S x is S b It should have a brightness level above the level (the minimum level that is not recognized as noise) and preferably not be saturated.

[0132] Also, S b can also be inferred based on Intensity and Exposure Time.

[0133] Meanwhile, as shown in mathematical expression 8 below, S b may be ignored if it is below a certain level.

[0134]

[0135] As in mathematical formula 8 and above, S bIt is desirable that S is a sufficiently small value. Therefore, S SL If saturation does not occur, S b It is desirable that the pixel location receives little reflected light.

[0136] Figure 6 is a diagram illustrating weight calculation.

[0137] The weights (W) can be calculated based on the selection of pixels of interest, the intensity and noise tendency of the selected pixels of interest, and the distance of the pixels of interest.

[0138] The weights calculated in this way can be used by averaging.

[0139] Referring to Fig. 6, a center pixel may be selected as a pixel of interest. Meanwhile, first pixels surrounding the center pixel may also be selected as pixels of interest.

[0140] The weight-based averaging method described above can be referenced in acquisition of 4-phase sample data, acquisition of distance data, etc. according to the present disclosure.

[0141] For example, ω' SL is the sum of the weights of all target pixels (or pixels of interest) (W sum ) can be obtained by dividing it into and performing normalization.

[0142] According to the embodiment, S x Pixel distance W d (S at saturated center pixel x Normalization can also be performed by weighting the distance to the pixel.

[0143] Therefore, S' SL can be defined as in mathematical formula 9 below.

[0144]

[0145] In the above mathematical expression 9, M may be a value indicating a pixel location that is ignored in the distance calculation from the general intensity to the noise level intensity. In the above, the noise level intensity may represent, for example, the background level.

[0146] ω SL Silver may be a value representing a weight obtained from a stray light model.

[0147] ω d is, S SL It can be a value representing a weight for the distance from .

[0148] In this disclosure, S' calculated by mathematical expression 9 SL Based on this, Re and Im signals can be generated. Then, the delay phase value (φ) is calculated according to Equation 10, and the measured distance D' is calculated based on the calculated delay phase value (φ) as in Equation 11. SL can be converted to

[0149]

[0150]

[0151] Referring to mathematical expressions 9 to 11, S having a brightness above a certain level x Distance calculation is performed at the location, and the distance value is calculated from Dx to D' SL The value can be estimated.

[0152] Meanwhile, S with brightness above a certain level x If the number of target pixels at the location is less than or equal to the threshold, S in Fig. 5 x , S2, S3, S4, S8, S9, etc., the distance value can also be obtained from non-saturated pixels using the weight sum method. In the above, if the number of pixels is a threshold, it can represent, for example, 2 or less.

[0153] In other words, the distance value obtained from the saturated pixel may be a distorted value, so when calculating the measured distance to the object (30) including the distance value of the saturated pixel, an accurate distance value cannot be obtained.

[0154] Accordingly, in the present disclosure, in a situation where saturation does not occur, based on a value at a location of a pixel that was not used for distance calculation, the location of a saturated pixel is detected by referring to a pixel at that location in a situation where saturation occurs, and a distance value from the saturated pixel can be calculated from a delay phase value.

[0155] Since the distance value of the saturated pixel calculated in this way is not a distorted value in the above-described situation, a more accurate distance value can be provided when calculating the measured distance value to the object (30).

[0156] Meanwhile, even in saturated conditions, the amplitude may be below a certain level. Furthermore, even when the signal level of the received signal is low, the amplitude may be below a certain level.

[0157] Fig. 7 is a block diagram of the configuration of a ToF sensor (20).

[0158] Referring to FIG. 7, the ToF sensor (20) may include a data input unit (710), a saturation detection unit (720), a first operation unit (730), a weight generation unit (740), a data generation unit (750), a second operation unit (760), a multiplexer (MUX) (770), a conversion unit (780), etc.

[0159] The data input unit (710) can receive reflected light having a delayed phase, i.e., a reflected signal, irradiated to an object (30) through the light emitting unit (10).

[0160] The reflected signal received in this manner can be input to at least one of the saturation detection unit (720), the first operation unit (730), the weight generation unit (740), and the data generation unit (750).

[0161] At this time, the input can be input in frame units. In the above, the frame unit can be any one of a phase unit frame, a depth frame, etc. In the former, the phase unit frame can refer to a sub-frame unit corresponding to each phase unit described above, i.e., 0 degrees, 180 degrees, 90 degrees, and 270 degrees.

[0162] The saturation detection unit (720) can detect whether a pixel is saturated. At this time, if a brightness higher than a threshold is detected, it can be estimated that at least one pixel is saturated.

[0163] The saturation detection unit (720) can detect a saturated pixel when at least one pixel is estimated to be saturated.

[0164] The first operation unit (730) can calculate the amplitude and phase value of a signal reflected from a signal input from the data input unit (710).

[0165] The weight generation unit (740) can generate a weight for at least one selected pixel of interest.

[0166] The data generation unit (750) can obtain sampling data. In particular, the data generation unit (750) can obtain sampling data from non-saturated pixels.

[0167] The second operation unit (760) can calculate the amplitude and phase value of a signal based on the sampling data obtained from the data generation unit (750) from the signal input from the data input unit (710).

[0168] Information about whether a saturated pixel is detected in the saturation detection unit (720), information about the detected saturated pixel, etc. can be transmitted to the multiplexer (MUX) (770).

[0169] The amplitude value and delay phase value calculated based on the sampling data received from the first operation unit (730) can be transmitted to a multiplexer (MUX) (770).

[0170] The amplitude value and delay phase value calculated based on the sampling data received from the data generation unit (750) in the second operation unit (760) can be transmitted to a multiplexer (MUX) (770).

[0171] The multiplexer (MUX) (770) can receive a value calculated in the first operation unit (730) and a value calculated in the second operation unit (760) and output a specific value to the conversion unit (480). At this time, the multiplexer (MUX) (770) can output either the value calculated in the first operation unit (730) or the value calculated in the second operation unit (760) as a specific value. At this time, the output can be an output value in pixel units.

[0172] Meanwhile, the multiplexer (MUX) (770) can receive the output of the saturation detection unit (720) as input. When outputting the specific value described above, the multiplexer (MUX) (770) can refer to the output value of the saturation detection unit (720) that is input.

[0173] For example, in a case where a pixel-by-pixel operation value is output from a multiplexer (MUX) (770) to a conversion unit (780), if a value indicating that the pixel is a saturated pixel is input from a saturation detection unit (720), a value calculated from a second operation unit (760), rather than a value calculated from a first operation unit (730), for the pixel can be output to the conversion unit (780). This operation can be performed on a pixel-by-pixel basis or a pixel area basis.

[0174] In the above, the pixel area unit may refer to an area of ​​a group unit including pixels with a saturation probability greater than a threshold value located around a pixel where saturation has occurred. In this case, when defining the pixel area unit, the PSF ratio may be referred to, for example.

[0175] The conversion unit (780) can convert the output value of the multiplexer (MUX) (770) into a distance value with respect to the object (30).

[0176] The configurations of the ToF sensor (20) of FIG. 7 described above can be implemented in the form of a single chip. Depending on the embodiment, the configurations of the ToF sensor (20) of FIG. 7 can be implemented as multiple chips.

[0177] Meanwhile, among the configurations of the ToF sensor (20) illustrated in FIG. 7, the configurations excluding the data input unit (410), multiplexer (MUX) (770), and conversion unit (480) can be implemented in the form of a single chip, and for convenience, can be described by being named a processor.

[0178] Depending on the embodiment, all of the configurations of the ToF sensor (20) shown in FIG. 7 except for the data input unit (410) may be implemented in the form of a single chip, and may be the processor described above.

[0179] Meanwhile, the processor may further include one or more configurations in addition to the configuration of FIG. 7 described above.

[0180] These processors can perform a controller function for the overall operation of the ToF sensor system (1).

[0181] Meanwhile, although not shown, the processor may exist separately as a controller that controls the operation of the components of the ToF sensor (20) of FIG. 7.

[0182] In some embodiments, when the ToF sensor system is attached or mounted on another device such as a display device, the operation control of the components of the ToF sensor (20) of FIG. 7 may be performed by a component (e.g., a timing controller (Tcon)) that performs a controller function of the device.

[0183] As described above, the present disclosure discloses a distance calculation method in a pixel where saturation occurs in a ToF sensor (20), i.e., a saturated pixel.

[0184] In particular, in the present disclosure, in the light-emitting unit (10) and the light-receiving unit (20) in the form of a point light source or a strip light source, the light-receiving unit (20) in particular may have a function of performing a demodulation method for the amplitude modulation method of the light-emitting unit (10).

[0185] The light receiving unit (20) can determine the amplitude, delayed phase, and intensity according to the input N (where N is a natural number) samplings. Meanwhile, N may be 3 or more.

[0186] The light receiving unit (20) can convert the delayed phase into a distance value.

[0187] At this time, the light receiving unit (20) can determine whether the target pixel is saturated based on the pre-calculated amplitude and brightness information. That is, the light receiving unit (20) can detect a saturated pixel.

[0188] Since the light receiving unit (20) can have information about the PSF ratio described above, for example, it can recollect N sampling information by referring to the PSF ratio from the surrounding non-saturated pixel(s) and estimate the phase delay value for the non-operational pixel, i.e., the saturated pixel.

[0189] And the light receiving unit (20) can convert the estimated phase delay value into a distance value.

[0190] Figure 8 is a flowchart illustrating a distance calculation method in saturated pixels.

[0191] For convenience, Fig. 8 is described from the perspective of a processor within the ToF sensor system (1). At this time, the processor may be a controller that controls the overall operation of the ToF sensor system (1).

[0192] In operation S110, the processor can irradiate modulated light to an object (30) through a light emitting unit (10).

[0193] In operation S120, the processor can receive the investigated modulated light reflected from the object (30) and the delayed light at the light receiving unit (20).

[0194] In operation S130, the processor can select the first pixel.

[0195] Here, the first pixel may include a pixel that was not used in the distance calculation.

[0196] At this time, a pixel that was not used for distance calculation refers to a pixel that was not used for distance calculation in a normal situation, i.e., when the pixel was not saturated. That is, when a certain pixel is a saturated pixel (or a central saturated pixel), when saturation did not occur in the pixel, the pixel that was not used for distance calculation in the pixel may be the first pixel.

[0197] Meanwhile, the processor may select the first pixel, for example, based on PSF ratio information, when a second pixel, i.e., a saturated pixel, is detected.

[0198] In operation S140, the processor can estimate a delay phase value of a second pixel based on the selected first pixel.

[0199] Meanwhile, the second pixel may be one of the pixels in the pixel area where saturation is estimated to have occurred due to the received delayed reflected light signal.

[0200] Additionally, the second pixel may represent the pixel where saturation occurred or the center pixel where saturation occurred.

[0201] In operation S150, the processor can calculate a distance value at the second pixel using a delay phase value of the second pixel estimated based on the first pixel.

[0202] At least one of the first pixel and the second pixel described above may be plural.

[0203] Meanwhile, the first pixel and the second pixel may be automatically selected by the processor or may refer to the detection result of the saturation detection unit (720).

[0204] In some embodiments, the processor may compute the distance value at the same saturated pixel multiple times.

[0205] In this case, if the difference between the distance values ​​in saturated pixels calculated multiple times is greater than a preset threshold, the calculated distance value can be judged to have low reliability.

[0206] In this case, the processor can be controlled to re-execute the entire process or part of the process.

[0207] Meanwhile, in the above, if the difference in the distance values ​​in the saturated pixels calculated multiple times is less than a preset threshold value, the calculated distance value can be judged to have high reliability.

[0208] In this case, the processor may select one of the maximum, median, or minimum values ​​among the calculated distance values ​​and use it as the distance value of the corresponding saturated pixel. Alternatively, in the above case, the processor may use the average value.

[0209] In some embodiments, the processor may compute distance values ​​from a central saturated pixel and at least one of the surrounding first saturated pixels, rather than computing distance values ​​from the same saturated pixel multiple times.

[0210] In this case, if the difference between the distance values ​​calculated from the central saturated pixel and the first saturated pixel is greater than a preset threshold, the distance value calculated from the central saturated pixel may be judged to have low reliability. Accordingly, the processor may be controlled to re-perform the entire process or part of the process.

[0211] Meanwhile, in the above, if the difference between the distance value calculated from the central saturated pixel and the first saturated pixel is less than a preset threshold, the processor can determine that the calculated distance value from the central saturated pixel has high reliability.

[0212] And in this case, the processor can use the calculated distance value of the central saturated pixel as the distance value of the corresponding saturated pixel.

[0213] Alternatively, in the above case, the processor may use the average value of the operation distance values ​​of the central saturated pixel and the surrounding first saturated pixels.

[0214] In the above, if a result value with low reliability is derived, the processor may re-perform the entire process or part of the process repeatedly as described above.

[0215] Alternatively, the processor may re-perform the aforementioned operations after newly selecting at least one of the first or second pixels in the above case.

[0216] The processor may include an artificial intelligence learning engine, and may learn using the aforementioned saturated pixel and distance operation values ​​as input or training datasets.

[0217] The results learned in this way can be used as a reference for judging the reliability of distance operation values ​​in saturated pixels.

[0218] Additionally, if there is feedback on the distance operation value at saturated pixels, it can be used to update the artificial intelligence learning engine (or model).

[0219] The artificial intelligence learning engine can be used to select the first pixel and the second pixel in the present disclosure.

[0220] Additionally, the artificial intelligence learning engine can be used to detect a specific amplitude abnormality after the century image is produced in FIGS. 9 and 10 described below, that is, to determine whether pixel saturation has occurred.

[0221] The processor can measure or calculate the distance to the object (30) using or including the distance calculation value from the second pixel.

[0222] Both Figures 9 and 10 below assume that the ToF sensor system (1) is in operation.

[0223] FIG. 9 is a flowchart illustrating a method for correcting phase information in a sensor when at least one pixel is saturated.

[0224] In operation S210, the processor can acquire a delayed phase image.

[0225] In operation S220, the processor can derive the intensity from the acquired delayed phase image.

[0226] In the S230 operation, the processor can determine whether an image above a certain brightness level is detected.

[0227] At this time, the processor can determine whether an image above a certain brightness is detected from the brightness image data generated based on the acquired phase image data. Here, the certain brightness may refer to, for example, the intensity of the received light at which a saturated pixel can be detected.

[0228] If the processor has training data through an artificial intelligence learning model, when an image above a certain brightness is produced, it can be detected that the above-mentioned brightness has occurred, i.e., saturation has occurred.

[0229] In operation S240, the processor can detect a saturated pixel area if an image having a brightness higher than a specific level is detected as a result of the judgment in operation S230 (i.e., if at least one pixel is estimated to be saturated).

[0230] In operation S250, the processor can estimate and calculate the value of a saturated pixel within a detected saturated pixel area.

[0231] In the S260 operation, the processor can compensate and output a delayed phase image.

[0232] FIG. 10 is a flowchart illustrating a method for compensating depth information in a sensor when at least one pixel is saturated.

[0233] The S310 operation is identical to the S210 operation of FIG. 9 described above.

[0234] In operation S320, the processor can produce amplitude, depth and brightness images based on the acquired delayed phase images.

[0235] In S330 operation, the processor can determine if a certain brightness abnormality is detected.

[0236] In operation S340, the processor can detect a saturated pixel area if an image having a brightness higher than a certain level is detected as a result of the S330 operation judgment (i.e., if at least one pixel is estimated to be saturated).

[0237] In the S350 operation, the processor can estimate and calculate the value of a saturated pixel within a detected saturated pixel area.

[0238] In S360 operation, the processor can correct and output depth images.

[0239] The distance calculation method according to the present disclosure described above is applicable not only to ToF sensors but also to various fields. For example, the present disclosure is applicable when pixel saturation occurs in sensors or ToF sensors used in various fields such as automobiles, robots, industrial automation, eXtended Reality, and environmental monitoring.

[0240] Although the present invention has been described above with reference to embodiments thereof, it will be readily understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0241] The present disclosure relates to a ToF sensor and a distance calculation method thereof, which can measure or calculate the distance to an object more accurately than in the past even in pixels where saturation has occurred, and thus can be applied to various systems including a ToF sensor, and thus has industrial applicability.

Claims

1. A light emitting part that irradiates light toward an object; A light receiving unit that receives reflected light with a delayed phase from the object; and A processor that calculates the distance to the object based on the reflected light; The above processor, Estimating a delay phase value in a second pixel based on a first pixel among a plurality of pixels included in the light receiving unit, ToF (Time of Flight) sensor.

2. In claim 1, The above processor, Based on the delay phase value at the estimated second pixel, a distance value for the object at the second pixel is calculated. ToF sensor.

3. In claim 2, The above processor, When at least one pixel included in the light receiving unit is judged to be saturated from the reflected light, detecting a saturated pixel area, ToF sensor.

4. In claim 3, The above processor, Obtaining phase image data from the above reflected light, calculating intensity image data, and determining that at least one pixel is saturated if the brightness is greater than a threshold. ToF sensor.

5. In claim 4, The above processor, When the above saturated pixel area is detected, a first pixel is selected from among a plurality of pixels included in the light receiving unit, The above first pixel is a pixel that was not used for distance calculation in the detected saturated pixel area. ToF sensor.

6. In claim 4, The above processor, When the above saturated pixel area is detected, a first pixel is selected from among a plurality of pixels included in the light receiving unit, The first pixel is selected based on the predefined ratio information of the second pixel, ToF sensor.

7. In claim 4, The second pixel above, The center pixel in the above-detected saturated pixel area, ToF sensor.

8. In claim 7, The above processor, Correcting and outputting the phase image of the reflected light based on the delay phase value in the estimated second pixel. ToF sensor.

9. In claim 7, The above processor, A depth image is derived from the reflected light, and the derived depth image is corrected and output based on the delay phase value at the estimated second pixel. ToF sensor.

10. In claim 1, The light irradiated from the above light emitting part toward the object is: Including point or line lights, ToF sensor.

11. Step of shining light toward the object; A step of receiving reflected light having a delayed phase from the object; and A step of calculating the distance to the object based on the reflected light; including: The above distance calculation step is, A step of estimating a delay phase value in a second pixel based on a first pixel among a plurality of pixels receiving the reflected light, How to calculate distance in a ToF sensor.

12. In claim 11, The above distance calculation step is, Further comprising a step of calculating a distance value for the object at the second pixel based on the delay phase value at the estimated second pixel. How to calculate distance in a ToF sensor.

13. In claim 12, The above distance calculation step is, If it is determined that at least one pixel is saturated by the reflected light, further comprising a step of detecting a saturated pixel area. How to calculate distance in a ToF sensor.

14. In claim 13, The above distance calculation step is, Obtaining phase image data from the above reflected light, calculating intensity image data, and determining that at least one pixel is saturated if the brightness is greater than a threshold. How to calculate distance in a ToF sensor.

15. In claim 14, The above distance calculation step is, When the above saturated pixel area is detected, a first pixel is selected from among a plurality of pixels included in the light receiving unit, The above first pixel is a pixel that was not used for distance calculation in the detected saturated pixel area. How to calculate distance in a ToF sensor.

16. In claim 14, The above distance calculation step is, When the above saturated pixel area is detected, a first pixel is selected from among a plurality of pixels included in the light receiving unit, The first pixel is selected based on the predefined ratio information of the second pixel, How to calculate distance in a ToF sensor.

17. In claim 14, The second pixel above, The center pixel in the above-detected saturated pixel area, How to calculate distance in a ToF sensor.

18. In claim 17, The above distance calculation step is, Further comprising a step of correcting and outputting a phase image of the reflected light based on the delay phase value in the estimated second pixel. How to calculate distance in a ToF sensor.

19. In claim 17, The above distance calculation step is, A depth image is derived from the reflected light, and further comprising a step of correcting and outputting the derived depth image based on the delay phase value in the estimated second pixel. How to calculate distance in a ToF sensor.

20. In claim 11, The light irradiated from the above light emitting part toward the object is: Including point or line lights, How to calculate distance in a ToF sensor.

Images