Image sensing device, image processing device, and imaging device including them

Abstract

To enhance accuracy of distance measurement for an imaging sensing device.SOLUTION: An image sensing device 100 according to one embodiment of the present invention may include: a phase difference calculator 210 that calculates a phase difference PDIF between an optical modulated signal MLS and reflection light RL on the basis of a plurality of capture data S0, S90, S180 and S270 generated by a plurality of modulated control signals MCS1 and MCS2, each having the optical modulate signal MLS and a predetermined modulated phase difference; a phase difference corrector 220 that uses the plurality of capture data S0, S90, S180 and S270 to calculate a contrast as a ratio of an intensity component to an amplitude component for the reflection light, and determines an aliasing value R corresponding to the contrast; and a distance calculator 230 that uses a corrected phase difference PDIF' obtained by correcting the phase difference PDIF according to the aliasing value R to calculate a distance from a target object 1.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image sensing device capable of sensing the distance to a target object, an image processing device, and an imaging device including the image sensing device and the image processing device. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of light-sensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical micro cameras.

[0003] In recent years, image sensing devices have been actively used not only to obtain color images but also to sense the distance to an object. In particular, the ToF (time of flight) method, which directly or indirectly measures the time it takes for irradiated light to be reflected from the object and reach it, is widely used. Summary of the Invention [Problem to be solved by the invention]

[0004] The technical idea of ​​the present invention can provide an image sensing device, an image processing device, and an imaging device including the image sensing device and the image processing device, which can improve the accuracy of distance measurement.

[0005] The technical problems of the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0006] An image processing device according to one embodiment of the present invention disclosed in this document may include a phase difference calculator that calculates a phase difference between an optical modulation signal and a reflected light based on a plurality of capture data generated by a plurality of modulation control signals each having a predetermined modulation phase difference with respect to the optical modulation signal, a phase difference corrector that calculates a contrast, which is a ratio between an amplitude component and an intensity component of the reflected light, using the plurality of capture data and determines an aliasing value corresponding to the contrast, and a distance calculator that calculates a distance to a target object using a corrected phase difference obtained by correcting the phase difference according to the aliasing value.

[0007] An imaging device according to one embodiment of the present invention may include an image sensing device that generates a plurality of capture data using a plurality of modulation control signals, each of which has a predetermined modulation phase difference with an optical modulation signal, and an image processing device that calculates a contrast, which is a ratio between an amplitude component and an intensity component of reflected light, using the plurality of capture data, and calculates a distance to a target object using a corrected phase difference obtained by correcting a phase difference between the optical modulation signal and the reflected light according to an aliasing value corresponding to the contrast.

[0008] An image sensing device according to one embodiment of the present invention includes a pixel that senses reflected light using a plurality of modulation control signals, each of which has a predetermined modulation phase difference with an optical modulation signal, to generate a plurality of pixel signals, and a readout circuit that processes the pixel signals to generate a plurality of capture data. An exposure period in which the pixel senses the reflected light may include a first period in which the plurality of modulation control signals maintain a logic high level, and a second period in which the plurality of modulation control signals alternately have the logic high level. Effect of the Invention

[0009] According to the embodiment disclosed in this document, the phase difference calculated by the ToF method is corrected, so that the distance to the target object can be obtained more accurately. In addition, various other benefits may be provided by this document, either directly or indirectly. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an imaging device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a circuit diagram illustrating an embodiment of a pixel included in the pixel array of FIG. 1. [Diagram 3] FIG. 2 is a simplified diagram illustrating the configuration of a depth frame for acquiring depth data. [Figure 4a] FIG. 2 is a diagram showing an optical modulation signal and a modulation control signal in a first microframe. [Figure 4b] FIG. 13 is a diagram showing an optical modulation signal and a modulation control signal in a second microframe. [Figure 4c] FIG. 13 is a diagram showing an optical modulation signal and a modulation control signal in a third microframe. [Figure 4d] FIG. 13 is a diagram showing an optical modulation signal and a modulation control signal in a fourth microframe. [Diagram 5] 1 is a graph showing the relationship between the distance to a target object and the contrast of reflected light. [Figure 6] 1 is a graph showing the relationship between the phase difference and the contrast of reflected light. [Figure 7] 1A and 1B are diagrams for explaining an embodiment capable of removing a component due to ambient light from the intensity of reflected light. [Figure 8a] 13A and 13B are diagrams for explaining another embodiment capable of removing a component due to ambient light from the intensity of reflected light. [Figure 8b] 13A and 13B are diagrams for explaining another embodiment capable of removing a component due to ambient light from the intensity of reflected light. [Figure 8c] 13A and 13B are diagrams for explaining another embodiment capable of removing a component due to ambient light from the intensity of reflected light. [Figure 8d] 13A and 13B are diagrams for explaining another embodiment capable of removing a component due to ambient light from the intensity of reflected light. [Figure 9] 13A and 13B are diagrams for explaining another embodiment capable of removing a component due to ambient light from the intensity of reflected light. [Figure 10] FIG. 2 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments, and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure can provide various effects that are directly or indirectly recognizable by the present disclosure.

[0012] FIG. 1 is a block diagram illustrating an imaging device according to one embodiment of the present disclosure. Referring to FIG. 1, an imaging device (ID) may refer to a device such as a digital still camera for taking still images or a digital video camera for taking videos. For example, the imaging device (ID) may be realized by, but is not limited to, a digital single lens reflex (DSLR), a mirrorless camera, or a smartphone, and the imaging device (ID) may be a concept including a device that includes an image sensor and can capture an object and generate an image. According to an embodiment, the imaging device (ID) may be a Lidar sensor.

[0013] The imaging device (ID) may include an image sensing device 100 and an image processing device 200 .

[0014] The image sensing device 100 may be a complementary metal oxide semiconductor image sensor (CIS) that converts incident light into an electrical signal. The image sensing device 100 may include a light source 10, a lens module 20, a light source driver 30, a pixel array 110, a sensor driver 120, a readout circuit 130, and a timing controller 140.

[0015] The light source 10 can irradiate the target object 1 with emitted light (EL) in response to a modulation light signal (MLS) applied from the light source driver 30. The light source 10 may be a monochromatic illumination source in which a laser diode (LD), a light emitting diode (LED), a near infrared laser (NIR), a point light source, a white lamp, and a monochromator are combined, or a combination of other laser light sources, which emit light in a specific wavelength band (e.g., near infrared, infrared, or visible light). For example, the light source 10 may be light in an infrared wavelength band having a wavelength of 800 nm to 1000 nm (hereinafter referred to as "infrared light"), and in the present disclosure, the description will be given on the assumption that the light source 10 irradiates infrared light. On the other hand, the light irradiated from the light source 10 may be pulsed light having a predetermined frequency, period, amplitude, and pulse width. Although only one light source 10 is shown in FIG. 1 for ease of explanation, multiple light sources may be arranged around the lens module 20 .

[0016] The lens module 20 can collect reflected light (RL) from the target object 1 and focus it onto the pixels (PX) of the pixel array 110. For example, the lens module 20 can include a focusing lens or other cylindrical optical element with a glass or plastic surface. The lens module 20 can include multiple lenses aligned around an optical axis.

[0017] The light source driver 30 can generate a light modulation signal (MLS) for driving the light source 10 according to the control of the timing controller 140, and in particular can control the waveform (frequency, period, amplitude, pulse width, etc.) of the illumination light (EL) output from the light source 10.

[0018] The pixel array 110 may include a plurality of pixels (PX) continuously arranged in a two-dimensional matrix structure (e.g., continuously arranged in a column direction and / or a row direction). Each of the plurality of pixels (PX) may sense incident light incident through the lens module 20 and generate a pixel signal according to the control of the sensor driver 120.

[0019] Each pixel (PX) may be an infrared pixel that generates a pixel signal by sensing incident light including reflected light (RL) that is incident by reflecting illumination light (EL) emitted from the light source 10 from the target object 1. In the present disclosure, the infrared pixel may be a depth pixel for calculating the distance to the target object 1, and in another embodiment, the infrared pixel may include a pixel for generating an infrared image by sensing infrared light incident from a scene instead of reflected light. In another embodiment, the pixel (PX) may include a pixel for generating a color image by sensing visible light incident from a scene. Hereinafter, it is assumed that each pixel (PX) is a 2-TAP pixel for detecting the distance to the target object 1 by an indirect ToF method. A more detailed structure and operation of each unit pixel (PX) will be described later with reference to FIG. 2.

[0020] The sensor driver 120 may drive pixels (PX) of the pixel array 110 in response to a timing signal output from the timing controller 140. For example, the sensor driver 120 may generate a control signal capable of selecting and controlling pixels (PX) included in at least one row line among a plurality of row lines of the pixel array 110.

[0021] The readout circuit 130 processes pixel signals output from the pixel array 110 under the control of the timing controller 140, and generates and stores image data (IDATA) for detecting the distance to the target object 1. The image data (IDATA) may be digital data obtained by analog-digital conversion of an analog pixel signal. To this end, the readout circuit 130 may include a correlated double sampler (CDS) for performing correlated double sampling on the pixel signals output from the pixel array 110. The readout circuit 130 may also include an analog-to-digital converter for converting an output signal from the correlated double sampler into a digital signal. The readout circuit 130 may also include a buffer circuit for temporarily storing pixel data output from the analog-to-digital converter and outputting the pixel data to the outside under the control of the timing controller 140. Meanwhile, two column lines for transmitting pixel signals are provided for each column of the pixel array 110, and a configuration for processing pixel signals output from each column line may also be provided corresponding to each column line.

[0022] The timing controller 140 may generate timing signals for controlling the operations of the light source driver 30, the sensor driver 120, and the readout circuit 130. According to an embodiment, the timing controller 140 may generate timing signals according to a predetermined setting value and / or a request of the image processing device 200. According to an embodiment, the timing controller 140 may include a logic control circuit, a phase lock loop (PLL) circuit, a timing control circuit, a communication interface circuit, and the like.

[0023] The image processing device 200 may generate processed image data by performing image signal processing on the image data (IDATA) received from the image sensing device 100. The image processing device 200 may perform image signal processing for improving image quality, such as reducing noise on the image data (IDATA) and performing interpolation and lens distortion correction on the image data (IDATA).

[0024] The image processing device 200 may include a phase difference calculator 210, a phase difference corrector 220, and a distance calculator 230. The image processing device 200 may be implemented in hardware, software, or a combination thereof.

[0025] The phase difference calculator 210 can calculate the phase difference between the illuminating light (EL) and the reflected light (RL) based on the image data (IDATA). There is a time delay between the illuminating light (EL) and the reflected light (RL) due to the distance between the image sensing device 100 and the target object 1, and this time delay appears as a phase difference between the illuminating light (EL) and the reflected light (RL). In the present disclosure, it is assumed that there is no time delay between the light modulation signal (MLS) and the illuminating light (EL). That is, the phase difference between the illuminating light (EL) and the reflected light (RL) can be the same as the phase difference between the light modulation signal (MLS) and the reflected light (RL).

[0026] The phase difference corrector 220 may correct the phase difference using the image data (IDATA). The illuminating light (EL) may have periodicity by being modulated at a certain frequency, and due to this periodicity, image data sensing the target object 1 located at different distances from the image sensing device 100 may show the same phase difference. The phase difference corrector 220 may generate a corrected phase difference by correcting the phase difference so that the phase difference may indicate the distance to the target object 1.

[0027] In addition, the phase difference corrector 220 can correct the phase difference by removing the ambient light component as much as possible in order to prevent the phase difference correction effect from being halved due to the ambient light component.

[0028] The distance calculator 230 can use the corrected phase difference to calculate the distance to the target object 1. The distance to the target object 1 can be calculated in units of each pixel of the pixel array 110.

[0029] In the following description, a process in which the image processing device 200 calculates the distance to the target object 1 based on one pixel will be described, but a substantially similar method can be applied to other pixels in the pixel array 110.

[0030] FIG. 2 is a circuit diagram illustrating an embodiment of a pixel included in the pixel array of FIG. Referring to FIG. 2, a pixel (PX) may correspond to an embodiment of each pixel (PX) included in the pixel array 110.

[0031] The pixel (PX) may include a photoelectric conversion element (PD), first and second transfer transistors (TX1, TX2), first and second reset transistors (RX1, RX2), first and second drive transistors (DX1, DX2), and first and second selection transistors (SX1, SX2).

[0032] The pixel (PX) may be a 2-TAP pixel including a first tap driven based on a first modulation control signal (MCS1) having a predetermined modulation phase difference with the light modulation signal (MLS), and a second tap driven based on a second modulation control signal (MCS2) having an opposite phase to the first modulation control signal (MCS1). The first tap may include a first transfer transistor (TX1), a first reset transistor (RX1), a first drive transistor (DX1), and a first selection transistor (SX1). The second tap may include a second transfer transistor (TX2), a second reset transistor (RX2), a second drive transistor (DX2), and a second selection transistor (SX2). In the present disclosure, the modulation phase difference may mean the phase difference between the light modulation signal (MLS) and the modulation control signals (MCS1, MCS2), and the phase difference may mean the phase difference between the light modulation signal (MLS) and the reflected light (RL).

[0033] Each of the modulation control signals (MCS1, MCS2) can have a predetermined modulation phase difference (for example, any one of 0 degrees, 90 degrees, 180 degrees, or 270 degrees) with the optical modulation signal (MLS).

[0034] The photoelectric conversion element (PD) can convert incident light into an electric signal and generate and accumulate a photoelectric charge corresponding to the intensity of the incident light. The photoelectric conversion element (PD) can be realized by at least one of a photodiode, a pinned photodiode, and a phototransistor, but the scope of the present invention is not limited thereto.

[0035] First, the structure and operation of the first tap will be described. The first transfer transistor (TX1) is connected between the photoelectric conversion element (PD) and a first floating diffusion region (FD1), and can be turned on or off in response to a first modulation control signal (MCS1). The turned-on first transfer transistor (TX1) can transfer photocharges accumulated in the photoelectric conversion element (PD) to the first floating diffusion region (FD1). The first floating diffusion region (FD1) can have a capacitance sufficient to accumulate photocharges generated in the photoelectric conversion element (PD), and can be realized as a junction capacitor, but the scope of the present invention is not limited thereto.

[0036] The first reset transistor RX1 is connected between a power supply voltage VDD and the first floating diffusion region FD1, and can be turned on or off in response to a first reset control signal RG1. The turned-on first reset transistor RX1 can drain photocharges accumulated in the first floating diffusion region FD1 by resetting the first floating diffusion region FD1 to the power supply voltage VDD.

[0037] The first drive transistor DX1 is connected between a power supply voltage VDD and the first selection transistor SX1, and is capable of generating an electrical signal corresponding to a voltage of the first floating diffusion region FD1.

[0038] The first selection transistor SX1 is connected between the first drive transistor DX1 and the first column line COL1 and can be turned on or off in response to a first selection control signal SEL1. The turned-on first selection transistor SX1 can output an electrical signal output from the first drive transistor DX1 through the first column line COL1. The control signals (MCS1, RG1, SEL1) for controlling the first tap may be provided from the sensor driver 120 of FIG.

[0039] The structure and operation of the first tap have been described above, but the structure and operation of the second tap are substantially similar, so a duplicated description will be omitted. However, the waveforms of the control signals (MCS2, RG2, SEL2) for controlling the second tap may differ from the waveforms of the control signals (MCS1, RG1, SEL1) for controlling the first tap. In particular, the second modulation control signal (MCS2) may have a phase opposite to that of the first modulation control signal (MCS1).

[0040] FIG. 3 is a diagram showing a simplified configuration of a depth frame for acquiring depth data. 3, the image sensing device 100 may operate in units of a depth frame to obtain depth data indicating a distance to a target object 1. That is, the image sensing device 100 may obtain the depth data after performing an operation corresponding to the depth frame.

[0041] The depth frame may include first to fourth subframes (SF1 to SF4). The first modulation control signal (MCS1) in each of the first to fourth subframes (SF1 to SF4) may have different phases from each other. In addition, the second modulation control signal (MCS2) having a phase opposite to that of the first modulation control signal (MCS1) may also have a different phase from each other in each of the first to fourth subframes (SF1 to SF4).

[0042] Each of the first to fourth subframes (SF1 to SF4) can include an exposure period and a readout period.

[0043] The exposure section may be a section in which the pixel (PX) senses the reflected light (RL) that is incident on the target object 1 and reflected from the illumination light (EL) output from the image sensing device 100 in response to the light modulation signal (MLS), and accumulates the generated photocharge inside the pixel (PX) (e.g., FD1, FD2).

[0044] The exposure section may include at least one microframe, which is a unit constituting the exposure section, and will be described in more detail below with reference to Figures 4a to 4d.

[0045] The exposure section may include multiple microframes to ensure sufficient light, and the number of microframes included in one exposure section may be a value determined experimentally in advance. In another embodiment, the number of microframes included in one exposure section may be variable depending on the illumination conditions.

[0046] The exposure section of the first subframe (SF1) may include a plurality of consecutive first microframes (MF1), the exposure section of the second subframe (SF2) may include a plurality of consecutive second microframes (MF2), the exposure section of the third subframe (SF3) may include a plurality of consecutive third microframes (MF3), and the exposure section of the fourth subframe (SF4) may include a plurality of consecutive fourth microframes (MF4).

[0047] The readout section may be a section in which the readout circuit 130 generates image data (IDATA) based on an electrical signal corresponding to the photocharge accumulated inside the pixel (PX) and transmits the image data (IDATA) to the image processing device 200.

[0048] FIG. 4a is a diagram showing an optical modulation signal and a modulation control signal in the first microframe. Referring to FIG. 4a, a first microframe (MF1) constituting an exposure section of a first subframe (SF1) in FIG. 3 may include a first section (P1) and a second section (P2).

[0049] The light modulation signal (MLS) in the first microframe (MF1) may be a square wave having alternating logic high levels (H) and logic low levels (L). The light source 10 may output the irradiation light (EL) in response to the light modulation signal (MLS) of logic high level (H), and may not output the irradiation light (EL) in response to the light modulation signal (MLS) of logic low level (L). Therefore, the waveform of the irradiation light (EL) may be the same as the waveform of the light modulation signal (MLS), and the light modulation signal (MLS) and the irradiation light (EL) may be used interchangeably in the following description.

[0050] The optical modulation signal (MLS) has a predetermined modulation frequency (F) and can have a period (T) corresponding to 1 / F. Since the pulses of the optical modulation signal (MLS) have a periodicity that repeats every period (T), the phase difference between the optical modulation signal (MLS) and the reflected light (RL) also has a periodicity. In the indirect ToF method, the distance to the target object 1 is estimated based on the phase difference between the optical modulation signal (MLS) and the reflected light (RL), but due to the periodicity of the phase difference, the phase difference can appear in the same way even if the distance to the target object 1 is different. The maximum measurement distance (Dfreq), which depends on the modulation frequency (F) of the optical modulation signal (MLS), can be calculated by the following Equation 1: where c can mean the speed of light.

[0051] [Formula 1]

number

[0052] For example, when the modulation frequency (F) of the optical modulation signal (MLS) is 100 MHz, the period (T) of the optical modulation signal (MLS) may be 10 ns. In this case, the maximum measurement distance (Dfreq) depending on the modulation frequency (F) may be 1.5 m. That is, the phase difference corresponding to the distance from the target object 1 being 0.1 m and the phase difference corresponding to the distance from the target object 1 being 1.6 m may be the same, and therefore it may be impossible to distinguish between them. This phenomenon may be defined as aliasing, and in this disclosure, aliasing may be overcome by using the contrast of the image data (IDATA). A more detailed description thereof will be given later with reference to FIG. 5 and FIG. 6. The period (T) may mean the period in which the pulse is repeated, and may also be referred to as the pulse period.

[0053] The first section (P1) of the first microframe (MF1) may have a length of N (N is an integer equal to or greater than 1)×T. In the first section (P1) of the first microframe (MF1), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) may maintain a logic high level (H). The first transfer transistor (TX1) and the second transfer transistor (TX2) receiving the first modulation control signal (MCS1) and the second modulation control signal (MCS2) of the logic high level (H), respectively, may be turned on, and the photoelectric charges generated in the first section (P1) may be equally distributed and stored in the first tap and the second tap. Here, N may be a value that satisfies the condition of the following Equation 2.

[0054] [Formula 2]

number

[0055] Here, the target measurement distance (Dmax) may mean the maximum measurement distance that the imaging device (ID) attempts to measure without aliasing, i.e., N may be an integer equal to or greater than the upper limit of the value obtained by dividing the target measurement distance Dmax by the maximum measurement distance Dfreq.

[0056] For example, when a distance of up to 6 m is to be measured without aliasing using an optical modulation signal (MLS) with a modulation frequency (F) of 100 MHz, N may be an integer equal to or greater than 6 / (1.5)=4.

[0057] In addition, even when the distance to the target object 1 is relatively short, the number of pulses (K) of the optical modulation signal (MLS) may be an integer equal to or greater than N+1 so that the phase difference between the optical modulation signal (MLS) and the reflected light (RL) can be detected normally.

[0058] The second section (P2) of the first microframe (MF1) may have a length of M (M is an integer equal to or greater than 1)×T. In the second section (P2) of the first microframe (MF1), the first modulation control signal (MCS1) may have the same phase as the optical modulation signal (MLS), and the second modulation control signal (MCS2) may have a modulation phase difference (i.e., opposite phase) of 180 degrees from the optical modulation signal (MLS). Thus, the photocharges generated in the second section (P2) may be distributed and accumulated in the first tap and the second tap in a ratio according to the time delay between the optical modulation signal (MLS) and the reflected light (RL). Here, M may be a value that satisfies the condition of the following Equation 3.

[0059] [Formula 3]

number

[0060] When Equation 3 is rearranged with respect to K, K may be a value equal to or less than (N+M)-1. In other words, the number of pulses (K) may be smaller than (N+M), which is the total time of the first microframe (MF1), (N+M)×T, divided by the period (T).

[0061] When the number of pulses (K) of the optical modulation signal (MLS), the number of periods (N) of the first section (P1), and the number of periods (M) of the second section (P2) are determined, the target measurement distance (Dmax) can be determined by the following Equation 4.

[0062] [Formula 4]

number

[0063] In the first microframe (MF1), the first tap can accumulate photocharges captured while the first modulation control signal (MCS1) has a logic high level (H), and in the first microframe (MF1), the second tap can accumulate photocharges captured while the second modulation control signal (MCS2) has a logic high level (H).

[0064] In the readout period of the first subframe (SF1), the first tap may output an electrical signal corresponding to the photocharges accumulated in a plurality of first microframes (MF1). Data converted from the electrical signal output from the first tap may be defined as first capture data (S0) of the first tap. The first capture data (S0) of the first tap may include a component for the photocharges captured by the first modulation control signal (MCS1) having a modulation phase difference (i.e., in phase) of 0 degrees with the optical modulation signal (MLS).

[0065] In the readout period of the first subframe (SF1), the second tap may output an electrical signal corresponding to the photocharges accumulated in the plurality of first microframes (MF1). Data converted from the electrical signal output from the second tap may be defined as third capture data (S180) of the second tap. The third capture data (S180) of the second tap may include a component corresponding to the photocharges captured by the second modulation control signal (MCS2) having a modulation phase difference of 180 degrees with the optical modulation signal (MLS).

[0066] FIG. 4b is a diagram showing the optical modulation signal and the modulation control signal in the second microframe. Referring to FIG. 4b, a second microframe (MF2) constituting an exposure section of a second subframe (SF2) of FIG. 3 may include a first section (P1) and a second section (P2).

[0067] The optical modulation signal (MLS), the first modulation control signal (MCS1), and the second modulation control signal (MCS2) of the second microframe (MF2) are substantially similar to the optical modulation signal (MLS), the first modulation control signal (MCS1), and the second modulation control signal (MCS2) described in Figure 4a, except for some differences, so duplicated explanations will be omitted.

[0068] The first modulation control signal (MCS1) of the second microframe (MF2) may have a modulation phase difference of 90 degrees with the optical modulation signal (MLS) in the second section (P2). Also, the second modulation control signal (MCS2) of the second microframe (MF2) may have a modulation phase difference of 270 degrees with the optical modulation signal (MLS) in the second section (P2).

[0069] In addition, the number of pulses (K) of the optical modulation signal (MLS) in the second microframe (MF2), the number of periods (N) of the first section (P1), the number of periods (M) of the second section (P2), and the target measurement distance (Dmax) may be the same as those in the first microframe (MF1).

[0070] In the second microframe (MF2), the first tap can accumulate photocharges captured while the first modulation control signal (MCS1) has a logic high level (H), and in the second microframe (MF2), the second tap can accumulate photocharges captured while the second modulation control signal (MCS2) has a logic high level (H).

[0071] In the readout period of the second subframe (SF2), the first tap may output an electrical signal corresponding to the photocharges accumulated in a plurality of second microframes (MF2). Data converted from the electrical signal output from the first tap may be defined as second capture data (S90) of the first tap. The second capture data (S90) of the first tap may include a component corresponding to the photocharges captured by the first modulation control signal (MCS1) having a modulation phase difference of 90 degrees with the optical modulation signal (MLS).

[0072] In the readout period of the second subframe (SF2), the second tap may output an electrical signal corresponding to the photocharges accumulated in a plurality of second microframes (MF2). Data converted from the electrical signal output from the second tap may be defined as fourth capture data (S270) of the second tap. The fourth capture data (S270) of the second tap may include a component corresponding to the photocharges captured by the second modulation control signal (MCS2) having a modulation phase difference of 270 degrees with the optical modulation signal (MLS).

[0073] FIG. 4c shows the optical modulation signal and the modulation control signal in the third microframe. Referring to FIG. 4c, a third microframe (MF3) constituting an exposure section of the third subframe (SF3) of FIG. 3 may include a first section (P1) and a second section (P2).

[0074] The optical modulation signal (MLS), the first modulation control signal (MCS1), and the second modulation control signal (MCS2) of the third microframe (MF3) are substantially similar to the optical modulation signal (MLS), the first modulation control signal (MCS1), and the second modulation control signal (MCS2) described in Figure 4a, except for some differences, so duplicated explanations will be omitted.

[0075] The first modulation control signal (MCS1) of the third microframe (MF3) may have a modulation phase difference of 180 degrees with the optical modulation signal (MLS) in the second section (P2). Also, the second modulation control signal (MCS2) of the third microframe (MF3) may have a modulation phase difference of 0 degrees with the optical modulation signal (MLS) in the second section (P2).

[0076] In addition, the number of pulses (K) of the optical modulation signal (MLS) in the third microframe (MF3), the number of periods (N) of the first section (P1), the number of periods (M) of the second section (P2), and the target measurement distance (Dmax) in the third microframe (MF3) may be the same as those in the first microframe (MF1).

[0077] In the third microframe (MF3), the first tap can accumulate photocharges captured while the first modulation control signal (MCS1) has a logic high level (H), and in the third microframe (MF3), the second tap can accumulate photocharges captured while the second modulation control signal (MCS2) has a logic high level (H).

[0078] In the readout period of the third subframe (SF3), the first tap may output an electrical signal corresponding to the photocharges accumulated in a plurality of third microframes (MF3). Data converted from the electrical signal output from the first tap may be defined as third capture data (S180) of the first tap. The third capture data (S180) of the first tap may include a component corresponding to the photocharges captured by the first modulation control signal (MCS1) having a modulation phase difference of 180 degrees with the optical modulation signal (MLS).

[0079] In the readout section of the third subframe (SF3), the second tap may output an electrical signal corresponding to the photocharges accumulated in a plurality of third microframes (MF3). Data converted from the electrical signal output from the second tap may be defined as the first capture data (S0) of the second tap. The first capture data (S0) of the second tap may include a component corresponding to the photocharges captured by the second modulation control signal (MCS2) having a modulation phase difference of 0 degrees with the optical modulation signal (MLS).

[0080] FIG. 4d shows the optical modulation signal and the modulation control signal in the fourth microframe. Referring to FIG. 4d, a fourth microframe (MF4) constituting an exposure section of the fourth subframe (SF4) of FIG. 3 may include a first section (P1) and a second section (P2).

[0081] The optical modulation signal (MLS), the first modulation control signal (MCS1), and the second modulation control signal (MCS2) of the fourth microframe (MF4) are substantially similar to the optical modulation signal (MLS), the first modulation control signal (MCS1), and the second modulation control signal (MCS2) described in Figure 4a, except for a few differences, so duplicated explanations will be omitted.

[0082] The first modulation control signal (MCS1) of the fourth microframe (MF4) may have a modulation phase difference of 270 degrees with the optical modulation signal (MLS) in the second section (P2). Also, the second modulation control signal (MCS2) of the fourth microframe (MF4) may have a modulation phase difference of 90 degrees with the optical modulation signal (MLS) in the second section (P2).

[0083] In addition, the number of pulses (K) of the optical modulation signal (MLS) in the fourth microframe (MF), the number of periods (N) of the first section (P1), the number of periods (M) of the second section (P2), and the target measurement distance (Dmax) may be the same as those in the first microframe (MF1).

[0084] In the fourth microframe (MF4), the first tap can accumulate photocharges captured while the first modulation control signal (MCS1) has a logic high level (H), and in the fourth microframe (MF4), the second tap can accumulate photocharges captured while the second modulation control signal (MCS2) has a logic high level (H).

[0085] In the readout period of the fourth subframe (SF4), the first tap may output an electrical signal corresponding to the photocharges accumulated in a plurality of fourth microframes (MF4). The data converted from the electrical signal output from the first tap may be defined as the fourth capture data (S270) of the first tap. The fourth capture data (S270) of the first tap may include a component corresponding to the photocharges captured by the first modulation control signal (MCS1) having a modulation phase difference of 270 degrees with the optical modulation signal (MLS).

[0086] In the readout period of the fourth subframe (SF4), the second tap may output an electrical signal corresponding to the photocharges accumulated in a plurality of fourth microframes (MF4). Data converted from the electrical signal output from the second tap may be defined as second capture data (S90) of the second tap. The second capture data (S90) of the second tap may include a component corresponding to the photocharges captured by the second modulation control signal (MCS2) having a modulation phase difference of 90 degrees with the optical modulation signal (MLS).

[0087] As described in FIG. 4a to FIG. 4d, the first to fourth capture data (S0 to S270) of the first tap and the second tap can be acquired from the first to fourth subframes (SF1 to SF4). The first to fourth capture data (S0 to S270) are necessary to calculate the distance to the target object 1. The first to fourth capture data (S0 to S270) can be acquired from the first and second subframes (SF1, SF2) through the first tap and the second tap, but the first tap and the second tap can output different signals even under similar operating conditions due to differences in hardware characteristics. Therefore, in the present disclosure, the first to fourth capture data (S0 to S270) are acquired from the first tap and the second tap in units of depth frames through the first to fourth subframes (SF1 to SF4), so that noise due to differences in characteristics between taps does not affect the process of calculating the distance to the target object 1.

[0088] The phase difference calculator 210 of the image processing device 200 may sum the first capture data (S0) of the first tap and the first capture data (S0) of the second tap to generate the first capture data (S0) of the corresponding depth frame. In addition, the phase difference calculator 210 may sum the second capture data (S90) of the first tap and the second capture data (S90) of the second tap to generate the second capture data (S90) of the corresponding depth frame. In addition, the phase difference calculator 210 may sum the third capture data (S180) of the first tap and the third capture data (S180) of the second tap to generate the third capture data (S180) of the corresponding depth frame. Similarly, the phase difference calculator 210 may sum the fourth capture data (S270) of the first tap and the fourth capture data (S270) of the second tap to generate the fourth capture data (S270) of the corresponding depth frame.

[0089] The phase difference calculator 210 can calculate the phase difference (PDIF) between the light modulation signal (MLS) (or the irradiated light (EL)) and the reflected light (RL) based on the first to fourth capture data (S0 to S270) of the depth frame using the following Equation 5.

[0090] [Formula 5]

number

[0091] In principle, the distance to the target object 1 must be accurately calculated based on the phase difference (PDIF), but when the target measurement distance (Dmax) exceeds the maximum measurement distance (Dfreq), the same phase difference (PDIF) may appear even when the target object 1 is located at different distances due to the periodicity of the optical modulation signal (MLS). Therefore, in the present disclosure, a method can be used to more accurately calculate the distance to the target object 1 based on a value obtained by correcting the phase difference (PDIF) rather than using the phase difference (PDIF) as it is.

[0092] FIG. 5 is a graph showing the relationship between the distance to the target object and the contrast of the reflected light. 5 shows an exemplary simulation result of the relationship between the distance to the target object 1 and the contrast of the reflected light (RL). Here, the contrast of the reflected light (RL) may mean the ratio between the amplitude of the reflected light (RL) and the intensity of the reflected light (RL). The amplitude of the reflected light (RL) can be calculated by the following Equation 6.

[0093] [Formula 6]

number

[0094] Moreover, the intensity of the reflected light (RL) can be calculated by the following Equation 7.

[0095] [Formula 7]

number

[0096] In Equation 6 and Equation 7, S0 to S270 may refer to the first to fourth captured data (S0 to S270) of the corresponding depth frame. Thus, the contrast of the reflected light (RL) can be calculated by the following Equation 8.

[0097] [Formula 8]

number

[0098] In Equation 8, the ratio between the amplitude component and the intensity component is illustrated as being multiplied by a coefficient of 2, but since the ratio between the amplitude component and the intensity component is important, the coefficient may be set to any value except 0.

[0099] 5 may be a result of a simulation performed under the conditions that the modulation frequency (F) is 100 MHz, the number of periods (N) of the first section (P1) is 4, the number of pulses (K) of the optical modulation signal (MLS) is 10, the number of periods (M) of the second section (P2) is 15, and the target measurement distance (Dmax) is 6 m. In addition, since the modulation frequency (F) is 100 MHz, the period (T) may be 10 ns and the maximum measurement distance (Dfreq) may be 1.5 m.

[0100] In FIG. 5, as the distance to the target object 1 increases, the contrast of the reflected light (RL) gradually increases up to the target measurement distance (Dmax) of 6 m, gradually decreases up to 12 m, remains constant up to 24 m, and then increases again from 24 m.

[0101] FIG. 6 is a graph showing the relationship between the phase difference and the contrast of the reflected light. Referring to FIG. 6, the relationship between the phase difference (PDIF) and contrast while the distance to the target object 1 increases from 0 m to the target measurement distance (Dmax) of 6 m is shown.

[0102] As the distance to the target object 1 increases from 0 m to 1.5 m, the phase difference (PDIF) can increase from 0 [rad] to 2π [rad], and the contrast can also increase gradually with the increase in the phase difference (PDIF). The relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 0 m to 1.5 m can be represented by a first measurement value (MV1). That is, the first measurement value (MV1) is a set of points indicating the relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 0 m to 1.5 m, and can be approximated by a first straight line (AL1) having a first slope and a first intercept (C1). A method of approximating a set of points to one straight line may be a method of determining a virtual straight line that minimizes the sum of squares of the errors of the points with respect to the virtual straight line, but the scope of the present invention is not limited thereto.

[0103] As the distance to the target object 1 increases from 1.5 m to 3 m, the phase difference (PDIF) can increase from 0 [rad] to 2π [rad], and the contrast can also increase gradually with the increase in the phase difference (PDIF). The relationship between the phase difference (PDIF) and the contrast when the distance to the target object 1 increases from 1.5 m to 3 m can be represented by a second measurement value (MV2). That is, the second measurement value (MV2) is a set of points showing the relationship between the phase difference (PDIF) and the contrast when the distance to the target object 1 increases from 1.5 m to 3 m, and can be approximated by a second straight line (AL2) having a first slope and a second intercept (C2).

[0104] As the distance to the target object 1 increases from 3 m to 4.5 m, the phase difference (PDIF) can increase from 0 [rad] to 2π [rad], and the contrast can also increase gradually with the increase in the phase difference (PDIF). The relationship between the phase difference (PDIF) and the contrast when the distance to the target object 1 increases from 3 m to 4.5 m can be represented by a third measurement value (MV3). That is, the third measurement value (MV3) is a set of points showing the relationship between the phase difference (PDIF) and the contrast when the distance to the target object 1 increases from 3 m to 4.5 m, and can be approximated by a third straight line (AL3) having a first slope and a third intercept (C3).

[0105] As the distance to the target object 1 increases from 4.5 m to 6 m, the phase difference (PDIF) can increase from 0 [rad] to 2π [rad], and the contrast can also increase gradually with the increase in the phase difference (PDIF). The relationship between the phase difference (PDIF) and the contrast when the distance to the target object 1 increases from 4.5 m to 6 m can be shown by the fourth measurement value (MV4). That is, the fourth measurement value (MV4) is a set of points showing the relationship between the phase difference (PDIF) and the contrast when the distance to the target object 1 increases from 4.5 m to 6 m, and can be approximated by a fourth straight line (AL4) having a first slope and a fourth intercept (C4).

[0106] For example, when the distance to the target object 1 is 0.75 m, 2.25 m, 3.75 m, and 5.25 m, the phase difference (PDIF) may appear as approximately π [rad] even though the distance to the target object 1 is different. However, when the distance to the target object 1 is 0.75 m, 2.25 m, 3.75 m, and 5.25 m, the contrast is different, so the phase difference corrector 220 can correct the phase difference (PDIF) using such characteristics.

[0107] According to an embodiment, the phase difference corrector 220 can correct the phase difference (PDIF) by using linear function characteristics of each of the first to fourth measurement values ​​(MV1 to MV4).

[0108] The first to fourth straight lines (AL1 to AL4) to which the first to fourth measurement values ​​(MV1 to MV4) respectively approximate have the same first slope (A) but may have first to fourth intercepts (C1 to C4) that gradually increase. The first slope (A) may mean the ratio between the increase in phase difference (PDIF) and the increase in contrast, and may be a value that is experimentally determined in advance.

[0109] The phase difference corrector 220 can use the first slope (A) of the first to fourth straight lines (AL1 to AL4) to calculate the measurement intercept (B) corresponding to the currently calculated phase difference (PDIF) and contrast according to the following Equation 9.

[0110] [Formula 9]

number

[0111] The phase difference corrector 220 divides the measurement section (B) into first to third reference sections (C a1 ~C a3 ), the aliasing value (R) corresponding to the measurement intercept (B) can be calculated. Here, the first to third reference intercepts (C a1 ~C a3 ) can be calculated using the first to fourth intercepts (C1 to C4) according to the following formula 10.

[0112] [Formula 10]

number

[0113] Specifically, the phase difference corrector 220 detects whether the measurement intercept (B) is a first to third reference intercept (C a1 ~C a3 ), the aliasing value (R) can be determined by determining which of the four ranges divided into three ranges (AL1-AL4) the phase difference (PDIF) belongs to. The first to fourth lines (AL1-AL4) may correspond to the first to fourth aliasing periods, respectively, and the contrast may increase as the phase difference (PDIF) increases in the first to fourth aliasing periods, but the contrast corresponding to the phase difference (PDIF) may be different in the first to fourth aliasing periods. The aliasing value (R) can indicate which of the first to fourth aliasing periods the aliasing period corresponding to the phase difference (PDIF) is.

[0114] In the first aliasing period, as the distance to the target object 1 increases from 0 m to 1.5 m, the phase difference (PDIF) increases from 0 to 2π. In the second aliasing period, as the distance to the target object 1 increases from 1.5 m to 3 m, the phase difference (PDIF) increases from 0 to 2π. In the third aliasing period, as the distance to the target object 1 increases from 3 m to 4.5 m, the phase difference (PDIF) increases from 0 to 2π. In the fourth aliasing period, as the distance to the target object 1 increases from 4.5 m to 6 m, the phase difference (PDIF) increases from 0 to 2π. That is, the distance to the target object 1 indicated by the phase difference (PDIF) can differ depending on which aliasing period among the first to fourth aliasing periods corresponds to the phase difference (PDIF). The aliasing value (R) is calculated by the following formula 11, which is the relationship between the measurement intercept (B) and the first to third reference intercepts (C a1 ~Ca3 ) can be determined by comparing

[0115] [Formula 11]

number

[0116] The aliasing value (R) may be a value indicating which of the first to fourth straight lines (AL1 to AL4) the measurement intercept (B) corresponding to the currently calculated phase difference (PDIF) and contrast corresponds to, and may indicate which of the first to fourth aliasing periods the aliasing period corresponding to the phase difference (PDIF) is. For example, if the aliasing value (R) is 1, the measurement intercept (B) corresponds to the first straight line (AL1), and the aliasing period corresponding to the phase difference (PDIF) may be the first aliasing period.

[0117] The phase difference corrector 220 may use the aliasing value (R) to correct the phase difference (PDIF) according to the following Equation 12 to calculate the corrected phase difference (PDIF').

[0118] [Formula 12]

number

[0119] As the aliasing value (R) increases by 1, the phase difference corrector 220 may increase the phase difference (PDIF) by 2π to calculate the corrected phase difference (PDIF'). In other words, (R-1)·2π may be defined as a phase correction value based on the aliasing value (R), and the corrected phase difference (PDIF') may be a value obtained by adding the phase correction value to the phase difference (PDIF).

[0120] That is, the phase difference corrector 220 can calculate a measurement intercept (B) corresponding to the currently calculated phase difference (PDIF) and contrast, determine an aliasing value (R) corresponding to the measurement intercept (B), and calculate a corrected phase difference (PDIF') by correcting the phase difference (PDIF) using the aliasing value (R). The distance calculator 230 can calculate the distance (Dr) to the target object 1 using the corrected phase difference (PDIF') according to the following Equation 13.

[0121] [Formula 13]

number

[0122] According to the present disclosure, despite the aliasing phenomenon, which is a limitation of the indirect ToF method, by determining the aliasing period corresponding to the calculated phase difference (PDIF) based on the contrast and correcting the phase difference (PDIF), it is possible to accurately measure the distance to the target object 1 within the target measurement distance (Dmax) that exceeds the maximum measurement distance (Dfreq) based on the modulation frequency (F).

[0123] The principle that such a method is possible is that the contrast can indicate an aliasing period. In the first section (P1) of the microframe (e.g., MF1), the first and second modulation control signals (MCS1, MCS2) both have a logic high level (H), so that the intensity component of the reflected light (RL) is mainly obtained, and in the second section (P2) of the microframe (e.g., MF1), the first and second modulation control signals (MCS1, MCS2) have a predetermined modulation phase difference with the illuminating light (EL) and alternately have a logic high level (H), so that the amplitude component of the reflected light (RL) can be mainly obtained. As a result, as the phase difference between the illuminating light (EL) and the reflected light (RL) increases, the amplitude component of the reflected light (RL) can become relatively stronger compared to the intensity component of the reflected light (RL), so that the contrast can indicate an aliasing period.

[0124] On the other hand, the first slope (A) and the first to third reference intercepts (C a1 ~C a3 ) is a value that depends on the characteristics (e.g., N, M, L, etc.) of the image sensing device 100, and the phase difference corrector 220 calculates the first slope (A) and the first to third reference intercepts (C a1 ~C a3 ) may be received from the image sensing device 100 or an external device (eg, a test device) and prestored.

[0125] To explain the operation of the image processing device 200 overall again, the phase difference calculator 210 can calculate the phase difference (PDIF) between the optical modulation signal (MLS) and the reflected light (RL) based on multiple capture data (e.g., S0, S90, S180, S270) generated by multiple modulation control signals (e.g., MCS1, MCS2), each of which has a predetermined modulation phase difference with the optical modulation signal (MLS).

[0126] The phase difference corrector 220 can calculate the contrast, which is the ratio between the amplitude component and the intensity component of the reflected light (RL), using multiple captured data (e.g., S0, S90, S180, S270), and determine the aliasing value (R) corresponding to the contrast.

[0127] Then, the distance calculator 230 can accurately calculate the distance to the target object 1 using the corrected phase difference (PDIF') obtained by correcting the phase difference (PDIF) according to the aliasing value (R).

[0128] FIG. 7 is a diagram for explaining an embodiment capable of removing the component due to the ambient light from the intensity of the reflected light. 7, in order to accurately determine the aliasing period based on the contrast using the above-mentioned method, the intensity component of the reflected light (RL) must be sufficiently decreased and the contrast must be significantly increased as the distance to the target object 1 increases. However, if the intensity component of the reflected light (RL) contains an excessively large component due to ambient light (hereinafter referred to as "ambient light component"), the change in contrast is insufficient despite the change in the distance to the target object 1, making it difficult to distinguish the aliasing period.

[0129] Moreover, if the ambient light component fluctuates drastically, it will not affect the amplitude component of the contrast, but may have a dominant effect on the intensity component of the contrast depending on the degree of the fluctuation. In order to accurately measure the distance to the target object 1, the contrast must change only depending on the distance to the target object 1, but since the contrast may change depending on the fluctuation of the ambient light component, the distance to the target object 1 cannot be accurately measured.

[0130] FIG. 7 shows one embodiment for obtaining the ambient light component. The depth frame according to an embodiment of Fig. 7 may further include a fifth subframe, unlike the depth frame of Fig. 3. The fifth subframe may include an exposure section and a readout section, similar to the first to fourth subframes (SF1 to SF4) of Fig. 3, and basic operations in the exposure section and the readout section are similar to those described in Fig. 3 except for some differences, so the following description will focus on the different operations.

[0131] The exposure section of the fifth subframe may include one fifth microframe (MF5). The light modulation signal (MLS) in the fifth microframe (MF5) maintains a logic low level (L), so that the irradiated light (EL) and the reflected light (RL) are not generated. In addition, the first modulation control signal (MCS1) and the second modulation control signal (MCS2) in the fifth microframe (MF5) each have a logic high level, so that the ambient light component may be captured at the first tap and the second tap. Image data corresponding to the ambient light component captured at the first tap and the second tap may be defined as sub-ambient light data (AMBsub). If the length of the exposure section of the fifth subframe is defined as a background exposure time (Tamb) and the length of each of the exposure sections of the first to fourth subframes (SF1 to SF4) is defined as a reference exposure time (Tdepth), the phase difference corrector 220 may calculate the ambient light data (AMB) according to the following Equation 14.

[0132] [Formula 14]

number

[0133] In the fifth microframe (MF5), the first tap and the second tap can capture the ambient light component without the reflected light (RL), and the background exposure time (Tamb) of the fifth microframe (MF5) (or the fifth subframe) does not necessarily have to be the same as the reference exposure time (Tdepth) of each of the first to fourth subframes (SF1 to SF4), and may be shorter than the reference exposure time (Tdepth). Even if the background exposure time (Tamb) is shorter than the reference exposure time (Tdepth), the phase difference corrector 220 can calculate the ambient light data (AMB) by multiplying the sub ambient light data (AMBsub) by the ratio between the background exposure time (Tamb) and the reference exposure time (Tdepth). That is, the time required for the fifth microframe (MF) can be reduced as much as possible, and the decrease in the frame rate of the depth frame can be minimized. The phase difference corrector 220 can remove the ambient light component from the intensity of the reflected light (RL) by using the ambient light data (AMB) according to the following Equation 15.

[0134] [Formula 15]

number

[0135] The phase difference corrector 220 can calculate the contrast using the intensity of the reflected light (RL) from which the ambient light component has been removed, thereby allowing the contrast to change according to the distance to the target object 1 regardless of the ambient light component, thereby allowing the distance to the target object 1 to be measured accurately.

[0136] 8a to 9 are diagrams for explaining another embodiment capable of removing the component due to ambient light from the intensity of reflected light. With reference to Fig. 8a to Fig. 9, another embodiment for removing an ambient light component from the intensity of the reflected light (RL) will be described. The depth frames (DepthFrame0 to DepthFrame3) according to the other embodiment of Fig. 8a to Fig. 8d can include a waveform of a light modulation signal (MLS) with adjusted timing, unlike the depth frames of Fig. 3. The depth frames (DepthFrame0 to DepthFrame3) can include an exposure section and a readout section like the depth frames of Fig. 3, and the basic operations in the exposure section and the readout section are the same as those described in Fig. 3 except for some differences, so the following description will focus on the different operations.

[0137] The first to fourth depth frames (DepthFrame0 to DepthFrame3) may be consecutive to each other, and the optical modulation signal (MLS) generated in a microframe (e.g., MF1) of at least one subframe (e.g., SF1) in each of the first to fourth depth frames (DepthFrame0 to DepthFrame3) may be generated to have a predetermined time delay from the start of the microframe. Below, the first microframe (MF1) included in each of the first to fourth depth frames (DepthFrame0 to DepthFrame3) will be described as an example, but the same description can be applied to the other microframes (MF2 to MF4).

[0138] In FIG. 8a, the pulse of the optical modulation signal (MLS) generated in the first microframe (MF1) of the first depth frame (DepthFrame0) may be generated with a first delay time (D1) from the start of the first microframe (MF1).

[0139] In FIG. 8b, the pulse of the optical modulation signal (MLS) generated in the first microframe (MF1) of the second depth frame (DepthFrame1) can be generated with a delay of a second delay time (D2) from the start of the first microframe (MF1).

[0140] In FIG. 8c, the pulse of the optical modulation signal (MLS) generated in the first microframe (MF1) of the third depth frame (DepthFrame2) can be generated with a delay of a third delay time (D3) from the start of the first microframe (MF1).

[0141] In FIG. 8d, the pulse of the optical modulation signal (MLS) generated in the first microframe (MF1) of the fourth depth frame (DepthFrame3) can be generated with a fourth delay time (D4) from the start of the first microframe (MF1).

[0142] That is, the optical modulation signal (MLS) in each of the first to fourth depth frames (DepthFrame0 to DepthFrame3) may include an intentionally delayed pulse, and the first to fourth delay times (D1 to D4) may be different from each other but may be set to values ​​less than a time (e.g., period / 2) corresponding to a predetermined phase (e.g., π), although the scope of the present invention is not limited thereto.

[0143] 9, in a graph showing the relationship between the phase difference (PDIF) and the contrast of the reflected light, the phase difference (PDIF) and the contrast obtained from each of the first to fourth depth frames (DepthFrame0 to DepthFrame3) are displayed. The contrast corresponding to each of the first to fourth depth frames (DepthFrame0 to DepthFrame3) can be plotted at a position deviated from the first measurement value (MV1). This is because the contrast changes depending on the ambient light component, and the phase difference corrector 220 can calculate the slope (Aamb) of an approximation line that approximates the four coordinates of the phase difference (PDIF) and the contrast obtained from each of the first to fourth depth frames (DepthFrame0 to DepthFrame3).

[0144] The phase difference corrector 220 calculates the intensity component (Intensity amb The phase difference corrector 220 calculates the intensity component (Intensity) of the reflected light (RL) obtained by removing the ambient light component from the fourth depth frame (DepthFrame3) by the following Equation 16. amb For the correction of the fourth depth frame (DepthFrame3), information obtained from the three depth frames (DepthFrame0 to DepthFrame2) preceding the fourth depth frame (DepthFrame3) can be used. This allows the Intensity component of the Reflected Light (RL) ambIt can prevent unnecessary frame rate drops that may occur due to correction of reflected light (RL) intensity components for each depth frame. amb ) can be corrected.

[0145] [Formula 16]

number

[0146] Here, A can mean a first slope of the first to fourth straight lines (AL1 to AL4) to which the first to fourth measurement values ​​(MV1 to MV4) respectively approximate.

[0147] The phase difference corrector 220 can calculate the contrast using the intensity component of the reflected light (RL) from which the ambient light component has been removed, and thus the contrast can change according to the distance to the target object 1 regardless of the ambient light component, so that the distance to the target object 1 can be accurately measured.

[0148] Furthermore, since in each depth frame (DepthFrame0 to 3), the pulse of the optical modulation signal (MLS) is intentionally delayed by each delay time (D1 to D4) when generating the pulse, the distance calculator 230 can compensate (e.g., add or subtract) the distance according to each delay time corresponding to each depth frame when calculating the distance to the target object 1.

[0149] In the examples of Figures 8a to 9, the slope (Aamb) is calculated using an approximation line that approximates the four coordinates of phase difference (PDIF) and contrast obtained from four depth frames for one pixel, but in other embodiments, a more accurate slope (Aamb) of the approximation line can be obtained by using a slope that is an average of slopes calculated for multiple pixels.

[0150] According to another embodiment, the intensity component of the reflected light (RL) is ambThe calculation of the slope ratio (Aamb / A) for correcting the slope ratio (Aamb / A) is not performed in all depth frames, but may be periodically performed in some depth frames according to a predetermined cycle. Here, in a depth frame in which the slope ratio (Aamb / A) is not calculated, the slope ratio (Aamb / A) calculated in the previous depth frame is used to calculate the intensity component (Intensity) of the reflected light (RL). amb ) can be corrected.

[0151] According to another embodiment, the intensity component of the reflected light (RL) is amb The calculation of the slope ratio (Aamb / A) for the correction of the reflected light (RL) intensity component (Intensity) may be performed when a certain condition (e.g., the change in the illuminance value measured by the illuminance sensor (not shown) is equal to or greater than a threshold, or the change in the sum (or average) of the intensity components of all pixels is equal to or greater than a threshold) is met. Here, in a depth frame in which the slope ratio (Aamb / A) is not calculated, the slope ratio (Aamb / A) calculated in the previous depth frame is used to calculate the reflected light (RL) intensity component (Intensity) amb ) can be corrected.

[0152] The number of subframes per depth frame, the order of the first section (P1) and the second section (P2) in each microframe, the order of the microframes, etc. described in this disclosure are merely illustrative and can be varied as much as possible within the scope of the application of the technical ideas of the present invention.

[0153] FIG. 10 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG. Referring to FIG. 10, a computing device 1000 may represent one embodiment of a hardware configuration for performing the operations of the image processing device 200 of FIG.

[0154] The computing device 1000 may be mounted on a chip separate from the chip on which the image sensing device 100 is mounted. According to an embodiment, the chip on which the image sensing device 100 is mounted and the chip on which the computing device 1000 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), but the scope of the present invention is not limited thereto.

[0155] Computing device 1000 may include a processor 1010, a memory 1020, an input / output interface 1030, and a communication interface 1040.

[0156] The processor 1010 is capable of processing data and / or instructions required to perform the operations of the components 210 to 230 of the image processing device 200 described in FIG.

[0157] The memory 1020 can store data and / or instructions necessary to perform operations of the components 210 to 230 of the image processing device 200, and can be accessed by the processor 1010. For example, the memory 1020 can be realized by a volatile memory (e.g., a Dynamic Random Access Memory (DRAM), a Static Random Access Memory (SRAM), etc.) or a non-volatile memory (e.g., a Programmable Read Only Memory (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), a flash memory, etc.).

[0158] That is, a computer program for performing the operations of the image processing device 200 disclosed in this document can be recorded in the memory 1020 and executed and processed by the processor 1010 to realize the operations of the image processing device 200.

[0159] The input / output interface 1030 can provide an interface to connect an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 1010 and enable data to be transmitted and received.

[0160] The communication interface 1040 is configured to be capable of transmitting and receiving various data to and from an external device (eg, an application processor, an external memory, etc.), and may be a device capable of supporting wired or wireless communication.

Claims

1. a phase difference calculator for calculating a phase difference between the optical modulation signal and the reflected light based on a plurality of capture data generated by a plurality of modulation control signals each having a predetermined modulation phase difference with the optical modulation signal; a phase difference corrector that calculates a contrast, which is a ratio between an amplitude component and an intensity component of the reflected light, using the plurality of captured data, and determines an aliasing value corresponding to the contrast; a distance calculator that calculates a distance to a target object using a corrected phase difference obtained by correcting the phase difference according to the aliasing value; 2. An image processing device comprising:

2. The image processing device according to claim 1 , wherein each of the plurality of modulation control signals has a modulation phase difference of any one of 0 degrees, 90 degrees, 180 degrees, and 270 degrees with respect to the optical modulation signal.

3. The plurality of capture data are generated in units of depth frames, The imaging device of claim 1 , wherein the depth frame comprises a plurality of sub-frames.

4. 4. The image processing device of claim 3, wherein each of the plurality of subframes includes an exposure section for accumulating photocharges generated by sensing the reflected light according to the plurality of modulation control signals, and a readout section for generating the capture data corresponding to the accumulated photocharges.

5. The exposure interval includes a plurality of microframes; 5. The image processing apparatus of claim 4, wherein each of the plurality of microframes includes a first period during which the plurality of modulation control signals maintain a logic high level, and a second period during which the plurality of modulation control signals alternately have the logic high level.

6. The image processing device according to claim 5 , wherein in the second section, the plurality of modulation control signals have opposite phases to each other.

7. The image processing device according to claim 5 , wherein in each of the plurality of microframes, the number of pulses of the optical modulation signal is smaller than a value obtained by dividing a time of the microframe by a pulse period.

8. The image processing apparatus of claim 4 , wherein the exposure period includes microframes during which the optical modulation signal maintains a logic low level and each of the plurality of modulation control signals maintains a logic high level.

9. The image processing apparatus of claim 8 , wherein the phase difference corrector removes an ambient light component from the intensity component using ambient light data obtained from the microframes.

10. 6. The image processing device according to claim 5, wherein in each of the plurality of microframes, a pulse of the optical modulation signal is generated with a delay of a predetermined delay time from a start point of the microframe.

11. The image processing device of claim 10 , wherein the phase difference corrector corrects the intensity component based on a slope of an approximation line that approximates coordinates of phase difference and contrast obtained from a current depth frame and at least one depth frame preceding the current depth frame.

12. 2. The image processing device of claim 1, wherein the phase difference corrector calculates a measurement intercept based on a first slope which is a ratio between an increase in the phase difference and an increase in the contrast, the contrast, and the phase difference, and compares the measurement intercept with a plurality of reference intercepts to determine an aliasing value corresponding to the contrast.

13. The image processing apparatus of claim 12 , wherein the plurality of reference intercepts are determined by intercepts of a line corresponding to the aliasing value.

14. The image processing apparatus according to claim 1 , wherein the corrected phase difference is a value obtained by adding a phase correction value based on the aliasing value to the phase difference.

15. an image sensing device for generating a plurality of capture data using a plurality of modulation control signals each having a predetermined modulation phase difference with an optical modulation signal; an image processing device that calculates a contrast, which is a ratio between an amplitude component and an intensity component of the reflected light, using the plurality of captured data, and calculates a distance to a target object using a corrected phase difference obtained by correcting a phase difference between the optical modulation signal and the reflected light according to an aliasing value corresponding to the contrast; 13. An imaging device comprising:

16. The plurality of capture data are generated in units of depth frames, the depth frame includes a plurality of subframes; The imaging device of claim 15, wherein each of the plurality of subframes includes an exposure section for accumulating photocharges generated by sensing the reflected light in accordance with the plurality of modulation control signals, and a readout section for generating the capture data corresponding to the accumulated photocharges.

17. The exposure interval includes a plurality of microframes; 17. The imaging device of claim 16, wherein each of the plurality of microframes includes a first interval during which the plurality of modulation control signals maintain a logic high level and a second interval during which the plurality of modulation control signals alternately have the logic high level.

18. The imaging device of claim 17 , wherein in each of the plurality of microframes, the number of pulses of the optical modulation signal is less than a value obtained by dividing a time of the microframe by a pulse period.

19. 16. The imaging device of claim 15, wherein the image processor calculates a measurement intercept based on a first slope, which is a ratio of an increase in the phase difference to an increase in the contrast, the contrast, and the phase difference, and compares the measurement intercept with a plurality of reference intercepts to determine an aliasing value corresponding to the contrast.

20. A pixel for sensing reflected light according to a plurality of modulation control signals each having a predetermined modulation phase difference with the light modulation signal to generate a plurality of pixel signals; a readout circuit for processing the pixel signals to generate a plurality of capture data; Including, an exposure section in which the pixel senses the reflected light includes a first section in which the plurality of modulation control signals maintain a logic high level, and a second section in which the plurality of modulation control signals alternately have the logic high level.

Images