Image sensing apparatus, image processing apparatus, and imaging apparatus

Abstract

The invention relates to an image sensing apparatus, an image processing apparatus, and an imaging apparatus. An image sensing apparatus includes: a phase difference calculator configured to calculate a phase difference between a modulated light signal and reflected light based on a plurality of captured data segments generated by a plurality of modulation control signals each having a predetermined modulation phase difference with respect to the modulated light signal; a phase difference corrector configured to calculate a contrast using the plurality of captured data segments and configured to determine an aliasing value corresponding to the contrast, the contrast being a ratio of an amplitude component of the reflected light to an intensity component of the reflected light; and a distance calculator configured to calculate a distance to the target object using a corrected phase difference obtained by correcting the phase difference according to the aliasing value.

Description

Technical Field

[0001] The disclosed technology and embodiments generally relate to an image sensing device capable of detecting a distance to a target object, an image processing device, and an imaging device including the image sensing device and the image processing device. Background Art

[0002] Image sensing devices are devices used to capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communication industries, the demand for high-performance image sensing devices is growing in various devices such as smartphones, wearable devices, digital cameras, game consoles, IOT (Internet of Things), robots, security cameras, and medical micro cameras.

[0003] Recently, image sensing devices have been actively used not only to acquire color images but also to sense the distance to a target object to be captured. In particular, a time-of-flight (ToF) method that directly or indirectly measures the duration in which light is reflected from a target object and returns to an image sensing device has been widely used. Summary of the invention

[0004] Various embodiments of the present disclosure relate to an image sensing device, an image processing device, and an imaging device including the image sensing device and the image processing device, which are capable of improving the accuracy of distance measurement.

[0005] According to an embodiment of the present disclosure, an image sensing device may include: a phase difference calculator, which is configured to calculate the phase difference between a modulated light signal and a reflected light based on a plurality of captured data segments generated by a plurality of modulation control signals, each modulation control signal having a predetermined modulation phase difference relative to the modulated light signal; a phase difference corrector, which is configured to calculate a contrast as a ratio of an amplitude component of the reflected light to an intensity component of the reflected light using the plurality of captured data segments, and is configured to determine an aliasing value corresponding to the contrast; and a distance calculator, which is configured to calculate the distance to a target object using a corrected phase difference obtained by correcting the phase difference according to the aliasing value.

[0006] According to another embodiment of the present disclosure, an imaging device may include: an image sensing device, which is configured to generate a plurality of captured data segments using a plurality of modulation control signals, each modulation control signal having a predetermined modulation phase difference relative to a modulated light signal; and an image processing device, which is configured to calculate a contrast as a ratio of an amplitude component of a reflected light to an intensity component of the reflected light using the plurality of captured data segments, and is configured to calculate a distance to a target object using a corrected phase difference obtained by correcting a phase difference between the modulated light signal and the reflected light according to an aliasing value corresponding to the contrast.

[0007] According to another embodiment of the present disclosure, an image sensing device may include: a pixel configured to generate a plurality of pixel signals by sensing reflected light using a plurality of modulation control signals each having a predetermined modulation phase difference relative to a modulated light signal during an exposure period; and a readout circuit configured to generate a plurality of captured data segments by processing the pixel signals. The exposure period includes: a first portion in which the plurality of modulation control signals maintain a logic high level; and a second portion in which the plurality of modulation control signals alternately have a logic high level.

[0008] It is to be understood that both the foregoing general description and the following detailed description of the disclosed technology are illustrative and descriptive and are intended to provide further description of embodiments of the disclosure as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above-described and other features and advantageous aspects of the disclosed technology will become apparent with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0010] Figure 1 The block diagram is a diagram showing an imaging device according to some embodiments of the present disclosure.

[0011] Figure 2 It is a diagram showing some embodiments of the present disclosure including Figure 1 Circuit diagram of a pixel in a pixel array.

[0012] Figure 3 is a schematic diagram showing the structure of a depth frame capable of acquiring depth data based on some embodiments of the present disclosure.

[0013] Figure 4A is a timing diagram showing a modulated optical signal and a modulated control signal for a first microframe according to some embodiments of the present disclosure.

[0014] Figure 4B is a timing diagram showing a modulated optical signal and a modulated control signal used in a second microframe according to some embodiments of the present disclosure.

[0015] Figure 4C is a timing diagram showing a modulated optical signal and a modulated control signal used in a third microframe according to some embodiments of the present disclosure.

[0016] Figure 4D is a timing diagram showing a modulated optical signal and a modulated control signal used in a fourth microframe according to some embodiments of the present disclosure.

[0017] Figure 5 is a graph showing the relationship between the distance to a target object and the contrast of reflected light according to some embodiments of the present disclosure.

[0018] Figure 6 is a graph showing the relationship between the phase difference and the contrast of the reflected light according to some embodiments of the present disclosure.

[0019] Figure 7 is a timing diagram illustrating a method for removing a component caused by ambient light from the intensity of reflected light according to some embodiments of the present disclosure.

[0020] FIG. 8A to FIG. 9 is a timing diagram illustrating another method for removing a component caused by ambient light from the intensity of reflected light according to some embodiments of the present disclosure.

[0021] Fig.10 It is a diagram showing some embodiments of the present disclosure. Figure 1 A block diagram of a computing device 1000 corresponding to an image processing device. DETAILED DESCRIPTION

[0022] The present disclosure provides embodiments and examples of image sensing devices, image processing devices, and imaging devices including image sensing devices and image processing devices that can detect the distance of a target object and can be used in various configurations to substantially solve one or more technical or engineering problems and alleviate limitations or disadvantages encountered in some other image sensing devices. Some embodiments of the present disclosure relate to image sensing devices, image processing devices, and imaging devices including image sensing devices and image processing devices that can improve the accuracy of distance measurement. The present disclosure provides various embodiments of image sensing devices that can more accurately obtain the distance to a target object to be captured by correcting the phase difference calculated by the time of flight (ToF) method.

[0023] Reference will now be made in detail to the embodiments of the present disclosure illustrated in the accompanying drawings. Where possible, the same reference numerals will be used throughout the accompanying drawings to refer to the same or similar components. Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments of the present disclosure are illustrated in the accompanying drawings by way of example. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0024] In the following, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the present disclosure can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0025] Figure 1 is a block diagram illustrating an imaging device ID according to some embodiments of the present disclosure.

[0026] refer to Figure 1, the imaging device (ID) may refer to a device such as a digital still camera for capturing still images or a digital video camcorder for capturing moving images. For example, the imaging device (ID) may be implemented as a digital single-lens reflex (DSLR) camera, a mirrorless camera, a smart phone, etc. The imaging device (ID) may include a device having both a lens and an image pickup element, so that the device can capture (or capture) a target object and thus can create an image of the target object. In some embodiments, the imaging device (ID) may be implemented as a lidar sensor.

[0027] The imaging device (ID) may include an image sensing device 100 and an image processing device 200 .

[0028] The image sensing device 100 may be a complementary metal oxide semiconductor image sensor (CIS) for converting incident light into an electrical signal and may include a light source 10, a lens module 20, a light source driver 30, a pixel array 110, a sensor driver 120, a readout circuit 130, and a timing controller 140.

[0029] The light source 10 can emit light to the target object 1 when receiving a modulated light signal (MLS) from the light source driver 30. The light source 10 can be a laser diode (LD) or a light emitting diode (LED) for emitting light having a specific wavelength band (for example, near infrared (NIR) light, infrared (IR) light, or visible light), or can be any one of a near infrared laser (NIR), a point light source, a monochromatic light source combined with a white light lamp or a monochromator, and a combination of other laser sources. For example, the light source 10 can be implemented as light in an infrared wavelength band with a wavelength of 800nm ​​to 1000nm (hereinafter referred to as "infrared light"), and the light source 10 of the disclosed technology can emit infrared light with a wavelength of 800nm ​​to 1000nm. The light emitted from the light source 10 can be a pulsed light having a predetermined period, amplitude, and pulse width. Although for ease of description, Figure 1 Only one light source 10 is shown, but the scope or spirit of the present disclosure is not limited thereto, and a plurality of light sources may also be arranged near the lens module 20 .

[0030] The lens module 20 may collect light reflected from the target object 1 and may allow the collected light to be focused on the pixel (PX) of the pixel array 110. For example, the lens module 20 may include a focusing lens having a surface formed of glass or plastic or another cylindrical optical element having a surface formed of glass or plastic. The lens module 20 may include a plurality of lenses arranged to focus on an optical axis.

[0031] The light source driver 30 may drive the light source 10 under the control of the timing controller 140. In particular, the light source driver 30 may control the waveform (eg, frequency, period, amplitude, pulse width, etc.) of the emission light (EL) output from the light source 10 using a modulated light signal (MLS).

[0032] The pixel array 110 may include a plurality of pixels (PX) continuously arranged in a two-dimensional (2D) matrix structure (e.g., continuously arranged in a column direction and / or a row direction). Each of the plurality of pixels (PX) may generate a pixel signal by sensing incident light incident through the lens module 20 under the control of the sensor driver 120.

[0033] Each pixel (PX) may be an infrared pixel for generating a pixel signal by sensing incident light, the incident light including reflected light (RL) generated when the emission light (EL) from the light source 10 is reflected from the target object 1. According to one embodiment, the infrared pixel may be a depth pixel for calculating the distance to the target object 1. According to another embodiment, the infrared pixel may include a pixel for generating an infrared image by simply sensing infrared light incident from a scene without sensing reflected light. According to yet another embodiment, the pixel (PX) may include a pixel for generating a color image by sensing visible light incident from a scene. Hereinafter, each pixel (PX) will be described as a two-tap (2-tap) pixel for detecting the distance to the target object 1 according to an indirect time of flight (ToF) method. Reference will be made to the description from Figure 2 The accompanying drawings describe a more detailed structure and operation of each unit pixel (PX).

[0034] The sensor driver 120 may drive the pixels (PX) of the pixel array 110 in response to the timing signals output from the timing controller 140. For example, the sensor driver 120 may generate control signals capable of selecting and controlling the pixels (PX) included in at least one row line among a plurality of row lines of the pixel array 110.

[0035] The readout circuit 130 may process the pixel signal received from the pixel array 110 under the control of the timing controller 140, and may generate and store image data (IDATA) for detecting the distance of the target object 1. The image data (IDATA) may be digital data obtained by analog-to-digital conversion (ADC) of an analog pixel signal. To this end, the readout circuit 130 may include a correlated double sampler (CDS) circuit for performing correlated double sampling (CDS) on the pixel signal generated from the pixel array 110. In addition, the readout circuit 130 may include an analog-to-digital converter (ADC) for converting the output signal of the CDS circuit into a digital signal. In addition, the readout circuit 130 may include a buffer circuit that temporarily stores the pixel data generated from the analog-to-digital converter (ADC) and outputs the pixel data under the control of the timing controller 140. In addition, two column lines for transmitting pixel signals may be allocated to each column of the pixel array 110, and a structure for processing the pixel signal generated from each column line may be configured to correspond to the corresponding column line.

[0036] The timing controller 140 may generate a timing signal to control the light source driver 30, the sensor driver 120, and the readout circuit 130. In some embodiments, the timing controller 140 may generate a timing signal according to a request or a predetermined value received from the image processing device 200. In some embodiments, the timing controller 140 may include a logic control circuit, a phase-locked loop (PLL) circuit, a timing control circuit, a communication interface circuit, and the like.

[0037] The image processing device 200 may perform at least one image signal processing on the image data (IDATA) received from the image sensing device 100, and may thus generate processed image data. The image processing device 200 may reduce noise of the image data (IDATA), and may perform various types of image signal processing (e.g., interpolation of the image data (IDATA), lens distortion correction, etc.) for image quality improvement of the image data.

[0038] The image processing apparatus 200 may include a phase difference calculator 210, a phase difference corrector 220, and a distance calculator 230. The image processing apparatus 200 may be implemented in hardware, software, or a combination thereof.

[0039] The phase difference calculator 210 may calculate the phase difference between the emitted light (EL) and the reflected light (RL) based on the image data (IDATA). Depending on the distance between the image sensing device 100 and the target object 1, there is a time delay between the emitted light (EL) and the reflected light (RL), and the time delay may be expressed as a phase difference between the emitted light (EL) and the reflected light (RL). In some embodiments, there is no time delay between the modulated light signal (MLS) and the emitted light (EL). That is, the phase difference between the emitted light (EL) and the reflected light (RL) may be the same as the phase difference between the modulated light signal (MLS) and the reflected light (RL).

[0040] The phase difference corrector 220 may correct the phase difference using the image data (IDATA). The emitted light (EL) may be modulated at a specific frequency so that the emitted light (EL) may have periodicity. Due to this periodicity, the image data obtained by detecting the target object 1 located at different distances from the image sensing device 100 may show the same phase difference. The phase difference corrector 220 may generate a corrected phase difference by correcting the phase difference so that the phase difference indicates the distance to the target object 1.

[0041] In addition, the phase difference corrector 220 may correct the phase difference by removing the ambient light component as much as possible to prevent the effect of phase difference correction from being deteriorated due to the ambient light component.

[0042] The distance calculator 230 may calculate the distance to the target object 1 using the corrected phase difference. The distance to the target object 1 may be calculated for each pixel of the pixel array 110 .

[0043] In the following description, a process in which the image processing apparatus 200 calculates the distance to the target object 1 based on one pixel will be described with reference to the accompanying drawings, but a method substantially the same as the above method may also be applied to other pixels of the pixel array 110 .

[0044] Figure 2 It is shown that some embodiments of the present disclosure are included in Figure 1 1 is a circuit diagram of a pixel in the pixel array 110.

[0045] refer to Figure 2 , the pixel (PX) may correspond to one instance of each pixel (PX) included in the pixel array 110 .

[0046] The pixel (PX) may include a photoelectric conversion element (PD), a first transfer transistor and a second transfer transistor (TX1, TX2), a first reset transistor and a second reset transistor (RX1, RX2), a first drive transistor and a second drive transistor (DX1, DX2), and a first selection transistor and a second selection transistor (SX1, SX2).

[0047] The pixel (PX) may be a 2-tap pixel, comprising a first tap driven by a first modulation control signal (MCS1) having a predetermined modulation phase difference with the modulated light signal (MLS), and a second tap driven by a second modulation control signal (MCS2) having an opposite phase to the first modulation control signal (MCS1). The first tap may include a first transfer transistor (TX1), a first reset transistor (RX1), a first drive transistor (DX1), and a first selection transistor (SX1). In addition, the second tap may include a second transfer transistor (TX2), a second reset transistor (RX2), a second drive transistor (DX2), and a second selection transistor (SX2). In some embodiments, the modulation phase difference may refer to a phase difference between the modulated light signal (MLS) and the modulation control signal (MCS1, MCS2), and the phase difference may refer to a phase difference between the modulated light signal (MLS) and the reflected light (RL).

[0048] Each of the modulation control signals (MCS1, MCS2) may have a predetermined modulation phase difference (eg, any one of 0 degrees, 90 degrees, 180 degrees, or 270 degrees) with the modulated optical signal (MLS).

[0049] The photoelectric conversion element (PD) can generate and accumulate photoelectric charges corresponding to the intensity of the incident light by photoelectric conversion of the incident light. For example, the photoelectric conversion element (PD) can be implemented as at least one of a photodiode, a clamped photodiode, and a phototransistor, but the scope of the present disclosure is not limited thereto.

[0050] First, the structure and operation of the first tap will be described below with reference to the accompanying drawings. The first transfer transistor (TX1) can be connected between the photoelectric conversion element (PD) and the first floating diffusion area (FD1), and can be turned on or off in response to the first modulation control signal (MCS1). The turned-on first transfer transistor TX1 can transfer the photocharge accumulated in the photoelectric conversion element (PD) to the first floating diffusion area (FD1). The first floating diffusion area (FD1) may have a capacitance sufficient to accumulate the photocharge generated by the photoelectric conversion element (PD), and may be implemented as a junction capacitor, but the scope of the present disclosure is not limited thereto.

[0051] The first reset transistor (RX1) can be connected between the power supply voltage (VDD) and the first floating diffusion area FD1, and can be turned on or off in response to the first reset control signal (RG1). The turned-on first reset transistor (RX1) can discharge the photocharge accumulated in the first floating diffusion area (FD1) by resetting the first floating diffusion area (FD1) to the power supply voltage (VDD).

[0052] The first drive transistor (DX1) may be connected between a power supply voltage (VDD) and the first selection transistor (SX1), and may generate an electrical signal corresponding to a voltage of the first floating diffusion area (FD1).

[0053] The first selection transistor (SX1) may be connected between the first drive transistor (DX1) and the first column line (COL1), and may be turned on or off in response to a first selection control signal (SEL1). The turned-on first selection transistor (SX1) may output an electrical signal output from the first drive transistor (DX1) through the first column line (COL1).

[0054] Can be obtained from Figure 1 The sensor driver 120 shown provides control signals ( MCS1 , RG1 , SEL1 ) for controlling the first tap.

[0055] Although the structure and operation of the first tap have been described above, the structure and operation of the second tap may also be substantially the same as the structure and operation of the first tap, and therefore, for the sake of brevity, its redundant description will be omitted herein. However, the waveform of the control signal (MCS2, RG2, SEL2) for controlling the second tap may be different from the waveform of the control signal (MCS1, RG1, SEL1) for controlling the first tap. Specifically, the second modulation control signal (MCS2) may have a phase opposite to that of the first modulation control signal (MCS1).

[0056] Figure 3 is a schematic diagram showing the structure of a depth frame capable of acquiring depth data based on some embodiments of the present disclosure.

[0057] refer to Figure 3 , the image sensing device 100 may operate in units of depth frames to acquire depth data indicating a distance to the target object 1. That is, the image sensing device 100 may acquire depth data after performing an operation corresponding to a depth frame.

[0058] The depth frame may include a first subframe to a fourth subframe (SF1-SF4). The first modulation control signal (MCS1) may have different phases in the first subframe to the fourth subframe (SF1-SF4). In addition, the second modulation control signal (MCS2) having a phase opposite to that of the first modulation control signal (MCS1) may also have different phases in the first subframe to the fourth subframe (SF1-SF4).

[0059] Each of the first to fourth subframes (SF1 ˜ SF4 ) may include an exposure period and a readout period.

[0060] The exposure period may refer to a time period during which photocharges are generated by a pixel that senses reflected light (RL) obtained when the emitted light (EL) output from the image sensing device 100 according to the modulated light signal (MLS) is reflected from the target object 1, and the generated photocharges are accumulated in components (e.g., FD1, FD2) located inside the pixel (PX).

[0061] The exposure period may include at least one microframe. A microframe may be a unit constituting an exposure period, and will be referred to later. 4A to 4D A more detailed description of microframes is given.

[0062] The exposure period may include a plurality of microframes to ensure a sufficient amount of light, and the number of microframes included in one exposure period may be predetermined by experiment. In some other implementations, the number of microframes included in one exposure period may vary according to lighting conditions.

[0063] The exposure period of the first subframe (SF1) may include a plurality of consecutive first microframes (MF1), and the exposure period of the second subframe (SF2) may include a plurality of consecutive second microframes (MF2). In addition, the exposure period of the third subframe (SF3) may include a plurality of consecutive third microframes (MF3), and the exposure period of the fourth subframe (SF4) may include a plurality of consecutive fourth microframes (MF4).

[0064] The readout period may refer to a time period in which the readout circuit 130 generates image data (IDATA) based on an electrical signal corresponding to photocharges accumulated in the pixel (PX) and transmits the image data (IDATA) to the image processing device 200 .

[0065] Figure 4A is a timing diagram showing a modulated optical signal and a modulated control signal used in a first microframe ( MF1 ) according to some embodiments of the present disclosure.

[0066] Reference Figure 4A ,constitute Figure 3 The first micro frame (MF1) of the exposure period of the first subframe (SF1) may include a first portion P1 and a second portion P2.

[0067] In the first microframe (MF1), the modulated light signal (MLS) may be a square wave signal alternately having a logic high level (H) and a logic low level (L). The light source 10 may output emission light (EL) in response to the modulated light signal (MLS) of a logic high level (H), and may not output emission light (EL) in response to the modulated light signal (MLS) of a logic low level (L). Therefore, the waveform of the emission light EL may be the same as the waveform of the modulated light signal (MLS). In the following description, the modulated light signal (MLS) and the emission light (EL) may be used interchangeably.

[0068] The modulated light signal (MLS) may have a predetermined modulation frequency (F) and a period (T) corresponding to 1 / F. Since the pulse of the modulated light signal (MLS) has a periodicity indicating that the modulated light signal (MLS) is repeated every period (T), the phase difference between the modulated light signal (MLS) and the reflected light (RL) may also have a periodicity. In the indirect TOF method, the distance to the target object 1 may be estimated based on the phase difference between the modulated light signal (MLS) and the reflected light (RL). Even when the distance to the target object 1 is different, the same phase difference may occur due to such periodicity of the phase difference.

[0069] The maximum measuring distance (D) depends on the modulation frequency (F) of the modulated light signal (MLS) freq ) can be calculated by the following formula 1.

[0070] [Formula 1]

[0071]

[0072] In Formula 1, 'c' may represent the speed of light (luminous flux).

[0073] For example, when the modulation frequency (F) of the modulated optical signal (MLS) is 100 MHz, the period (T) of the modulated optical signal (MLS) may be 10 ns. In this case, the maximum measurement distance (D) depends on the modulation frequency (F). freq ) may be 1.5 m. That is, the first phase difference corresponding to the example case where the distance to the target object 1 is 0.1 m may be the same as the second phase difference corresponding to the example case where the distance to the target object 1 is 1.6 m, and the first phase difference and the second phase difference may not be distinguished from each other. The above phenomenon may be defined as "aliasing". In some embodiments, this aliasing may be overcome by using the contrast of the image data (IDATA). This will be referred to later. Figure 5 and Figure 6 A more detailed description of the aliasing operation is given. The period (T) may refer to the time period during which a pulse repeats, and may also be referred to as a pulse period.

[0074] The first part P1 of the first microframe (MF1) may have a length of (N×T) (where N is an integer of 1 or greater). In the first part P1 of the first microframe (MF1), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) may be maintained at a logic high level (H). The first transfer transistor (TX1) receiving the first modulation control signal (MCS1) of a logic high level and the second transfer transistor (TX2) receiving the second modulation control signal (MCS2) of a logic high level may be turned on. The photocharges generated in the first part P1 may be equally distributed to and accumulated in the first tap and the second tap. Here, N may be a value satisfying the condition of the following formula 2.

[0075] [Formula 2]

[0076]

[0077] In Formula 2, the target measurement distance (D max ) may refer to the maximum measurement distance to be measured by the imaging device (ID) without aliasing. That is, N may be greater than or equal to the maximum measurement distance (D max ) divided by the maximum measuring distance (D freq ) is the integer obtained by rounding up the value (ceiling value).

[0078] For example, when a distance of up to 6 meters is to be measured without aliasing by using a modulated light signal (MLS) having a modulation frequency of 100 MHz, N may be an integer equal to or greater than 6 / (1.5)=4.

[0079] In addition, the number of pulses (K) of the modulated light signal (MLS) can also be an integer equal to or greater than (N+1), so that the phase difference between the modulated light signal (MLS) and the reflected light (RL) can be detected normally even when the distance to the target object 1 is relatively short.

[0080] The second part P2 of the first microframe (MF1) may have a length of (M×T), where M is an integer of 1 or greater. In the second part P2 of the first microframe (MF1), the first modulation control signal (MCS1) may have the same phase as the modulated light signal (MLS), and the second modulation control signal (MCS2) may have a modulation phase difference of 180 degrees (i.e., opposite phase) relative to the phase of the modulated light signal (MLS). Therefore, according to the time delay between the modulated light signal (MLS) and the reflected light (RL), the photocharge generated in the second part P2 may be distributed to and accumulated in the first tap and the second tap at a preset ratio.

[0081] Here, M may be a value satisfying the condition of the following Formula 3.

[0082] [Formula 3]

[0083] M≥K+1-N

[0084] When Formula 3 is generalized for 'K', 'K' may be a value equal to or less than '(N+M)-1'. That is, the number of pulses (K) may be less than (N+M) obtained by dividing the total duration '(N+M)×T' of the first microframe (MF1) by the period 'T'.

[0085] Once the number of pulses (K) of the modulated light signal (MLS), the number of cycles (N) of the first part (P1), and the number of cycles (M) of the second part (P2) are determined, the target measurement distance ( Dmax ).

[0086] [Formula 4]

[0087] D max =(N+ML-1)·D freq

[0088] In the first microframe (MF1), the first tap may accumulate photocharges captured when the first modulation control signal (MCS1) has a logic high level (H). In addition, in the first microframe (MF1), the second tap may accumulate photocharges captured when the second modulation control signal (MCS2) has a logic high level (H).

[0089] In a readout period of a first subframe (SF1), a first tap may output an electrical signal corresponding to photocharges accumulated in a plurality of first microframes (MF1). A data segment obtained by converting the electrical signal output from the first tap may be defined as a first captured data segment (S0) of the first tap. The first captured data segment (S0) of the first tap may include a component of photocharges captured by a first modulation control signal (MCS1) having a modulation phase difference of 0 degrees (i.e., in phase) relative to a modulated light signal (MLS).

[0090] In the readout period of the first subframe (SF1), the second tap may output an electrical signal corresponding to the photocharge accumulated in the plurality of first microframes (MF1). The data segment obtained by converting the electrical signal output from the second tap may be defined as a third captured data segment (S180) of the second tap. The third captured data segment (S180) of the second tap may include a component for the photocharge captured by the second modulation control signal (MCS2) having a modulation phase difference of 180 degrees relative to the modulated light signal (MLS).

[0091] Figure 4Bis a timing diagram showing a modulated optical signal and a modulated control signal used in a second microframe (MF2) according to some embodiments of the present disclosure.

[0092] Reference Figure 4B ,constitute Figure 3 The second microframe ( MF2 ) of the exposure period of the illustrated second subframe ( SF2 ) may include a first portion P1 and a second portion P2 .

[0093] Besides some differences, Figure 4B The modulated optical signal (MLS) of the second microframe (MF2), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) can be Figure 4A The modulated optical signal (MLS) of the first microframe (MF1), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) are substantially the same, and therefore, for the sake of brevity, redundant descriptions thereof will be omitted herein.

[0094] The first modulation control signal (MCS1) of the second microframe (MF2) may have a modulation phase difference of 90 degrees relative to the modulated optical signal (MLS) in the second portion P2. In addition, the second modulation control signal (MCS2) of the second microframe (MF2) may have a modulation phase difference of 270 degrees relative to the modulated optical signal (MLS) in the second portion P2.

[0095] In addition, the number of pulses (K) of the modulated light signal (MLS) in the second microframe (MF2), the number of cycles (N) in the first part (P1) of the second microframe (MF2), the number of cycles (M) in the second part (P2) of the second microframe (MF2), and the target measurement distance (D) in the second microframe (MF2) are max ) may be the same as those of the first microframe (MF1).

[0096] In the second microframe (MF2), the first tap may accumulate photocharges captured when the first modulation control signal (MCS1) has a logic high level (H). In addition, in the second microframe (MF2), the second tap may accumulate photocharges captured when the second modulation control signal (MCS2) has a logic high level (H).

[0097] In the readout period of the second subframe (SF2), the first tap may output an electrical signal corresponding to the photocharge accumulated in the plurality of second microframes (MF2). The data segment obtained by converting the electrical signal output from the first tap may be defined as a second captured data segment S90 of the first tap. The second captured data segment S90 of the first tap may include a component for the photocharge captured by the first modulation control signal (MCS1) having a modulation phase difference of 90 degrees relative to the modulated light signal (MLS).

[0098] In the readout period of the second subframe (SF2), the second tap may output an electrical signal corresponding to the photocharge accumulated in the plurality of second microframes (MF2). The data segment obtained by converting the electrical signal output from the second tap may be defined as a fourth captured data segment S270 of the second tap. The fourth captured data segment S270 of the second tap may include a component for the photocharge captured by the second modulation control signal (MCS2) having a modulation phase difference of 270 degrees relative to the modulated light signal (MLS).

[0099] Figure 4C is a timing diagram showing a modulated optical signal and a modulated control signal used in a third microframe MF3 according to some embodiments of the present disclosure.

[0100] Reference Figure 4C ,constitute Figure 3 The third micro frame (MF3) of the exposure period of the illustrated third subframe SF3 may include a first portion P1 and a second portion P2.

[0101] Besides some differences, Figure 4C The modulated optical signal (MLS) of the third microframe (MF3), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) can be Figure 4A The modulated optical signal (MLS) of the first microframe (MF1), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) are substantially the same, and therefore, for the sake of brevity, redundant descriptions thereof will be omitted herein.

[0102] The first modulation control signal (MCS1) of the third microframe (MF3) may have a modulation phase difference of 180 degrees relative to the modulated optical signal (MLS) in the second portion P2. In addition, the second modulation control signal (MCS2) of the third microframe (MF3) may have a modulation phase difference of 0 degrees relative to the modulated optical signal (MLS) in the second portion P2.

[0103] In addition, the number of pulses (K) of the modulated light signal (MLS) in the third microframe (MF3), the number of cycles (N) in the first part (P1) of the third microframe (MF3), the number of cycles (M) in the second part (P2) of the third microframe (MF3), and the target measurement distance (D) in the third microframe (MF3) are max ) may be the same as those of the first microframe (MF1).

[0104] In the third microframe (MF3), the first tap may accumulate photocharges captured when the first modulation control signal (MCS1) has a logic high level (H). In addition, in the third microframe (MF3), the second tap may accumulate photocharges captured when the second modulation control signal (MCS2) has a logic high level (H).

[0105] In the readout period of the third subframe (SF3), the first tap may output an electrical signal corresponding to the photocharge accumulated in the plurality of third microframes (MF3). The data segment obtained by converting the electrical signal output from the first tap may be defined as a third captured data segment S180 of the first tap. The third captured data segment S180 of the first tap may include a component for the photocharge captured by the first modulation control signal (MCS1) having a modulation phase difference of 180 degrees relative to the modulated light signal (MLS).

[0106] In the readout period of the third subframe (SF3), the second tap may output an electrical signal corresponding to the photocharge accumulated in the plurality of third microframes (MF3). The data segment obtained by converting the electrical signal output from the second tap may be defined as a first captured data segment S0 of the second tap. The first captured data segment S0 of the second tap may include a component for the photocharge captured by the second modulation control signal (MCS2) having a modulation phase difference of 0 degrees relative to the modulated light signal (MLS).

[0107] Figure 4D is a timing diagram showing a modulated optical signal and a modulated control signal for use in a fourth microframe (MF4) according to some embodiments of the present disclosure.

[0108] Reference Figure 4D ,constitute Figure 3 The fourth microframe ( MF4 ) of the exposure period of the illustrated fourth subframe ( SF4 ) may include a first portion P1 and a second portion P2 .

[0109] Besides some differences, Figure 4D The modulated optical signal (MLS) of the fourth microframe (MF4), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) can be Figure 4A The modulated optical signal (MLS) of the first microframe (MF1), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) are substantially the same, and therefore, for the sake of brevity, redundant descriptions thereof will be omitted herein.

[0110] The first modulation control signal (MCS1) of the fourth microframe (MF4) may have a modulation phase difference of 270 degrees relative to the modulated optical signal (MLS) in the second portion P2. In addition, the second modulation control signal (MCS2) of the fourth microframe (MF4) may have a modulation phase difference of 90 degrees relative to the modulated optical signal (MLS) in the second portion P2.

[0111] In addition, the number of pulses (K) of the modulated light signal (MLS) in the fourth microframe (MF4), the number of cycles (N) in the first part (P1) of the fourth microframe (MF4), the number of cycles (M) in the second part (P2) of the fourth microframe (MF4), and the target measurement distance (D) in the fourth microframe (MF4) are max ) may be the same as those of the first microframe (MF1).

[0112] In the fourth microframe (MF4), the first tap may accumulate photocharges captured when the first modulation control signal (MCS1) has a logic high level (H). In addition, in the fourth microframe (MF4), the second tap may accumulate photocharges captured when the second modulation control signal (MCS2) has a logic high level (H).

[0113] In the readout period of the fourth subframe (SF4), the first tap may output an electrical signal corresponding to the photocharge accumulated in the plurality of fourth microframes (MF4). A data segment obtained by converting the electrical signal output from the first tap may be defined as a fourth captured data segment S270 of the first tap. The fourth captured data segment S270 of the first tap may include a component for the photocharge captured by the first modulation control signal (MCS1) having a modulation phase difference of 270 degrees relative to the modulated light signal (MLS).

[0114] In the readout period of the fourth subframe (SF4), the second tap may output an electrical signal corresponding to the photocharge accumulated in the plurality of fourth microframes (MF4). The data segment obtained by converting the electrical signal output from the second tap may be defined as a second captured data segment S90 of the second tap. The second captured data segment S90 of the second tap may include a component for the photocharge captured by the second modulation control signal (MCS2) having a modulation phase difference of 90 degrees relative to the modulated light signal (MLS).

[0115] like FIG. 4A to FIG. 4DAs described in, the first to fourth captured data segments (S0 to S270) of each of the first tap and the second tap can be obtained from the first subframe to the fourth subframe (SF1 to SF4). In order to calculate the distance to the target object 1, the first to fourth captured data segments (S0 to S270) may be required. The first to fourth captured data segments (S0 to S270) can be obtained by the first tap and the second tap in the first subframe and the second subframe (SF1, SF2), but due to the difference in hardware characteristics between the first tap and the second tap, the first tap and the second tap may output different signals even under the same operating conditions. Therefore, in some embodiments, the first to fourth captured data segments (S0 to S270) can be obtained from each of the first tap and the second tap in units of depth frames through the first subframe to the fourth subframe (SF1 to SF4), so that the noise caused by the characteristic difference between the taps can be prevented from affecting the process of calculating the distance to the target object 1.

[0116] The phase difference calculator 210 of the image processing device 200 may sum the first captured data segment S0 of the first tap and the first captured data segment S0 of the second tap to generate the first captured data segment S0 of the corresponding depth frame. In addition, the phase difference calculator 210 may generate the second captured data segment S90 of the corresponding depth frame by summing the second captured data segment S90 of the first tap and the second captured data segment S90 of the second tap. The phase difference calculator 210 may generate the third captured data segment S180 of the corresponding depth frame by summing the third captured data segment S180 of the first tap and the third captured data segment S180 of the second tap. Similarly, the phase difference calculator 210 may sum the fourth captured data segment S270 of the first tap and the fourth captured data segment S270 of the second tap to generate the fourth captured data segment S270 of the corresponding depth frame.

[0117] The phase difference calculator 210 may calculate a phase difference (PDIF) between a modulated light signal (MLS) (or emission light EL) and a reflected light (RL) based on first to fourth capture data segments (S0-S270) of a corresponding depth frame using Formula 5 below.

[0118] [Formula 5]

[0119]

[0120] In principle, the distance to the target object 1 should be accurately calculated based on the phase difference (PDIF). However, in the case of the target measurement distance (D max ) exceeds the maximum measuring distance (D freq), due to the periodicity of the modulated light signal (MLS), the same phase difference (PDIF) may appear even when the target object 1 is located at different distances from the image sensing device. Therefore, in some embodiments, a method of more accurately calculating the distance to the target object 1 may be used using a correction value based on the phase difference (PDIF) as needed instead of using the phase difference (PDIF) without change.

[0121] Figure 5 is a graph showing the relationship between the distance to a target object and the contrast of reflected light (RL) according to some embodiments of the present disclosure.

[0122] refer to Figure 5 , showing simulation results of the relationship between the contrast of the reflected light (RL) and the distance to the target object 1. Here, the contrast of the reflected light (RL) may refer to a ratio between the amplitude of the reflected light (RL) and the intensity of the reflected light (RL).

[0123] The magnitude of the reflected light (RL) can be calculated by the following formula 6.

[0124] [Formula 6]

[0125]

[0126] In addition, the intensity of the reflected light (RL) can be calculated by the following formula 7.

[0127] [Formula 7]

[0128]

[0129] In Formula 6 and Formula 7, S0, S90, S180, and S270 may respectively refer to a first captured data segment, a second captured data segment, a third captured data segment, and a fourth captured data segment of a corresponding depth frame.

[0130] Therefore, the contrast of the reflected light RL can be calculated by the following formula 8.

[0131] [Formula 8]

[0132]

[0133] In Formula 8, it is shown that the ratio between the magnitude component and the intensity component is multiplied by a coefficient of 2. However, since the ratio between the magnitude component and the intensity component is important, the coefficient may be set to any value except '0'.

[0134] Figure 5The simulation results show that when the modulation frequency (F) is 100 MHz, the number of cycles (N) of the first part P1 is 4, the number of pulses (K) of the modulated light signal (MLS) is 10, the number of cycles (M) of the second part P2 is 15, and the target measurement distance (D max ) is 6 meters (m). In addition, since the modulation frequency (F) is 100MHz, the period (T) can be 10ns, and the maximum measurement distance (D freq ) can be 1.5 meters (m).

[0135] exist Figure 5 As the distance to the target object 1 increases, the contrast of the reflected light (RL) gradually increases until it is used as the target measurement distance (D max ), then gradually decreases until 12 meters (m), remains constant until 24 meters (m), and then increases again from 24 meters (m).

[0136] Figure 6 is a graph showing the relationship between the contrast and phase difference of reflected light according to some embodiments of the present disclosure.

[0137] refer to Figure 6 , shows the relationship between the phase difference (PDIF) and the contrast when the distance to the target object 1 increases from 0 m to 6 m as the target measurement distance (Dmax).

[0138] As the distance to the target object 1 increases from 0m to 1.5m, the phase difference (PDIF) may increase from 0[rad] to 2π[rad], and the contrast may also gradually increase as the phase difference (PDIF) increases. As the distance to the target object 1 increases from 0m to 1.5m, the relationship between the phase difference (PDIF) and the contrast may be represented by a first measurement value (MV1). That is, the first measurement value (MV1) may be a set (or aggregation) of points representing the relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 0m to 1.5m, and may approximate a first straight line (AL1) having a first slope and a first intercept C1. A method of approximating a set of points to a single straight line may be a method of determining a virtual straight line that minimizes the sum of squares of errors of points relative to a virtual straight line, but the scope of the present disclosure is not limited thereto.

[0139] As the distance to the target object 1 increases from 1.5 m to 3 m, the phase difference (PDIF) may increase from 0 [rad] to 2π [rad], and the contrast may also gradually increase as the phase difference (PDIF) increases. The relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 1.5 m to 3 m may be represented by a second measurement value (MV2). That is, the second measurement value (MV2) may be a set (or aggregation) of points representing the relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 1.5 m to 3 m, and may approximate a second straight line (AL2) having a first slope and a second intercept C2.

[0140] As the distance to the target object 1 increases from 3m to 4.5m, the phase difference (PDIF) may increase from 0 [rad] to 2π [rad], and the contrast may also gradually increase as the phase difference (PDIF) increases. The relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 3m to 4.5m may be represented by a third measurement value (MV3). That is, the third measurement value (MV3) may be a set (or aggregation) of points representing the relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 3m to 4.5m, and may approximate a third straight line (AL3) having a first slope and a third intercept C3.

[0141] As the distance to the target object 1 increases from 4.5 m to 6 m, the phase difference (PDIF) may increase from 0 [rad] to 2π [rad], and the contrast may also gradually increase as the phase difference (PDIF) increases. The relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 4.5 m to 6 m may be represented by a fourth measurement value (MV4). That is, the fourth measurement value (MV4) may be a set (or aggregation) of points representing the relationship between the phase difference (PDIF) and the contrast as the distance to the target object 1 increases from 4.5 m to 6 m, and may approximate a fourth straight line (AL4) having a first slope and a fourth intercept C4.

[0142] For example, in the case where the distance to the target object 1 is set to 0.75 m, 2.25 m, 3.75 m, and 5.25 m, the phase difference (PDIF) may all appear to be about π [rad] even when the distance to the target object 1 is set to different distances. However, when the distance to the target object 1 is set to 0.75 m, 2.25 m, 3.75 m, and 5.25 m, the contrast may appear different, so that the phase difference corrector 220 may correct the phase difference (PDIF) using such characteristics.

[0143] In some embodiments, the phase difference corrector 220 may correct the phase difference (PDIF) using a linear function characteristic of each of the first to fourth measurement values ​​(MV1-MV4).

[0144] The first straight line to the fourth straight line (AL1 to AL4) to which the first measurement value to the fourth measurement value (MV1 to MV4) are respectively approximated have the same first slope (A), but may have gradually increasing first intercepts to fourth intercepts (C1 to C4). The first slope (A) may refer to a ratio of an increase in the phase difference (PDIF) to an increase in the contrast, and may be a value determined in advance by experiments.

[0145] The phase difference corrector 220 may calculate a measurement intercept (B) corresponding to the currently calculated phase difference (PDIF) and contrast by using the first slope (A) of the first to fourth straight lines (AL1 ˜ AL4 ), as shown in Formula 9 below.

[0146] [Formula 9]

[0147] B=A·PDIF+contrast

[0148] The phase difference corrector 220 can be configured to correct the phase difference by comparing the measured intercept (B) with the first reference intercept to the third reference intercept (C). a1 ~C a3 ) is compared to calculate the aliasing value (R) corresponding to the measured intercept (B). Here, the first to fourth intercepts (C1-C4) can be used to calculate the first reference intercept to the third reference intercept (C a1 ~C a3 ), as shown in Formula 10 below.

[0149] [Formula 10]

[0150]

[0151] Specifically, the phase difference corrector 220 can determine the phase difference from the first reference intercept to the third reference intercept (C a1 ~C a3 ) includes the measurement intercept (B) to determine the aliasing value (R). The first straight line to the fourth straight line (AL1~AL4) may correspond to the first aliasing cycle to the fourth aliasing cycle, respectively. In each of the first aliasing cycle to the fourth aliasing cycle, the contrast may also increase as the phase difference (PDIF) increases. In each of the first aliasing cycle to the fourth aliasing cycle, the contrast corresponding to the phase difference (PDIF) may vary. The aliasing value (R) may indicate which of the first aliasing cycle to the fourth aliasing cycle includes the aliasing cycle corresponding to the phase difference (PDIF).

[0152] In the first aliasing period, as the distance to the target object 1 increases from 0m to 1.5m, the phase difference (PDIF) may increase from 0 to 2π. In the second aliasing period, as the distance to the target object 1 increases from 1.5m to 3m, the phase difference (PDIF) may increase from 0 to 2π. In the third aliasing period, as the distance to the target object 1 increases from 3m to 4.5m, the phase difference (PDIF) may increase from 0 to 2π. In the fourth aliasing period, as the distance to the target object 1 increases from 4.5m to 6m, the phase difference (PDIF) may increase from 0 to 2π. That is, the distance to the target object 1 indicated by the phase difference (PDIF) may vary depending on which of the first to fourth aliasing periods includes the aliasing period corresponding to the phase difference (PDIF).

[0153] The aliasing value (R) can be obtained by comparing the measured intercept (B) with the first reference intercept to the third reference intercept (C a1 ~C a3 ) is determined as shown in the following formula 11.

[0154] [Formula 11]

[0155] R=1(B <C a1 )

[0156] R=2(C a1 ≤B <C a2 )

[0157] R=3(C a2 ≤B <C a3 )

[0158] R=4(C a3 ≤B)

[0159] The aliasing value (R) may refer to a value indicating which straight line among the first to fourth straight lines (AL1-AL4) corresponds to the measurement intercept (B) corresponding to the currently calculated phase difference (PDIF) and contrast, and may indicate which aliasing cycle among the first to fourth aliasing cycles is the aliasing cycle corresponding to the phase difference (PDIF). For example, if the aliasing value R is set to '1', the measurement intercept (B) corresponds to the first straight line (AL1), and the aliasing cycle corresponding to the phase difference (PDIF) may be the first aliasing cycle.

[0160] The phase difference corrector 220 may calculate a corrected phase difference (PDIF) by correcting the phase difference (PDIF) using the aliasing value (R), as shown in Formula 12.

[0161] [Formula 12]

[0162] PDIF′=PDIF+(R-1)·2π

[0163] When the aliasing value (R) increases by 1, the phase difference corrector 220 may calculate the corrected phase difference (PDIF) by increasing the phase difference (PDIF) by 2π. In other words, (R-1)·2π may be defined as a phase correction value depending on the aliasing value (R), and the corrected phase difference (PDIF) may be a value obtained by summing the phase correction value and the phase difference (PDIF).

[0164] That is, the phase difference corrector 220 can calculate a measurement intercept (B) corresponding to the currently calculated phase difference (PDIF) and contrast, can determine an aliasing value (R) corresponding to the measurement intercept (B), and can calculate a corrected phase difference (PDIF) by correcting the phase difference (PDIF) using the aliasing value (R).

[0165] The distance calculator 230 may calculate the distance (Dr) to the target object 1 using the corrected phase difference (PDIF), as shown in Formula 13 below.

[0166] [Formula 13]

[0167]

[0168] In some embodiments, despite the aliasing phenomenon that is a limitation of the indirect TOF method, the aliasing period corresponding to the calculated phase difference (PDIF) can be determined based on the contrast to correct the phase difference (PDIF), so that the measurement distance (D) can be exceeded according to the modulation frequency (F). freq ) of the target measurement distance (D max ) to accurately measure the distance to the target object 1.

[0169] The principle that makes the above method possible is that the contrast can indicate the aliasing period. In the first part P1 of the microframe (e.g., MF1), when the first modulation control signal (MCS1) and the second modulation control signal (MCS2) have a logic high level (H) together, the intensity component of the reflected light (RL) can be mainly obtained. In the second part P2 of the microframe (e.g., MF1), when the first modulation control signal (MCS1) and the second modulation control signal (MCS2) have a predetermined modulation phase difference with the emitted light (EL), the first modulation control signal (MCS1) and the second modulation control signal (MCS2) alternately have a logic high level (H), so that the amplitude component of the reflected light (RL) can be mainly obtained. Therefore, as the phase difference between the emitted light (EL) and the reflected light (RL) increases, the amplitude component of the reflected light (RL) can become relatively stronger than the intensity component of the reflected light (RL), so that the contrast can indicate the aliasing period.

[0170] In addition, the first slope (A) and the first reference intercept to the third reference intercept (C a1 ~C a3 ) is a value depending on the characteristics (e.g., N, M, L, etc.) of the image sensing device 100, and the phase difference corrector 220 may receive the first slope (A) and the first reference intercept to the third reference intercept (C) from the image sensing device 100 or an external device (e.g., a test device). a1 ~C a3 ), and pre-store the received first slope (A) and reference intercept (C a1 ~C a3 ).

[0171] To describe the operation of the image processing device 200 as a whole, the phase difference calculator 210 can calculate the phase difference (PDIF) between the modulated light signal (MLS) and the reflected light (RL) based on multiple captured data segments (e.g., S0, S90, S180, and S270) generated by multiple modulation control signals (e.g., MCS1 and MCS2), each of which has a predetermined modulation phase difference with the modulated light signal (MLS).

[0172] The phase difference corrector 220 may calculate a contrast ratio as a ratio of an amplitude component of the reflected light (RL) to an intensity component of the reflected light (RL) using a plurality of captured data segments (e.g., S0, S90, S180, and S270), and may calculate an aliasing value (R) corresponding to the calculated contrast ratio.

[0173] In addition, the distance calculator 230 may accurately calculate the distance to the target object 1 using the corrected phase difference (PDIF) obtained by correcting the phase difference (PDIF) according to the aliasing value (R).

[0174] Figure 7 is a timing diagram illustrating a method for removing a component caused by ambient light from the intensity of reflected light (RL) according to some embodiments of the present disclosure.

[0175] refer to Figure 7 In order to accurately determine the aliasing period based on the contrast according to the above method, the intensity component of the reflected light (RL) is sufficiently reduced as the distance to the target object 1 increases, so that the contrast should be significantly increased. However, when the intensity component of the reflected light RL includes an excessive amount of ambient light component (hereinafter referred to as "ambient light component"), although the distance to the target object 1 changes, the change in contrast is not sufficient, so that it may become difficult to distinguish the aliasing periods from each other.

[0176] Furthermore, if the ambient light component fluctuates significantly, the magnitude component of the contrast is not affected, but the intensity component of the contrast may be greatly affected depending on the extent of such fluctuation. In order to accurately measure the distance to the target object 1, the contrast must be changed only according to the distance to the target object 1. However, the contrast may change according to the change in the ambient light component, so that the distance to the target object 1 may not be accurately measured.

[0177] Figure 7 is a timing diagram illustrating a method for acquiring an ambient light component according to some embodiments of the present disclosure.

[0178] and Figure 3 The depth frame is different. Figure 7 The depth frame shown in FIG. 1 may also include a fifth subframe. The fifth subframe may be Figure 3 The first to fourth subframes (SF1 to SF4) of FIG. 1 include an exposure period and a readout period in the same manner. Except for some differences, Figure 7 The basic operations during the exposure period and readout period are similar to Figure 3 The operation is the same as that of , and therefore for the sake of brevity, this article will omit its redundant description, so the following will focus on Figure 3 Different characteristics are used to describe Figure 7 operation.

[0179] The exposure period of the fifth subframe may include a fifth microframe (MF5). Since the modulated light signal (MLS) maintains a logic low level L in the fifth microframe (MF5), the emitted light (EL) and the reflected light (RL) may not appear. In addition, since each of the first modulation control signal (MCS1) and the second modulation control signal (MCS2) in the fifth microframe (MF5) has a logic high level, the ambient light component may be captured in the first tap and the second tap. Image data corresponding to the ambient light components captured in the first tap and the second tap may be defined as sub-ambient light data (AMB). sub ). When the length of the exposure period of the fifth subframe is defined as the ambient exposure time (T amb ) and the length of the exposure period of each of the first to fourth subframes (SF1 to SF4) is defined as a reference exposure time (T depth ), the phase difference corrector 220 may calculate the ambient light data (AMB) using the following formula 14.

[0180] [Formula 14]

[0181]

[0182] In the fifth microframe MF5, the first tap and the second tap can capture the ambient light component without using the reflected light (RL), and the ambient exposure time (T amb ) is not necessarily the same as the reference exposure time (T depth ) and can be shorter than the reference exposure time (T depth ). Even if the environmental exposure time (T amb ) is greater than the reference exposure time (T depth ) is short, the phase difference corrector 220 can also convert the sub-ambient light data (AMB sub ) multiplied by the ambient exposure time (T amb ) and the reference exposure time (T depth ) ratio, so that the ambient light data (AMB) can be calculated. That is, the degradation of the frame rate of the depth frame can be minimized by reducing the duration required for the fifth microframe (MF) as much as possible.

[0183] The phase difference corrector 220 may remove the ambient light component from the intensity of the reflected light (RL) using the ambient light data (AMB), as shown in Formula 15 below.

[0184] [Formula 15]

[0185]

[0186] The phase difference corrector 220 can calculate the contrast using the intensity of the reflected light (RL) from which the ambient light component has been removed. Therefore, regardless of the ambient light component, the contrast can be changed according to the distance to the target object 1, and the distance to the target object 1 can be accurately measured.

[0187] FIG. 8A to FIG. 9 is a timing diagram illustrating another method for removing a component caused by ambient light from the intensity of reflected light according to some embodiments of the present disclosure.

[0188] The following will refer to FIG. 8A to FIG. 9 Another embodiment of removing the ambient light component from the intensity of the reflected light (RL) is described. Figure 3 The depth frame is different, according to FIG. 8A to FIG. 8D The depth frames (DepthFrame0 to DepthFrame3) of another embodiment may include a waveform of a modulated light signal (MLS) whose timing point is adjusted. Each of the depth frames (DepthFrame0 to DepthFrame3) may be in the same Figure 3 The depth frame includes an exposure period and a readout period in the same manner and except for some differences, FIG. 8A to FIG. 9The basic operations during the exposure period and readout period shown in Figure 3 The same as those of , and therefore for the sake of brevity, the redundant description thereof will be omitted herein, so that the following will focus on Figure 3 Different characteristics are used to describe FIG. 8A to FIG. 9 operation.

[0189] The first to fourth depth frames (DepthFrame0~DepthFrame3) can be arranged continuously with each other. In each of the first to fourth depth frames (DepthFrame0~DepthFrame3), the modulated light signal (MLS) generated in a microframe (e.g., MF1) of at least one subframe (e.g., SF1) can be generated to have a predetermined time delay from the start time of the microframe. Hereinafter, the first microframe (MF1) included in each of the first to fourth depth frames (DepthFrame0~DepthFrame3) will be described as an example, but the same description can also be applied to other microframes (MF2~MF4) as needed.

[0190] exist Fig. 8A In the embodiment, a pulse of a modulated light signal (MLS) generated in a first microframe (MF1) of a first depth frame (DepthFrame0) may be delayed by a first delay time amount (D1) from a start time point of the first microframe (MF1).

[0191] exist Figure 8B In the present invention, a pulse of a modulated light signal (MLS) generated in a first microframe (MF1) of a second depth frame (DepthFrame1) may be delayed by a second delay time amount (D2) from a start time point of the first microframe (MF1).

[0192] exist Figure 8C In the embodiment, a pulse of the modulated light signal (MLS) generated in the first microframe (MF1) of the third depth frame (DepthFrame2) may be delayed by a third delay time amount (D3) from a start time point of the first microframe (MF1).

[0193] exist Fig.8D In the embodiment, a pulse of the modulated light signal (MLS) generated in the first microframe (MF1) of the fourth depth frame (DepthFrame3) may be delayed by a fourth delay time amount (D4) from a start time point of the first microframe (MF1).

[0194] In each of the first to fourth depth frames (DepthFrame0 to DepthFrame3), the modulated light signal (MLS) may include an intentionally delayed pulse. The first to fourth delay time amounts (D1 to D4) may be different from each other, and each of the first to fourth delay time amounts (D1 to D4) may be set to a value less than or equal to a duration (e.g., cycle / 2) corresponding to a predetermined phase (e.g., π), but the scope of the present disclosure is not limited thereto.

[0195] exist Fig. 9 , the contrast and phase difference (PDIF) obtained from each of the first to fourth depth frames (DepthFrame0 to DepthFrame3) are displayed in a graph showing the relationship between the contrast of the reflected light (RL) and the phase difference (PDIF). The contrast corresponding to each of the first to fourth depth frames (DepthFrame0 to DepthFrame3) may be plotted at a position deviating from the first measurement value (MV1). This is because the contrast varies according to the ambient light component, and the phase difference corrector 220 may calculate the slope of the approximate straight line (A amb ), the approximate straight line approximates four coordinates of the contrast and phase difference (PDIF) obtained from the first depth frame to the fourth depth frame (DepthFrame0~DepthFrame3).

[0196] The phase difference corrector 220 may calculate the intensity component (Intensity amb ) is removed, as shown in the following formula 16. That is, the phase difference corrector 220 can use the information obtained from the three depth frames (DepthFrame0 to DepthFrame2) before the fourth depth frame (DepthFrame3) to correct the intensity component of the reflected light (RL) obtained from the fourth depth frame (DepthFrame3). As a result, the intensity component (Intensity) of the reflected light (RL) obtained by correcting the intensity component (Intensity) of the reflected light (RL) can be prevented. amb ) and may cause unnecessary frame rate degradation, and may perform an intensity component (Intensity amb ) correction.

[0197] [Formula 16]

[0198]

[0199] In Formula 16, 'A' may refer to a first slope of each of first to fourth straight lines ( AL1 ˜ AL4 ) to which the first to fourth measurement values ​​( MV1 ˜ MV4 ) are respectively approximated.

[0200] The phase difference corrector 220 can calculate the contrast using the intensity component (Intensity) of the reflected light (RL) from which the ambient light component is removed. Therefore, regardless of the ambient light component, the contrast can be changed according to the distance to the target object 1, and the distance to the target object 1 can be accurately measured.

[0201] In addition, since the pulses of the modulated light signal (MLS) are intentionally delayed by a corresponding delay time amount (D1~D4) in each depth frame (DepthFrame0~DepthFrame3), the distance calculator 230 can compensate (for example, add or subtract) the distance corresponding to each delay time amount (D1~D4) corresponding to each depth frame (DepthFrame0~DepthFrame3) in the process of calculating the distance to the target object 1.

[0202] exist FIG. 8A to FIG. 9 In the example of FIG. 1 , the slope (A) can be calculated using an approximate straight line formed by approximating four coordinates of the contrast and phase difference (PDIF) obtained from four depth frames for one pixel, respectively. amb ). However, a more accurate slope of the approximate straight line (Aamb) can be obtained using a slope obtained by averaging slopes calculated for a plurality of pixels.

[0203] According to another embodiment, the intensity component (Intensity amb ) has a slope ratio (A amb The calculation of the slope ratio (A) may not be performed in all depth frames, but may be performed periodically in some depth frames according to a predetermined period. amb In the depth frame calculated with a slope ratio (A / A), the slope ratio calculated in the previous depth frame can be used amb / A) to perform the intensity component of the reflected light (RL) amb ) correction.

[0204] According to yet another embodiment, when a specific condition is satisfied (e.g., a condition indicating that a change in an illuminance value measured by an illuminance sensor (not shown) is a threshold value or more, or a condition indicating that a change in a sum (or average) of intensity components of all pixels is a threshold value or more), correction of an intensity component (Intensity Component) of reflected light (RL) may be performed. amb ) has a slope ratio (A ambHere, without performing the slope ratio (A amb In the depth frame calculated with a slope ratio (A / A), the slope ratio calculated in the previous depth frame can be used amb / A) to perform the intensity component of the reflected light (RL) amb ) correction.

[0205] The number of subframes of each depth frame, the order of the first part P1 and the second part P2 in each microframe, the order of microframes, etc. described in the present disclosure are merely examples, and other implementations are also possible.

[0206] Fig.10 is shown corresponding to Figure 1 1000 is a block diagram of a computing device of an image processing device.

[0207] refer to Fig.10 , the computing device 1000 may represent a computer program product for executing Figure 1 An embodiment of a hardware configuration of an operation of the image processing apparatus 200.

[0208] The computing device 1000 may be mounted on a chip separate from the chip on which the image sensing device 100 is mounted. According to one embodiment, the chip on which the image sensing device 100 is mounted and the chip on which the computing device 1000 is mounted may be implemented in one package (e.g., a multi-chip package (MCP)), but the scope of the disclosed technology is not limited thereto.

[0209] in addition, Figure 1 The internal configuration or arrangement of the image sensing device 100 and the image processing device 200 described in the above may vary according to the implementation. For example, at least a portion of the image sensing device may be included in the image processing device 200. Alternatively, at least a portion of the computing device 1000 may be included in the image sensing device. In this case, at least a portion of the computing device 1000 may be mounted together on a chip on which the image sensing device 100 is mounted.

[0210] The computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0211] Processor 1010 can process execution Figure 1 The processor 1010 may be used to represent the image processing device 200, but the scope of the present disclosure is not limited thereto.

[0212] The memory 1020 may store data and / or instructions required to perform operations of the components (210-230) of the image processing apparatus 200, and may be accessed by the processor 1010. For example, the memory 1020 may be a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a non-volatile memory (e.g., a programmable read-only memory (PROM), an erasable PROM (EPROM), etc.), an EEPROM (electrically erasable PROM), a flash memory, etc.).

[0213] That is, a computer program for executing the operation of the image processing apparatus 200 disclosed in the present disclosure is recorded in the memory 1020 and executed and processed by the processor 1010 , thereby realizing the operation of the image processing apparatus 200 .

[0214] The input / output interface 1030 is an interface that connects an external input device (eg, a keyboard, a mouse, a touch panel, etc.) and / or an external output device (eg, a display) to the processor 1010 to allow data to be transmitted and received.

[0215] The communication interface 1040 is a component that can transmit and receive various data to / from an external device (eg, an application processor, an external memory, etc.), and may be a device that supports wired or wireless communication.

[0216] As is apparent from the above description, the image sensing device according to some embodiments of the present disclosure may more accurately obtain the distance to the target object to be captured by correcting the phase difference calculated by the time of flight (ToF) method.

[0217] The embodiments of the present disclosure can provide various effects that can be recognized directly or indirectly.

[0218] Although a number of exemplary embodiments have been described, it will be appreciated that modifications and enhancements to the disclosed embodiments and other embodiments may be devised based on what is described and / or illustrated in this disclosure.

[0219] CROSS-REFERENCE TO RELATED APPLICATIONS

[0220] This patent application claims priority to and the benefit of Korean Patent Application No. 10-2023-0119738, filed on September 8, 2023, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image processing device, comprising: a phase difference calculator that calculates a phase difference between the modulated light signal and the reflected light based on a plurality of captured data segments generated by a plurality of modulation control signals each having a predetermined modulation phase difference with respect to the modulated light signal; a phase difference corrector that calculates a contrast ratio using the plurality of captured data segments and determines an aliasing value corresponding to the contrast ratio, the contrast ratio being a ratio of a magnitude component of the reflected light to an intensity component of the reflected light; as well as A distance calculator calculates a distance to a target object using a corrected phase difference obtained by correcting the phase difference according to the aliasing value.

2. The image processing device according to claim 1, wherein: The predetermined modulation phase difference is one of 0 degrees, 90 degrees, 180 degrees and 270 degrees.

3. The image processing device according to claim 1, wherein: The plurality of captured data segments are generated in units of depth frames, and Each of the depth frames includes a plurality of sub-frames.

4. The image processing device according to claim 3, wherein: Each of the plurality of subframes comprises: an exposure period in which photo charges are generated when the reflected light is sensed by the plurality of modulated control signals, and the generated photo charges are accumulated, and A readout period in which the captured data segment corresponding to the accumulated photocharge is generated.

5. The image processing device according to claim 4, wherein: The exposure period includes a plurality of microframes, and Each of the plurality of microframes comprises: a first portion, in which the plurality of modulation control signals maintain a logic high level, and A second part, in which the plurality of modulation control signals alternately have a logic high level.

6. The image processing device according to claim 5, wherein: The plurality of modulated control signals have opposite phases to each other in the second portion.

7. The image processing device according to claim 5, wherein: The number of pulses of the modulated optical signal in each of the plurality of microframes is smaller than a value obtained by dividing a duration of the microframe by a pulse period.

8. The image processing device according to claim 4, wherein: The exposure period includes a microframe in which the modulated light signal maintains a logic low level and each of the plurality of modulated control signals maintains a logic high level.

9. The image processing apparatus according to claim 8, wherein: The phase difference corrector also removes an ambient light component from the intensity component using ambient light data obtained from the microframe.

10. The image processing device according to claim 5, in, The pulse of the modulated optical signal is delayed from the starting time point of each of the plurality of microframes by a predetermined delay time amount, and Wherein a delayed pulse is generated in each of the plurality of microframes.

11. The image processing apparatus according to claim 10, wherein: The phase difference corrector further corrects the intensity component based on a slope of an approximate straight line obtained by approximating coordinates of contrast and phase difference obtained from a current depth frame and at least one depth frame before the current depth frame.

12. The image processing device according to claim 1, in, The phase difference corrector further calculates a measurement intercept based on a first slope indicating a ratio of an increase in the phase difference to an increase in the contrast, the contrast, and the phase difference, and The phase difference corrector determines the aliasing value by comparing the measured intercept with a plurality of reference intercepts.

13. The image processing apparatus according to claim 12, wherein: The plurality of reference intercepts are determined by intercepts of straight lines corresponding to the aliasing values.

14. The image processing apparatus according to claim 1, wherein: The corrected phase difference is a value obtained by adding a phase correction value depending on the aliasing value to the phase difference.

15. An imaging device, comprising: an image sensing device that generates a plurality of captured data segments using a plurality of modulation control signals each having a predetermined modulation phase difference relative to the modulated light signal; as well as An image processing device that uses the multiple captured data segments to calculate a contrast ratio, which is a ratio of an amplitude component of the reflected light to an intensity component of the reflected light, and calculates a distance to a target object using a corrected phase difference obtained by correcting a phase difference between the modulated light signal and the reflected light according to an aliasing value corresponding to the contrast ratio.

16. The imaging device according to claim 15, wherein: The plurality of captured data segments are generated in units of depth frames, Each of the depth frames includes a plurality of subframes, and Each of the plurality of subframes comprises: an exposure period in which photo charges are generated when the reflected light is sensed by the plurality of modulated control signals and the generated photo charges are accumulated, and A readout period in which the captured data segment corresponding to the accumulated photocharge is generated.

17. The imaging device according to claim 16, wherein: The exposure period includes a plurality of microframes, and Each of the plurality of microframes comprises: a first portion, in which the plurality of modulation control signals maintain a logic high level, and A second part, in which the plurality of modulation control signals alternately have a logic high level.

18. The imaging device according to claim 17, wherein: The number of pulses of the modulated optical signal in each of the plurality of microframes is smaller than a value obtained by dividing a duration of the microframe by a pulse period.

19. The imaging device according to claim 15, wherein: The image processing device further: calculating a measurement intercept based on a first slope, the contrast, and the phase difference, the first slope indicating a ratio of an increase in the phase difference to an increase in the contrast; as well as The aliasing value is determined by comparing the measured intercept to a plurality of reference intercepts.

20. An image sensing device, comprising: pixels that generate a plurality of pixel signals by sensing reflected light using a plurality of modulation control signals each having a predetermined modulation phase difference with respect to the modulated light signal during an exposure period; as well as a readout circuit that generates a plurality of captured data segments by processing the pixel signal, The exposure period includes: a first portion, in which the plurality of modulation control signals maintain a logic high level, and A second part, in which the plurality of modulation control signals alternately have a logic high level.

Images