Image capture device

Abstract

This application relates to image capture devices. An image capture device can include a first ToF (time of flight) sensor configured to calculate a distance to a target object using a time difference between a reference pulse time at which an illumination modulated light signal is emitted and a pulse sensing time at which a reflected modulated light signal reflected from the target object and incident thereon is sensed, a second ToF sensor configured to calculate the distance to the target object using a phase difference between the modulated light signal and the reflected modulated light signal, and a controller configured to enable one of the first ToF sensor and the second ToF sensor based on first pixel data generated by the first ToF sensor sensing the reflected modulated light signal.

Description

Technical Field

[0001] Various embodiments generally relate to an image capture device including a sensor capable of sensing a distance to a target object based on a ToF (Time of Flight) method. Background Art

[0002] Image sensors are semiconductor devices that capture optical image light and convert it into electrical signals for displaying and further processing the captured image. The development of various industries and sectors, including the computer, automotive, medical, and communications industries, has created a diverse demand for advanced image sensors in various devices and systems, such as smartphones, digital cameras, game consoles, the Internet of Things (IoT), robots, security cameras, and medical miniature cameras.

[0003] One common type of image sensing device is the charge coupled device (CCD), which has long dominated the field of image sensors. Another common type of image sensing device is the complementary metal oxide semiconductor (CMOS) image sensing device. CCD image sensors provide higher image quality and better noise characteristics than CMOS image sensors. However, CMOS image sensors are now widely used due to certain advantages over CCD image sensors (including, for example, higher frame rates, shutter speeds, and various scanning methods). In addition, CMOS image sensors and signal processing circuits can be integrated into a single chip, which can miniaturize electronic devices while achieving low power consumption. In addition, the use of CMOS manufacturing technology can reduce production costs. These characteristics make CMOS image sensors more suitable for implementation in mobile devices. Summary of the Invention

[0004] Embodiments of the disclosed technology relate to an image capture device that can be used to measure a distance between the image capture device and an object by changing an operating mode.

[0005] In an embodiment, an image capture device may include: a first time-of-flight (ToF) sensor, which is configured to include a first photosensitive pixel to detect a modulated light signal and is configured to process a pixel signal from the first photosensitive pixel to calculate the distance to the target object using a time difference between a reference pulse time of the modulated light signal irradiating the target object and a pulse sensing time of a reflected modulated light signal sensed by the first photosensitive pixel and reflected from the target object and incident thereon; a second ToF sensor, which is configured to include a second photosensitive pixel to detect the modulated light signal and is configured to process a pixel signal from the second photosensitive pixel to calculate the distance to the target object using a phase difference between the modulated light signal and the reflected modulated light signal; and a controller, which is connected to the first ToF sensor and the second ToF sensor and is configured to enable one of the first ToF sensor and the second ToF sensor for sensing the modulated light signal based on first pixel data generated by the first ToF sensor when sensing the reflected modulated light signal.

[0006] In an embodiment, an image capture device may include: an RBO (reference background light offset) storage unit configured to store a reference background light offset generated based on pixel data generated using sunlight as background light; and a controller configured to use the reference background light offset to select one of a first operating mode and a second operating mode. The first operating mode is a mode for calculating the distance to the target object using the time difference between a modulated light signal irradiated onto the target object and a reflected modulated light signal reflected from the target object and incident on the image capture device. The second operating mode is a mode for calculating the distance to the target object using the phase difference between the modulated light signal and the reflected modulated light signal.

[0007] Based on this embodiment, the image capturing device can select the optimal ToF method according to the image capturing environment and sense the distance to the object.

[0008] Furthermore, various effects directly or indirectly understood through this document can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram illustrating an example of an image capturing device based on an embodiment of the disclosed technology.

[0010] Figure 2 This is an example Figure 1 A block diagram of an example of a first ToF sensor is shown.

[0011] Figure 3 is a timing diagram illustrating an example of how a first ToF sensor measures the distance between an image capture device and a target object.

[0012] Figure 4 This is an example Figure 1 A block diagram of an example of a second ToF sensor is shown.

[0013] Figure 5A is a timing diagram illustrating an example of how the second ToF sensor measures the distance between the image capture device and the target object.

[0014] Figure 5B is a timing diagram illustrating an example of how the second ToF sensor measures the distance between the image capture device and the target object.

[0015] Figure 6 This is an example Figure 1 A flowchart of a method of operating an image capture device is shown.

[0016] Figure 7 An example histogram of the first pixel data PD1 is illustrated.

[0017] Figure 8 The spectral irradiance of different light sources is illustrated. DETAILED DESCRIPTION

[0018] Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments, but includes various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the present disclosure can provide various effects that can be directly / indirectly recognized by the present disclosure.

[0019] Figure 1 is a diagram illustrating an example of an image capturing device based on an embodiment of the disclosed technology.

[0020] Reference Figure 1 The image capture device 10 may refer to a device such as a digital still camera for capturing still images or a digital video camera for capturing videos. For example, the image capture device 10 may be implemented as a DLSR (digital single-lens reflex) camera, a mirrorless camera, or a mobile phone (specifically, a smartphone), but is not limited thereto. The concept of the image capture device 10 may include a device that includes a lens and an image pickup element, and thus can generate an image by capturing an image of an object.

[0021] The image capture device 10 may include a light source module 50, a first ToF (Time of Flight) sensor 100, a second ToF sensor 200, a controller 300, an image signal processor 400, and a reference background light offset (RBO) storage unit 500 for storing RBO data. The image capture device 10 can measure the distance to the target object 30 by using the ToF principle, thereby calculating the distance based on the time during which light irradiated from the image capture device 10 is reflected by the target object 30 and incident on the image capture device 10.

[0022] Depending on the type of ToF sensor enabled, the image capture device 10 may have two modes, namely, a first operating mode and a second operating mode. The first operating mode in which the first ToF sensor 100 is enabled and the second ToF sensor 200 is disabled may indicate a mode in which the image capture device 10 uses the first ToF sensor 100 to measure the distance to the target object 30. The second operating mode in which the first ToF sensor 100 is disabled and the second ToF sensor 200 is enabled may indicate a mode in which the image capture device 10 uses the second ToF sensor 200 to measure the distance to the target object 30.

[0023] In response to the control signal of the controller 300, the light source module 50 can irradiate a modulated light signal (MLS) toward the target object 30. Examples of the light source module 50 may include a combination of an LD (laser diode) or an LED (light emitting diode), an NIR (near infrared laser), a point light source, a monochromatic illumination source, and other laser sources. The LD or LED emits light of a specific wavelength band (e.g., infrared or visible light), and the monochromatic illumination source is configured by combining a white light and a monochromator. For example, the light source module 50 can emit infrared light having a wavelength of 800nm ​​to 1000nm. The modulated light signal MLS can be an optical pulse signal modulated based on predetermined modulation characteristics (e.g., waveform, wavelength, period, amplitude, frequency, phase, duty cycle, etc.). In addition, the modulated light signal MLS can have different modulation characteristics in the first operating mode and the second operating mode, respectively. For example, the pulse of the modulated light signal MLS irradiated in the first operating mode can have a larger amplitude than the pulse of the modulated light signal MLS irradiated in the second operating mode.

[0024] The first ToF sensor 100 can measure the distance to the target object 30 using a first ToF method under the control of the controller 300. In an embodiment, the first ToF method is a direct ToF method. According to the direct ToF method, the first ToF sensor 100 can calculate the distance to the target object 30 by directly measuring a round-trip time and performing an operation on the round-trip time and the speed of light. The round-trip time indicates the time difference between the time point when the modulated light signal MLS modulated with a predetermined modulation characteristic is irradiated to the target object 30 and the time point when the modulated light signal MLS is reflected from the target object 30 and incident on the first ToF sensor 100.

[0025] The first ToF sensor 100 may operate according to a control signal received from the controller 300. The first ToF sensor 100 may provide the controller 300 with pixel data generated during the process of calculating the distance to the target object 30. In addition, the first ToF sensor 100 may transmit a result obtained by measuring the distance to the target object 30 to the image signal processor 400.

[0026] The second ToF sensor 200 can measure the distance to the target object 30 using a second ToF method under the control of the controller 300. In an embodiment, the second ToF method can be an indirect ToF method. According to the indirect ToF method, the second ToF sensor 200 calculates the distance to the target object 30 by irradiating a modulated light signal MLS modulated with a predetermined modulation characteristic toward the target object 30, sensing a reflected modulated light signal MLS_R reflected from the target object 30 and incident thereon, and then calculating a phase difference between the modulated light signal MLS and the reflected modulated light signal MLS_R.

[0027] The second ToF sensor 200 may operate according to a control signal received from the controller 300. In addition, the second ToF sensor 200 may transmit a result obtained by measuring the distance to the target object 30 to the image signal processor 400.

[0028] The controller 300 may perform overall control of components included in the image capturing device 10. Specifically, the controller 300 may select any one of the first operation mode and the second operation mode as the operation mode of the image capturing device 10 and enable the first ToF sensor 100 or the second ToF sensor 200.

[0029] The controller 300 may determine the operation mode of the image capturing device 10 based on the pixel data provided from the first ToF sensor 100 and the RBO provided from the RBO storage unit 500, and will be referred to below. Figure 6 The following figures describe the process.

[0030] The image signal processor 400 can collect the distance data of each pixel input from each of the first ToF sensor 100 and the second ToF sensor 200 and generate a depth image indicating the distance to the target object 30. In addition, the image signal processor 400 can perform image signal processing on the generated depth image to remove noise and improve the quality of the image. The depth image output from the image signal processor 400 can be automatically stored in the internal memory or external memory of the image capture device 10 or the device in which the image capture device 10 is installed, or stored in the image capture device 10 or the internal memory or external memory according to the user's request, or displayed through a display. Alternatively, the depth image output from the image signal processor 400 can be used to control the operation of the image capture device 10 or the device in which the image capture device 10 is installed.

[0031] The RBO storage unit 500 may be a database for storing RBOs corresponding to at least one type of light source. The RBO may indicate pixel data generated by the first ToF sensor 100 sensing background light within the same wavelength range as the modulated light signal MLS, and the RBO is experimentally stored for each type of light. Even when the modulated light signal MLS is not generated, the first ToF sensor 100 may sense the background light around the image capture device 10 and generate pixel data corresponding to the intensity of the background light. The image capture device 10 may generate pixel data corresponding to the intensity of the background light while changing the background light at a test step, and the RBO storage unit 500 may store the generated pixel data as the RBO of the corresponding background light. For example, the background light may be sunlight.

[0032] Such an RBO can have different values ​​for each type of background light. Therefore, the RBO can be used as information for identifying background light. The controller 300 of the image capture device 10 based on the present embodiment can compare the information obtained from the pixel data provided by the first ToF sensor 100 with the RBO provided from the RBO storage unit 500, identify the background light applied to the image capture device 10, and determine whether the image capture device 10 is located outdoors or indoors. When it is determined that the image capture device 10 is located outdoors, the controller 300 can operate the image capture device 10 in the first operating mode. Conversely, when it is determined that the image capture device 10 is located indoors, the controller 300 can operate the image capture device 10 in the second operating mode.

[0033] In an outdoor area, sunlight may serve as the main background light, and due to the open area, the distance between the image capture device 10 and the target object 30 may be relatively large. In an indoor area, indoor light may serve as the main background light instead of sunlight, and due to the enclosed area, the distance between the image capture device 10 and the target object 30 may be relatively small.

[0034] The first ToF sensor 100 may have relatively weak interference with sunlight and a relatively large effective measurement distance. The effective measurement distance may indicate the maximum distance that can be substantially measured. On the other hand, the second ToF sensor 200 may have relatively strong interference with sunlight and a relatively small effective measurement distance. Therefore, when the image capture device 10 is located outdoors, the image capture device 10 may operate in the first operating mode to enable the first ToF sensor 100, which is more suitable for outdoor areas. Furthermore, when the image capture device 10 is located indoors, the image capture device 10 may operate in the second operating mode to enable the second ToF sensor 200, which is more suitable for indoor areas.

[0035] Figure 2 This is an example Figure 1A block diagram of an example of a first ToF sensor is shown. Figure 3 is a timing diagram illustrating an example of how a first ToF sensor measures the distance between an image capture device and a target object.

[0036] Reference Figure 2 The first ToF sensor 100 may include a first pixel array 110 of photosensitive pixels for detecting MLS from the light source module 50, a first readout circuit 120 connected to receive an output from the first pixel array 110, and a first computing unit 130 connected to receive readout data from the first readout circuit 120.

[0037] The first pixel array 110 may include, for example, a plurality of first pixels arranged continuously in a 2D matrix in the column and row directions. Each first pixel is a photosensitive pixel that can generate a first pixel signal PS1 as an electrical signal corresponding to the intensity of the reflected modulated light signal MLS_R by photoelectrically converting the reflected modulated light signal MLS_R received through the lens module (not shown) and output the generated first pixel signal PS1 to the first readout circuit 120. In this case, the first pixel signal PS1 may be a signal that does not indicate the color of the target object 30 but rather indicates information corresponding to the distance to the target object 30.

[0038] The lens module (not shown) may include a focusing lens and a filter. The focusing lens collects the reflected modulated light signal MLS_R reflected from the target object 30 and focuses the collected light onto the first pixel array 110. The focusing lens may include a focusing lens or a positive lens formed of a suitable lens material (e.g., glass, plastic, or other dielectric materials) in the form of a curved surface or a cylindrical optical element. The focusing lens may include one or more lenses. The filter in the lens module is configured to selectively transmit light corresponding to the same wavelength range as the wavelength range of the modulated light signal MLS, while blocking light of other wavelengths. Therefore, the light incident on the first pixel array 110 can be limited to light included in the wavelength range of the modulated light signal MLS, which makes it possible to suppress noise generated by light outside the wavelength range of the modulated light signal MLS (such as background light (BGL)). This design of the filter of the ToF sensor is used to capture MLS light reflected from the imaged object to measure the distance between the imaged object and the imaging device, and is therefore different from the color filtering design of the filter in the image sensing pixel used to capture color information in the incident image.

[0039] In an embodiment, a condenser lens and a filter may be provided to correspond to each first pixel, and thus different pairs of condenser lenses and filters may be provided for different first pixels and for spatially matched arrays of pairs of condenser lenses and filters, respectively.

[0040] In an embodiment, for a direct ToF method, each first pixel may be a direct pixel. Each first pixel may include a photoelectric conversion element configured to generate and accumulate photocharges corresponding to the intensity of incident light, and a pixel signal circuit configured to generate a first pixel signal PS1 corresponding to the amount of photocharges.

[0041] For example, each first pixel can be a SPAD (single photon avalanche diode) pixel. According to the working principle of the SPAD pixel, by applying a reverse bias to the SPAD to increase the electric field, the electrons generated by the incident photons due to the strong electric field migrate to generate electron-hole pairs (impact ionization). Specifically, in a SPAD operated in Geiger mode with a reverse bias applied higher than the breakdown voltage, countless carriers can be generated when the carriers (electrons or holes) generated by the incident light collide with the electrons and holes generated by impact ionization. Therefore, although a single photon is incident on the SPAD, the single photon can trigger an avalanche breakdown to generate a measurable current pulse.

[0042] The first readout circuit 120 may generate first pixel data PD1 as digital data corresponding to the analog first pixel signal PS1 by processing the first pixel signal PS1 output from each first pixel. For example, the first readout circuit 120 may include an analog-to-digital converter for converting the first pixel signal PS1 into the first pixel data PD1.

[0043] The first readout circuit 120 may transmit the first pixel data PD1 to the first calculation unit 130 and the controller 300 .

[0044] The first calculation unit 130 can calculate the flight time from the reference pulse time to the pulse sensing time by comparing the reference pulse time provided by the controller 300 with the pulse sensing time determined by analyzing the first pixel data PD1, calculate the distance between the target object 30 and the first ToF sensor 100 based on the calculated flight time, and transmit the calculation result to the image signal processor 400. The reference pulse time may indicate a point in time when the modulated light signal MLS is irradiated, and the pulse sensing time may indicate a point in time when the reflected modulated light signal MLS_R reflected from the target object 30 and incident on the first ToF sensor 100 is sensed.

[0045] exist Figure 1 In the implementation of the image capture device 10 in FIG, the first ToF sensor 100 and the second ToF sensor 200 are configured differently and operate or are enabled for MLS sensing at different times, while the light source module 50 is controlled to generate different modulated light signals to be detected by the first ToF sensor 100 and the second ToF sensor 200, respectively.

[0046] Figure 3 is a timing diagram illustrating an example of how the first ToF sensor measures the distance between the image capture device and the target object. In some implementations, the first ToF sensor 100 can calculate the distance between the image capture device and the target object, as will be discussed below.

[0047] Reference Figure 3 、 Figure 5A and Figure 5B As shown in the optical modulation signal, the pulse of the modulated light signal MLS irradiated while the first ToF sensor 100 is enabled may have a larger amplitude and a smaller pulse width than the pulse of the modulated light signal MLS irradiated while the second ToF sensor 200 is enabled, and the time point of generating the pulse may be defined and used as a reference pulse time RPT to determine the flight time of the reflected MLS received by the first ToF sensor 100. The first calculation unit 130 may receive a control signal, and the controller 300 applies the control signal to control the light source module 50 to emit the MLS and set the reference pulse time RPT.

[0048] The first pixel array 110 and the first readout circuit 120 may generate first pixel data PD1 by sensing a reflection modulated light signal MLS_R reflected from the target object 30 and incident on the first ToF sensor 100 .

[0049] The first calculation unit 130 may analyze the first pixel data PD1 to determine that a reflection pulse in the reflection MLS arrives at a time referred to as a pulse sensing time (PST), which is a point in time when the first pixel data PD1 has a value equal to or greater than a threshold data to indicate the presence of a reflection pulse.

[0050] The first calculation unit 130 can calculate the distance between the target object 30 and the first ToF sensor 100 by calculating the flight time corresponding to the time difference between the reference pulse time RPT and the pulse sensing time PST, and performing an operation on the calculated flight time and the speed of light (for example, multiplying the speed of light by a value obtained by dividing the flight time by 2 (because the MLS light travels a round trip)).

[0051] Figure 4 This is an example Figure 1 A block diagram of an example of a second ToF sensor is shown. Figure 5A is a diagram for describing an example of a method in which the second ToF sensor measures a distance to a target object. Figure 5B is a diagram for describing another example of a method in which the second ToF sensor measures a distance to a target object.

[0052] Reference Figure 4 , the second ToF sensor 200 may include a second pixel array 210 , a second readout circuit 220 , and a second calculation unit 230 .

[0053] The second pixel array 210 may include, for example, a plurality of second pixels continuously arranged in a 2D matrix in the column direction and the row direction. Each second pixel may generate a second pixel signal PS2 as an electrical signal corresponding to the intensity of the reflected modulated light signal MLS_R by photoelectrically converting the reflected modulated light signal MLS_R received through the lens module (not shown), and output the generated second pixel signal PS2 to the second readout circuit 220. At this time, the second pixel signal PS2 may be a signal that does not indicate the color of the target object 30 but indicates information corresponding to the distance to the target object 30. Since reference has been made to Figure 2 The lens module (not shown) is described, and thus a repeated description will be omitted herein.

[0054] In an embodiment, each second pixel can be an indirect pixel for an indirect ToF method. For example, each of the second pixels can be a CAPD (current-assisted photon demodulation) pixel. However, the scope of the present disclosure is not limited thereto. Unlike the pixel structure in the first ToF sensor 100, each second pixel in the second pixel array 210 of the second ToF sensor 200 may include: a photoelectric conversion element configured to generate and accumulate photocharges corresponding to the intensity of incident light; and two pixel signal circuits (a first pixel signal circuit and a second pixel signal circuit) instead of one pixel signal circuit, each configured to generate a second pixel signal PS2 corresponding to the amount of photocharge generated at different timings. For this operation, the first pixel signal circuit and the second pixel signal circuit can respectively receive a first modulation control signal MCS1 and a second modulation control signal MCS2 as different signals for controlling the timing of sensing photocharges. The first modulation control signal MCS1 and the second modulation control signal MCS2 can be controlled to have different phase values ​​relative to each other during operation, for example, completely out of phase with each other (i.e., a phase difference of 180 degrees). For example, the first modulation control signal MCS1 and the second modulation control signal MCS2 may have a phase difference of 0 degrees (i.e., in phase) and a phase difference of 180 degrees with the modulated optical signal MLS, respectively. Alternatively, the first modulation control signal MCS1 and the second modulation control signal MCS2 may have a phase difference of 90 degrees (i.e., 90 degrees out of phase) and a phase difference of 270 degrees with the modulated optical signal MLS, respectively.

[0055] like Figure 5AAs shown in the example in FIG, the first pixel signal circuit for each second pixel in the second ToF sensor 200 may generate a second pixel signal PS2 corresponding to the amount of photocharge generated in the first period (PR1) in response to the first modulation control signal MCS1, and transmit the generated second pixel signal PS2 to the second readout circuit 220. The second pixel signal circuit may generate a second pixel signal PS2 corresponding to the amount of photocharge generated in a second period (PR2) different from the first period (PR1) in response to the second modulation control signal MCS2, and transmit the generated second pixel signal PS2 to the second readout circuit 220.

[0056] The second readout circuit 220 can generate second pixel data PD2 as digital data corresponding to the analog second pixel signal PS2 by processing the second pixel signal PS2 output from each second pixel. For example, the second readout circuit 220 may include an analog-to-digital converter for converting the second pixel signal PS2 into the second pixel data PD2.

[0057] The second readout circuit 220 may transmit the second pixel data PD2 to the second calculation unit 230 .

[0058] The second calculation unit 230 can calculate the phase difference between the modulated light signal MLS and the reflected modulated light signal MLS_R by comparing the second pixel data PD2 obtained by converting the second pixel signal PS2 generated by the first pixel signal circuit and the second pixel signal circuit respectively, calculate the distance between the target object 30 and the second ToF sensor 200 based on the calculated phase difference, and send the calculation result to the image signal processor 400.

[0059] Figure 5A is a timing diagram illustrating an example of how the second ToF sensor 200 measures the distance between the image capture device and the target object by detecting MLS_R by the second pixel via MCS1 and MCS2. In implementation, the second ToF sensor 200 can calculate the distance between the image capture device and the target object, as discussed below.

[0060] Figure 5A An example method in which the second ToF sensor 200 calculates the distance may be defined as a first phase difference sensing method. The first phase difference sensing method may be referred to as a two-phase modulation method because it uses modulation signals having two different phases.

[0061] The pulse of the modulated light signal MLS irradiated while the second ToF sensor 200 is enabled may have a smaller amplitude and a larger pulse width than the pulse of the modulated light signal MLS irradiated while the first ToF sensor 100 is enabled. Due to the flight time during which the modulated light signal MLS is reflected from the target object 30 and incident on the second ToF sensor 200, the reflected modulated light signal MLS_R may have a predetermined phase difference from the modulated light signal MLS.

[0062] The first modulation control signal MCS1 may be in phase with the modulated light signal MLS (i.e., have a phase difference of 0 degrees with the modulated light signal MLS) and have an enable voltage (or a logic high level) in the first period PR1. Second pixel data PD2 indicating the amount Q(0) of photocharge generated in the first period PR1 may be generated by the first modulation control signal MCS1 having the enable voltage.

[0063] The second modulation control signal MCS2 may be completely out of phase with the modulated light signal MLS (i.e., have a phase difference of 180 degrees from the modulated light signal MLS) and have an enable voltage (or a logic high level) in the second period PR2. Second pixel data PD2 indicating the amount Q(π) of photocharge generated in the second period PR2 may be generated by the second modulation control signal MCS2 having the enable voltage.

[0064] Under such timing control by controlling both modulation of MLS via MLS_R and detection by the second pixel via MCS1 and MCS2 , the reflected modulated light signal MLS_R can be captured separately and individually in the time domain in the first and second periods PR1 and PR2 .

[0065] The second calculation unit 230 can calculate the phase difference θ according to the following formula 1 based on the second pixel data PD2 indicating the amounts Q(0) and Q(π) of photo charges captured in the first period PR1 and the second period PR2, and calculate the distance between the second ToF sensor 200 and the target object 30 according to the flight time proportional to the phase difference θ.

[0066]

[0067] Figure 5B is a timing diagram illustrating another example of how the second ToF sensor measures the distance between the image capture device and the target object by detecting MLS_R by the second pixel via MCS1 and MCS2.

[0068] Figure 5BThe example method in which the second ToF sensor 200 calculates the distance may be defined as a second phase difference sensing method. The second phase difference sensing method may be referred to as a four-phase modulation method because a modulation signal having four different phases is used.

[0069] An operation interval in which the second ToF sensor 200 calculates the distance to the target object 30 may be divided into a first sensing period SP1 and a second sensing period SP2 in sequence.

[0070] The modulated optical signal MLS and the reflected modulated optical signal MLS_R are Figure 5A The modulated light signal MLS and the reflected modulated light signal MLS_R illustrated in FIG are substantially the same signal and are based on the assumption that the modulated light signal MLS and the reflected modulated light signal MLS_R have a predetermined phase difference θ. In addition, it is assumed that the phase difference θ between the modulated light signal MLS and the reflected modulated light signal MLS_R remains constant during the first sensing period SP1 and the second sensing period SP2.

[0071] In the first sensing period SP1, the first modulation control signal MCS1 may be completely in phase with the modulated light signal MLS (i.e., have a phase difference of 0 degrees with the modulated light signal MLS) and have an enable voltage (or a logic high level) in the first period PR1. Second pixel data PD2 indicating the amount Q(0) of photocharge generated in the first period PR1 may be generated by the first modulation control signal MCS1 having the enable voltage.

[0072] In the first sensing period SP1, the second modulation control signal MCS2 may be completely out of phase with the modulated light signal MLS (i.e., have a phase difference of 180 degrees with the modulated light signal MLS), and have an enable voltage (or a logic high level) in the second period PR2. Second pixel data PD2 indicating the amount Q(π) of photocharge generated in the second period PR2 may be generated by the second modulation control signal MCS2 having the enable voltage.

[0073] In the second sensing period SP2, the first modulation control signal MCS1 may have a phase difference of 90 degrees with the modulated light signal MLS and have an enable voltage (or a logic high level) in the third period PR3. Second pixel data PD2 indicating the amount Q(π / 2) of photocharge generated in the third period PR3 may be generated by the first modulation control signal MCS1 having the enable voltage.

[0074] In the second sensing period SP2, the second modulation control signal MCS2 may have a phase difference of 270 degrees from the modulated light signal MLS and have an enable voltage (or a logic high level) in the fourth period PR4. Second pixel data PD2 indicating the amount Q (3π / 2) of photocharge generated in the fourth period PR4 may be generated by the second modulation control signal MCS2 having the enable voltage.

[0075] That is, the reflection modulated light signal MLS_R can be separately and individually captured in the time domain during the first and second periods PR1 and PR2 of the first sensing period SP1, and the reflection modulated light signal MLS_R can be separately and individually captured in the time domain during the third and fourth periods PR3 and PR4 of the second sensing period SP2. The total amount of photocharge generated by the reflection modulated light signal MLS_R can be defined as the sum of Q(0) and Q(π) or the sum of Q(π / 2) and Q(3π / 2).

[0076] The absolute value of the difference between Q(0) and Q(π) can be defined as ΔQ(0) (=│Q(0)-Q(π)│), and the absolute value of the difference between Q(π / 2) and Q(3π / 2) can be defined as ΔQ(π / 2) (=│Q(π / 2)-Q(3π / 2)│). The first modulation control signal MCS1 for obtaining Q(0) and the second modulation control signal MCS2 for obtaining Q(π) can have a phase difference of 90 degrees with the first modulation control signal MCS1 for obtaining Q(π / 2) and the second modulation control signal MCS2 for obtaining Q(3π / 2), respectively. Therefore, the sum of ΔQ(0) and ΔQ(π / 2) can have a constant value (i.e., total charge).

[0077] For the sum of ΔQ(0) and ΔQ(π / 2) having constant values, ΔQ(0) and ΔQ(π / 2) can each linearly increase / decrease according to the change in the phase difference θ. That is, as the phase difference θ increases, ΔQ(0) can linearly decrease in the period when the phase difference θ changes from 0 to π, and linearly increase in the period when the phase difference θ changes from π to 2π. In addition, ΔQ(π / 2) can linearly increase in the period when the phase difference θ changes from 0 to π / 2, linearly decrease in the period when the phase difference θ changes from π / 2 to 3π / 2, and linearly increase in the period when the phase difference θ changes from 3π / 2 to 2π. Therefore, based on the ratio relationship between ΔQ(0) and ΔQ(π / 2), the phase difference θ can be calculated.

[0078] The second calculation unit 230 can calculate ΔQ(0) and ΔQ(π / 2) based on the second pixel data PD2 indicating the amounts Q(0), Q(π), Q(π / 2) and Q(3π / 2) of photo charges captured in the first period PR1 to the fourth period PR4, calculate the phase difference θ according to the following formula 2, and calculate the distance between the second ToF sensor 200 and the target object 30 based on the flight time proportional to the phase difference θ.

[0079]

[0080] Specifically, the second phase difference sensing method (four-phase modulation method) can use differential values ​​such as ΔQ(0) and ΔQ(π / 2) when calculating the phase difference. Therefore, the second phase difference sensing method can remove the background noise components contained in Q(0), Q(π), Q(π / 2), and Q(3π / 2), respectively, thereby more accurately calculating the distance.

[0081] Figure 6 This is an example Figure 1 A flowchart of a method of operating an image capture device is shown. Figure 7 is a diagram illustrating an example of a histogram of the first pixel data PD1. Figure 8 is a graph illustrating the spectral irradiance of each light source.

[0082] Reference Figure 6 , when the operation of the image capture device 10 starts, the initial operation mode may be set to the first operation mode as a default mode. Therefore, the controller 300 may enable the first ToF sensor 100 and disable the second ToF sensor 200. In step S10, the enabled first ToF sensor 100 may generate first pixel data PD1 by sensing the reflected modulated light signal MLS_R reflected from the target object 30 and incident on the first ToF sensor 100. The first pixel data PD1 may indicate data obtained by converting the first pixel signal PS1 of one first pixel included in the first pixel array 110, but the scope of the present disclosure is not limited thereto. In another embodiment, the first pixel data PD1 may indicate an average value of data obtained by converting the first pixel signals PS1 of the first pixels belonging to a row included in the first pixel array 110 or an average value of data obtained by converting the first pixel signals PS1 of all first pixels included in the first pixel array 110.

[0083] In step S20, the controller 300 may obtain a histogram of the first pixel data PD1 from the first pixel data PD1. The first pixel data PD1 may be periodically generated according to a predetermined cycle. The histogram of the first pixel data PD1 may be a graph indicating accumulation of sequentially generated first pixel data PD1 over time.

[0084] Figure 7 An example histogram of the first pixel data PD1 accumulated over time is illustrated.

[0085] The first pixel data PD1 sequentially generated during the predetermined accumulation period CP may be accumulated. The first pixel data PD1 may be divided into first pixel data PD1_1 having a value equal to or greater than the threshold data TD and first pixel data PD1_2 having a value less than the threshold data TD. The threshold data TD may be experimentally determined to determine whether the first pixel data PD1 corresponds to a pulse of the reflected modulated light signal MLS_R. That is, the first pixel data PD1_1 may indicate the intensity of the pulse of the reflected modulated light signal MLS_R, while the first pixel data PD1_2 may indicate the intensity of the background light (BGL) incident on the image capture device 10, rather than the intensity of the pulse of the reflected modulated light signal MLS_R.

[0086] In step S30, the controller 300 may calculate a BGL offset based on the histogram of the first pixel data PD1. Figure 7 In this embodiment, the controller 300 may calculate an average value of the first pixel data PD1_2 corresponding to a result obtained by sensing the background light during the accumulation period (CP), and decide the calculated average value as a BGL offset.

[0087] In step S40, the controller 300 may compare the BGL offset (RBO) with the RBO value of sunlight as the BGL provided from the RBO storage unit 500. The RBO of sunlight may be experimentally determined by collecting first pixel data PD1 generated by the first ToF sensor 100 while sunlight serves as the background light (BGL) incident on the first ToF sensor 100.

[0088] In one implementation of the disclosed ToF technology, an embodiment is described for determining whether the image capture device 10 is located outdoors or indoors based on the RBO of sunlight. However, the scope of the present disclosure is not limited thereto, and the operating mode of the image capture device 10 can be changed by determining the surrounding environment of the image capture device 10 using the RBO of another light source, which is background light (BGL).

[0089] Figure 8 A graph indicating the spectral irradiance of each type of light source is illustrated. Spectral irradiance may indicate the light output density at a specific wavelength (unit: nm) and is expressed in W / (m 2 *nm)(W: watt, m: meter).

[0090] Figure 8Illustrates the spectral irradiance of the solar spectrum, sulfur lamp, xenon arc lamp, QTH (quartz tungsten halogen) lamp, and mirror incandescent lamp according to wavelength changes.

[0091] The average value of the spectral irradiance in the wavelength range of the modulated optical signal MLS (MLS wavelength range) can have the following relationship (sulfur lamp < xenon arc lamp < solar spectrum < QTH lamp < mirror incandescent lamp). The wavelength range of the modulated optical signal MLS can be a wavelength range corresponding to near-infrared light, but the scope of the present disclosure is not limited thereto. Since the spectral irradiance can respectively indicate the light output density of the corresponding light source, the RBO of the corresponding light source can be respectively proportional to the average value of the spectral irradiance of the light source in the wavelength range of the modulated optical signal MLS. That is, as the average value of the spectral irradiance of the corresponding light source changes, the RBO of the corresponding light source can have different values and become data capable of identifying a specific light source.

[0092] For example, the RBOs of the sulfur lamp, xenon arc lamp, solar spectrum, QTH lamp, and mirror incandescent lamp are 20, 60, 100, 130, and 150 respectively. That is, when the first pixel data PD1 generated by the first ToF sensor 100 is similar to 100, it is very likely that the background light acting on the first ToF sensor 100 is sunlight. Therefore, it can be considered that the image capture device 10 is located outdoors. On the contrary, when the first pixel data PD1 generated by the first ToF sensor 100 is not similar to 100, it is very likely that the background light acting on the first ToF sensor 100 is not sunlight. Therefore, it can be considered that the image capture device 10 is located indoors.

[0093] In step S50, the controller 300 can compare the BGL offset with the RBO of sunlight provided by the RBO storage unit 500, and determine whether the image capture device 10 is currently located outdoors based on whether the BGL offset is included within a predetermined range between a first value (e.g., 85) and a second value (e.g., 115) around the RBO of sunlight (e.g., 100). The first value can be less than the RBO and the second value can be greater than the RBO.

[0094] When it is determined that the image capture device 10 is currently located outdoors because the BGL offset is included within the predetermined range (Yes in step S50), in step S60, the controller 300 can keep the image capture device 10 in the first operation mode and measure the distance to the target object 30 using the first ToF sensor 100. This is because, since the first ToF sensor 100 senses the reflected modulated optical signal MLS_R of a pulse with a relatively large amplitude and a small pulse width, the interference of sunlight is relatively weak and the effective measurement distance is relatively large.

[0095] When it is determined that the image capturing device 10 is currently located indoors because the BGL offset is not included in the predetermined range (No in step S50), the controller 300 may change the operation mode of the image capturing device 10 to the second operation mode and measure the distance to the target object 30 using the second ToF sensor 200. This is because, since the second ToF sensor 200 senses the reflected modulated light signal MLS_R of a pulse having a relatively small amplitude and a large pulse width, interference from sunlight is relatively strong and the effective measurement distance is relatively small.

[0096] When the BGL offset is not within the predetermined range but is greater than the RBO of sunlight ("Yes" in step S70), in step S80, the controller 300 may operate the second ToF sensor 200 according to the second phase difference sensing method. This is because, since the image capturing device 10 is located indoors but background light with relatively high spectral irradiance (such as a QTH lamp or a mirror incandescent lamp) may act on the second ToF sensor 200, it is more advantageous for the second ToF sensor 200 to measure the distance to the target object 30 according to the second phase difference sensing method that can remove the background light using a differential value.

[0097] When the BGL offset is not included in the predetermined range but is less than the RBO of sunlight ("No" in step S70), in step S90, the controller 300 may operate the second ToF sensor 200 according to the first phase difference sensing method. This is because, since the image capturing device 10 is located indoors and background light with relatively low spectral irradiance (such as a sulfur lamp or a xenon arc lamp) may act on the second ToF sensor 200, it is more advantageous for the second ToF sensor 200 to measure the distance to the target object 30 according to the first phase difference detection method having a relatively high operating speed.

[0098] It can be assumed that the image capture device 10 is moved to the outside while the distance is measured using the second ToF sensor 200 because the image capture device 10 is located indoors. Alternatively, it can be assumed that the image capture device 10 is moved to the inside while the distance is measured using the first ToF sensor 100 because the image capture device 10 is located outdoors. In this case, the image capture device 10 may continue to operate in an operation mode that is not suitable for the image capture environment. To prevent this, the controller 300 may execute again according to a specific cycle, a user's request, or a request of the image signal processor 400 that has recognized that the distance cannot be measured normally according to the calculation result of the first ToF sensor 100 or the second ToF sensor 200 (for example, the calculated distance is repeatedly 0 or the effective measured distance). Figure 6Specifically, when the current operation mode is the second operation mode, the controller 300 can operate the image capturing device 10 in the first operation mode by changing the operation mode of the image capturing device 10 so as to perform Figure 6 The process shown.

[0099] In the image capturing device 10 according to the present embodiment, the controller 300 may analyze background light of the image capturing device 10 and operate the image capturing device 10 in an optimal operation mode according to the analysis result.

[0100] Although various embodiments have been described above, those skilled in the art will understand that the described embodiments are merely examples, and modifications and improvements to the disclosed embodiments and other embodiments may be made based on the contents described or illustrated in this patent document.

[0101] CROSS-REFERENCE TO RELATED APPLICATIONS

[0102] This patent document claims priority to and the benefit of Korean Application No. 10-2021-0046976, filed on April 12, 2021, which is hereby incorporated by reference in its entirety.

Claims

1. An image capturing device, comprising: a first ToF sensor including first photosensitive pixels to detect a modulated light signal and to process pixel signals from the first photosensitive pixels to calculate a distance to the target object using a time difference between a reference pulse time at which the modulated light signal is irradiated onto the target object and a pulse sensing time at which a reflected modulated light signal reflected from the target object and incident thereon is sensed by the first photosensitive pixels; a second ToF sensor including second photosensitive pixels to detect the modulated light signal and to process pixel signals from the second photosensitive pixels to calculate a distance to the target object using a phase difference between the modulated light signal and the reflected modulated light signal; as well as a controller coupled to the first ToF sensor and the second ToF sensor and enabling one of the first ToF sensor and the second ToF sensor for sensing the modulated light signal based on first pixel data generated by the first ToF sensor when sensing the reflected modulated light signal, The controller calculates a background light (BGL) offset caused by background light received by the first ToF sensor and indicating a level of the background light based on a histogram obtained by accumulating the first pixel data, and enables one of the first ToF sensor and the second ToF sensor for sensing the modulated light signal by comparing the BGL offset with a previously stored reference BGL offset.

2. The image capture device according to claim 1, wherein The controller calculates the BGL offset by calculating an average value of the first pixel data having a value less than threshold data in the histogram.

3. The image capturing device according to claim 1, wherein The reference BGL offset is determined based on first pixel data generated while background light acting on the first ToF sensor is sunlight.

4. The image capture device according to claim 1, wherein The reference BGL offset is proportional to the spectral irradiance of sunlight.

5. The image capturing device according to claim 1, wherein When the BGL offset is included in a predetermined range between a first value and a second value, the controller enables the first ToF sensor, and Wherein, the first value is smaller than the reference BGL offset and the second value is larger than the reference BGL offset.

6. The image capture device according to claim 1, wherein When the BGL offset is not included in a predetermined range between a first value and a second value, the controller enables the second ToF sensor, and The first value is smaller than the reference BGL offset and the second value is larger than the reference BGL offset.

7. The image capture device according to claim 6, wherein: When the BGL offset is greater than the reference BGL offset while the BGL offset is not included in the predetermined range, the controller operates the second ToF sensor using modulation signals having four different phases.

8. The image capture device according to claim 6, wherein When the BGL offset is smaller than the reference BGL offset while the BGL offset is not included in the predetermined range, the controller operates the second ToF sensor using modulation signals having two different phases.

9. The image capture device according to claim 1, wherein The first ToF sensor includes: a first pixel array including a plurality of first pixels, each of the first pixels generating a first pixel signal as an electrical signal corresponding to an intensity of the reflected modulated light signal; a first readout circuit that generates the first pixel data by converting the first pixel signal into digital data; and A first calculation unit determines a time point when the first pixel data has a value equal to or greater than threshold data as the pulse sensing time, and calculates a distance to the target object using a time difference between the reference pulse time and the pulse sensing time.

10. The image capture device according to claim 9, wherein Each of the plurality of first pixels is a single photon avalanche diode (SPAD) pixel.

11. The image capture device according to claim 1, wherein The second ToF sensor includes: a second pixel array including a plurality of second pixels, each of the second pixels generating a second pixel signal as an electrical signal corresponding to an intensity of the reflected modulated light signal in response to a first modulation control signal and a second modulation control signal having different phases; a second readout circuit that generates second pixel data by converting the second pixel signal into digital data; and a second calculation unit that calculates the phase difference by comparing the second pixel data generated in response to the first modulation control signal with the second pixel data generated in response to the second modulation control signal, and calculates the distance to the target object using the phase difference.

12. The image capture device according to claim 11, wherein Each of the plurality of second pixels is a current-assisted photon demodulation (CAPD) pixel.

13. The image capture device according to claim 1, wherein the modulated light signal illuminated while the first ToF sensor is enabled for sensing the modulated light signal has a magnitude greater than a magnitude of the modulated light signal illuminated while the second ToF sensor is enabled for sensing the modulated light signal, The modulated light signal irradiated while the first ToF sensor is enabled has a pulse width smaller than a pulse width of the modulated light signal irradiated while the second ToF sensor is enabled for sensing the modulated light signal.

14. An image capturing device, the image capturing device comprising: a reference background light offset (RBO) storage unit storing a reference BGL offset generated based on pixel data generated using sunlight as background light; as well as a controller that selects one of a first operating mode and a second operating mode using the reference BGL offset, wherein the first operation mode is a mode for calculating the distance to the target object using a time difference between a modulated light signal irradiated onto the target object and a reflected modulated light signal reflected from the target object and incident on the image capturing device, and The second operation mode is a mode for calculating the distance to the target object using a phase difference between the modulated light signal and the reflected modulated light signal.

Images