Enhancing depth estimation of luminance images using guided filters

By combining guided filters with brightness images to enhance the depth estimation of the ToF camera system, the problem of insufficient accuracy in estimating the depth of object boundaries in the ToF camera system is solved, and the system's performance in applications such as virtual reality, augmented reality, and vehicle navigation is improved.

CN118922744APending Publication Date: 2024-11-08ANALOG DEVICES INT UNLTD CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380028458.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-08-18
Filing Date
2023-02-16
Publication Date
2024-11-08

Smart Images

  • Figure BDA0005048323550000093
    Figure BDA0005048323550000093
  • Figure BDA0005048323550000143
    Figure BDA0005048323550000143
  • Figure BDA0005048323550000144
    Figure BDA0005048323550000144
Patent Text Reader

Abstract

The depth estimation system may use a guide filter to enhance depth estimation using a luminance image. The light is projected onto an object. The object reflects at least a portion of the projection light. The reflected light is at least partially captured by the image sensor. The depth estimation system may generate a depth image based on a phase shift between the captured light and the projected light, and generate a brightness image based on a brightness of the captured light. The depth estimation system may identify pixels representing at least a portion of the boundary of the object using a guided filter. The guide filter may determine a depth value for a corresponding pixel in the luminance image based on a value of the pixel. The depth estimation system may assign depth values to pixels and generate an enhanced depth image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 310,859, filed on February 16, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to depth estimation and, more particularly, to using guided filters to enhance depth estimation of luminance images. Background Art

[0004] One technique for measuring depth is to directly or indirectly calculate the time it takes for a signal to travel from a source on a sensor to a reflecting surface and back to the sensor. The time of travel is proportional to the distance from the sensor to the reflecting surface. This travel time is often referred to as time of flight (ToF). ToF sensors can use various types of signals, the most common being sound and light. Given the advantages of light in terms of speed, range, power, and weight, some sensors use light as a carrier. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] For a more complete understanding of the present disclosure and its features and advantages, please refer to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts, and in which:

[0006] Figure 1 A depth estimation system according to some embodiments of the present disclosure is shown;

[0007] Figure 2A shows a continuous wave of a projection signal 210 according to some embodiments of the present disclosure;

[0008] Figure 2B shows a continuous wave of a capture signal 220 according to some embodiments of the present disclosure;

[0009] Figure 3 shows a period of a continuous wave of modulated light according to some embodiments of the present disclosure;

[0010] Figure 4 is a block diagram illustrating a controller according to some embodiments of the present disclosure;

[0011] Figure 5A shows an example depth image according to some embodiments of the present disclosure;

[0012] Figure 5B The present invention shows some embodiments of the present invention. Figure 5A Example brightness image corresponding to the depth image in;

[0013] Figure 5CAccording to some embodiments of the present disclosure, Figure 5A The depth image and Figure 5B Example depth-enhanced images generated from the brightness images in ;

[0014] Figure 6 An example system including a depth estimation system according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A mobile device incorporating a depth estimation system according to some embodiments of the present disclosure is shown;

[0016] Figure 8 An entertainment system incorporating a depth estimation system according to some embodiments of the present disclosure is shown;

[0017] Fig. 9 shows an example robot 902 incorporating a depth estimation system according to some embodiments of the present disclosure;

[0018] Fig.10 is a flow chart illustrating a method of enhancing depth estimation using guided filters according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Overview

[0020] ToF camera systems are distance imaging systems. ToF camera systems typically include a light source that projects light and an imaging sensor that receives reflected light. ToF camera systems can estimate the distance between the imaging sensor and the object by measuring the round trip of light. Continuous wave ToF camera systems can project multiple cycles of a continuous light wave and determine the distance based on the phase difference between the projected light and the received reflected light. A depth image can be generated based on the phase difference. However, ToF camera systems are generally unable to accurately estimate the depth of boundaries, such as the edges of objects, reflectivity boundaries (e.g., the boundary between two areas with different reflectivity characteristics), etc. Inaccurate depth estimation may limit the use of ToF camera systems in many applications, such as virtual reality (VR), augmented reality (AR) or mixed reality (MR) applications, vehicle navigation applications, or other applications. Therefore, there is a need to improve depth estimation technology.

[0021] Embodiments of the present disclosure relate to a depth estimation system that can use a guided filter to enhance ToF depth estimation using a brightness image. The brightness image can be an active brightness image, such as an infrared (IR) image, or an RGB (red, green, blue) image. The depth estimation system can simultaneously acquire a brightness image and a ToF depth estimate. The brightness image and the ToF depth estimate can be based on the same light source (e.g., IR) or different light sources (e.g., visible light for a brightness image and IR for a ToF depth estimate). The brightness image can better detect boundaries and can be used to enhance the depth image generated by a ToF camera system.

[0022] An example of a depth estimation system includes an illuminator assembly, a camera assembly, and a controller. Light is projected onto an object. The illuminator assembly can project light to illuminate a local area, such as an area including an object. The light can be modulated light, such as modulated IR. The illuminator assembly can project pulsed light. Alternatively, the illuminator assembly can project one or more continuous waves, such as continuous waves of different frequencies. The object can reflect at least a portion of the projected light. The camera assembly captures at least a portion of the reflected light, and can convert the captured photons into electric charge and accumulate the charge. The depth estimation system can generate a depth image and a brightness image based on the charge accumulated in the camera assembly. In some embodiments, the depth image is based on a phase shift between the captured light and the projected light, and the brightness image is based on the brightness of the captured light. The depth image may include a plurality of depth pixels, each of which may correspond to a pixel in the brightness image.

[0023] The depth estimation system can use a guided filter to determine whether a pixel represents any boundary (or any part of a boundary) of an object. For pixels representing the boundary (or part of the boundary) of an object, the guided filter can determine an enhanced depth value of the pixel based on the value of the corresponding pixel in the luminance image. The enhanced depth value may be different from and / or independent of the depth value of the pixel in the depth image. For pixels representing the plane of an object (i.e., not representing any boundary of an object), the guided filter may determine an enhanced depth value of the pixel based on the depth values ​​of the surrounding pixels in the depth image. The enhanced depth value may be the average of the depth values ​​of the surrounding pixels. The depth estimation system may generate an enhanced depth image with an enhanced depth value.

[0024] Because the depth estimation system can enhance the depth estimation of boundaries by utilizing the brightness image that shows clearer boundaries, and can also smooth planes by using average depth values, the enhanced depth image generated by the depth estimation system can better display boundaries than a conventional ToF depth image. With more accurate depth estimation, the enhanced depth image can be used in a variety of applications.

[0025] Depth Estimation System Example

[0026] Figure 1A depth estimation system 100 is shown according to some embodiments of the present disclosure. The depth estimation system 100 may use ToF technology to generate a depth image. The depth estimation system 100 includes an illuminator assembly 110, a camera assembly 120, and a controller 130. The illuminator assembly 110 includes an emitter 160 and a diffuser 165. The camera assembly 120 includes a lens 190 and an image sensor 195. In alternative configurations, different and / or additional components may be included in the depth estimation system 100. In addition, the functionality attributed to one component of the depth estimation system 100 may be implemented by a different component included in the depth estimation system 100 or a different system than that shown. For example, the illuminator assembly 110 may not include a diffuser or multiple diffusers. As another example, the camera assembly 120 may not include a lens or more than one lens.

[0027] The illuminator assembly 110 projects light 170 to a local area including the object 140. The emitter 160 is a light source that emits light ("emitted light"). In some embodiments, the emitter 160 may include a laser, such as an IR or near IR (NIR) laser, an edge-emitting laser, a vertical cavity surface emitting laser (VCSEL), etc. In other embodiments, the emitter 160 may include one or more light emitting diodes (LEDs). The emitter 160 may emit light in the visible light band (i.e., about 380nm to 750nm), the near infrared band (i.e., about 750nm to 1mm), the ultraviolet band (i.e., 10nm to 380nm), the short-wave infrared (SWIR) band (e.g., about 900nm to 2200nm), some other part of the electromagnetic spectrum, or some combination thereof. In some embodiments, the illuminator assembly 110 may include a plurality of emitters 160, each of which may emit a different wavelength. For example, the illuminator assembly 110 may include a first emitter that emits IR and a second emitter that emits visible light. Before light 170 is projected into the local area, diffuser 165 spreads or scatters the emitted light. Diffuser 165 can also control the brightness of the emitted light. In some embodiments, diffuser 165 can be translucent or semi-transparent. In other embodiments, illuminator assembly 110 can include more, fewer, or different components. For example, illuminator assembly 110 can include one or more additional diffusers to direct light from emitter 160 to one or more additional objects in the local area.

[0028] The illuminator assembly 110 may project the light 170 as modulated light, for example, according to a periodic modulation waveform. An example of a periodic modulation waveform may be a sinusoidal modulation waveform. The frequency of the periodic modulation waveform is the frequency of the modulated light.

[0029] The illuminator assembly 110 can project one or more continuous waves. For a single continuous wave, the illuminator assembly 110 can project multiple cycles. Different continuous waves may have different wavelengths and frequencies. For example, the illuminator assembly 110 can project a continuous wave with a modulation frequency in the range of 50 MHz to 200 MHz. In one embodiment, the illuminator assembly 110 includes multiple (i.e., at least two) light projectors. The light projectors can project continuous waves with different frequencies. The light projectors can alternate and project continuous waves at different times. For example, a first light projector projects a first continuous wave with a first frequency during a first time period. After the first time period, a second light projector projects a second continuous wave with a second frequency during a second time period. After the second time period, a third light projector projects a third continuous wave with a third frequency during a third time period. These three continuous waves can constitute a cycle. This cycle can be repeated.

[0030] In another embodiment, the illuminator assembly 110 may include a light projector that projects all three continuous waves. In other embodiments, the illuminator assembly 110 may project a different number of continuous waves, such as two or more than three. One cycle may constitute a frame. The total time of a cycle may be 10-20ms. The illuminator assembly 110 may project light through multiple cycles to obtain multiple frames. There may be time intervals between cycles. More information about modulated light with multiple frequencies is provided below in conjunction with Figure 2.

[0031] At least a portion of object 140 is illuminated by light 170. For simplicity and illustration purposes, Figure 1 The object 140 in the embodiment has a cubic shape. In other embodiments, the object 140 may have other shapes or structures. Figure 1 Not shown, the local area may also include other objects that may be illuminated by the light. The object reflects the light, reflecting light 180, and the reflected light 180 may be captured by the camera assembly 120.

[0032] The camera assembly 120 captures image data of at least a portion of the local area illuminated with the light 170. For example, the camera assembly 120 captures the reflected light 180 and generates image data based on the reflected light 180. The reflected light 180 may be IR. In some embodiments, the camera assembly 120 may also capture visible light reflected by the object 140. The visible light may be projected by the illuminator assembly 110, ambient light, or a combination of both.

[0033] Even in Figure 1In some embodiments, the camera assembly 120 is separate from the illuminator assembly 110. In some embodiments, the camera assembly 120 is also located in the same location as the illuminator assembly 110 (e.g., can be part of the same device). The lens 190 receives the reflected light 180 and directs the reflected light 180 to the image sensor 195. The image sensor 195 includes a plurality of pixels 197. Even if Figure 1 Although the pixels 197 shown in FIG. 1 are arranged in one column, the pixels 197 of the image sensor 195 may also be arranged in multiple columns.

[0034] In some embodiments, pixel 197 includes a photodiode that is sensitive to light and converts collected photons into electric charge (e.g., photoelectrons). Each photodiode has one or more storage areas for storing electric charge. Image sensor 195 can be a ToF sensor and a brightness sensor. Pixel 197 can be a depth sensing pixel, a brightness sensing pixel, or both. The depth sensing pixel is configured to present a depth output signal that depends on the distance from the image sensor 195 to the track of the object 140 imaged on the depth sensing pixel. Such a distance is the "depth" of the track of the object 140. Each depth sensing pixel can independently determine the distance to the object 140 seen by the pixel. The depth output signal of the depth sensing pixel in the image sensor 195 can be used to generate a depth image of a local area. The brightness sensing pixel is configured to present a brightness output signal that depends on the brightness of the light reflected from the track of the object 140 imaged on the brightness sensing pixel. The brightness output signal of the brightness sensing pixel in the image sensor 195 can be used to generate a brightness image of a local area. The brightness image can be an active brightness image. An example of a brightness image is an IR image. In some embodiments, each pixel 197 of the image sensor 195 can generate a depth output signal and a brightness output signal from the reflected light captured by the pixel 197. In other embodiments, the image sensor 195 includes two groups of pixels 197: one group for sensing depth and the other group for detecting brightness. The output signal of the image sensor 195 can be an analog signal, such as an electric charge.

[0035] In embodiments where the illuminator assembly 110 projects multiple continuous waves or multiple cycles of modulated light, the image sensor 195 may be synchronized with the projection of the illuminator assembly 110. For example, the image sensor 195 may have one or more exposure intervals during which the image sensor 195 exposes a portion of a local area and accumulates charge in the image sensor 195. Outside of the exposure interval, the image sensor 195 is not exposed. In some embodiments, the exposure interval of the image sensor 195 may be synchronized with the continuous wave or cycle projected by the illuminator assembly 110. For example, the exposure interval starts before or when the continuous wave or cycle starts and ends after or when the continuous wave or cycle ends. In other embodiments, the image sensor 195 may have multiple exposure intervals for a single continuous wave. For example, the image sensor 195 may make multiple exposures during a continuous wave, and the multiple exposures may correspond to different phase offsets. In one example, a continuous wave has three exposure intervals with different phase offsets, such as 0° (0), 120° (2π / 3), and 240° (4π / 3). There may be time intervals between the exposure intervals. The time intervals may be 1-2 milliseconds (ms). The exposure intervals may have a constant duration, for example approximately 100 microseconds (μs).In alternative embodiments, the exposure intervals may have different durations.

[0036] In some embodiments, the image sensor 195 can be scanned using a global shutter. The image sensor 195 includes a global shutter that can be opened and scanned during each exposure interval and closed at the end of the exposure interval. Additionally or alternatively, the image sensor 195 can include a tunable filter. The tunable filter blocks light from reaching the detector and can be mounted anywhere in the optical path of the reflected light 180. For example, the tunable filter is attached to the top of the image sensor 195 or the front of the camera assembly 120. The tunable filter can be switched between on (active) and off (inactive). The tunable filter can be inactive during the exposure interval and active at the end of the exposure interval. When the tunable filter is inactive, light can pass through the tunable filter and reach the image sensor 195. When the tunable filter is active, light is blocked from the image sensor 195.

[0037] In some embodiments, when the tunable filter is active, it may pass light of a particular wavelength (or a particular wavelength band) but block light of other wavelengths. For example, the tunable filter may pass light of a wavelength projected by the illuminator assembly 110 (e.g., light 170) but block light of other wavelengths, which may, for example, reduce noise in image data captured by the image sensor 195. In examples where the light 170 is IR, the tunable filter may block visible light. In other embodiments, when the tunable filter is active, it may block light of all wavelengths to avoid charge accumulation in the image sensor 195. In embodiments where the tunable filter blocks all light, dark noise calibration may be performed on the image sensor 195.

[0038] The camera assembly 120 can read out the stored photoelectrons from the image sensor 195 to obtain image data, for example, from the storage area of ​​each pixel 197 of the image sensor 95. During the readout, the camera assembly 120 can convert the photoelectrons into digital signals (i.e., analog-to-digital conversion). In embodiments where the illuminator assembly 110 includes multiple light projectors, the photoelectrons corresponding to the modulated light pulses projected by the different light projectors can be stored in separate storage areas of each photodiode. The camera assembly 120 can read out the separate storage areas to obtain image data. In some embodiments, the camera assembly 120 can read out all the image data stored in the image sensor 195. In other embodiments, the camera assembly 120 can read out some of the image data stored in the image sensor 195. For example, in embodiments where the exposure interval of the image sensor 195 is synchronized with the continuous wave projected by the illuminator assembly 110, the camera assembly 120 can perform multiple readout intervals for the continuous wave. Each readout interval can correspond to a different phase offset. In one example, a continuous wave has three readout intervals at three different phase offsets (e.g., 0°, 120°, and 240°). There may be a time interval between the readout intervals. The time interval may be 1-2 milliseconds (ms). The readout interval may have a constant duration, for example about 100 microseconds (μs). In alternative embodiments, the readout interval may have a different duration.

[0039] The controller 130 controls the illuminator assembly 110 and the camera assembly 120. For example, the controller 130 provides a lighting instruction to the lighting assembly 110, and the lighting assembly 110 projects light 170 according to the lighting instruction. The controller 130 may also provide an imaging instruction to the camera assembly 120, and the camera assembly 120 performs exposure and reads out image data according to the imaging instruction.

[0040] The controller 130 also uses the image data from the camera assembly 120 to determine depth information. For example, the controller 130 can generate a depth image based on the image data. The depth image includes a plurality of depth pixels. Each depth pixel has a value corresponding to an estimated depth, for example, an estimated distance from a light point of the object 140 to the image sensor 195. A single depth image may also be referred to as a depth frame or a depth map. In embodiments where the illuminator assembly 110 projects a continuous wave of modulated light, the controller 130 may determine depth information based on a phase shift between the light 170 projected by the illuminator assembly 110 and the reflected light 180. In embodiments where the camera assembly 120 reads out image data corresponding to different phase offsets of the modulated light, the controller 130 may perform phase unwrapping to determine depth information. In some embodiments (e.g., embodiments where the illuminator assembly 110 projects multiple cycles of modulated light), the controller 130 may generate multiple depth frames.

[0041] The controller 130 may also generate a brightness image corresponding to the depth image. The camera assembly 120 may generate image data of the brightness image and image data of the depth image from the same light (e.g., reflected light 180). In some embodiments, the brightness image and the depth image are generated simultaneously. For example, the camera assembly 120 may read out image data of the brightness image and image information of the depth image simultaneously. In one embodiment, the image data for the brightness image and the image for the depth image are the same image data. The brightness image may include a plurality of brightness pixels. Each brightness pixel has a value corresponding to light intensity (e.g., IR intensity). The brightness pixels in the brightness image may correspond to the depth pixels in the depth image. The brightness pixels and the depth pixels may be generated by light reflected from the same track of the object 140. For example, the brightness pixels and the depth pixels may be generated based on a signal from the same pixel 197 of the image sensor 1195, and the pixel 197 captures light reflected from the track of the object 140.

[0042] The controller 130 may further enhance the depth estimate in the depth image by applying a guided filter to the depth image using the corresponding luminance image as a reference. The luminance image may display one or more boundaries of the object 140 more clearly than the depth image. For example, one or more depth pixels representing at least a portion of the boundary of the object 140 may be invalid. The boundary may be an edge of the object 140. Alternatively, the boundary is a boundary between two areas of the object 140 having different reflectivity characteristics, such as a boundary between a fluorescent strip having relatively high reflectivity and a low reflectivity surface. The invalid depth pixel is the result of an invalid depth estimate, and the difference between the value of the invalid depth pixel and the ground truth depth value may exceed a threshold, such as 5%, 10%, or other percentage of the ground truth depth. The controller 130 may utilize more accurate information of the boundary of the object 140 in the luminance image to generate an enhanced depth image that includes a better depth estimate than the original depth image. The following is combined with Figure 4 Certain aspects of controller 130 are described.

[0043] Modulation Signal Example

[0044] Figure 2A 2 shows a continuous wave of a projection signal 210 according to some embodiments of the present disclosure. The projection signal 210 is a modulated signal, for example, Figure 1 The projection signal 210 has a sinusoidal modulation waveform. Figure 2A In the embodiment, the sinusoidal modulation waveform can be represented by the following equation:

[0045] S(t)=A s sin(2πft)+B s

[0046] Where t represents time, S represents the optical power of the projection signal, f is the frequency of the modulation signal (i.e., the modulation frequency), π is a mathematical constant, and A s represents the amplitude of the modulated signal, B s Represents the offset of the modulated signal, which may include the attenuated original offset and / or the offset due to the presence of ambient light (such as sunlight or light from artificial sources).

[0047] Figure 2B 2 shows a continuous wave of a captured signal 220 according to some embodiments of the present disclosure. The captured signal 220 is a captured portion of the modulated light reflected by an object illuminated by the projection signal 210. The captured signal 220 may be captured by the image sensor 195 and may be Figure 1 At least a portion of the reflected light 180 in Figure 2A In the embodiment of FIG. 2 , the captured signal 220 may be represented by the following equation:

[0048]

[0049]

[0050]

[0051] Where r represents the optimal power of the captured signal 220, α represents the attenuation factor of the captured signal 200, represents the phase shift between the waveform of the captured signal 220 and the waveform of the projected signal 210, δ is the time delay between the captured signal 210 and the projected signal 220, d represents the distance from the image sensor 195 to the object 140 (ie, the depth of the object 140), and c is the speed of light.

[0052] Figure 3 A cycle 310 of continuous waves 315A-C, 325A-C, and 335A-C of modulated light is shown according to some embodiments of the present disclosure. Figure 3 In the embodiment, continuous waves 315A-C, 325A-C, and 335A-C are sinusoidal waves. In other embodiments, continuous waves 315A-C, 325A-C, and 335A-C may have different waveforms. Continuous waves 315A-C have a frequency 317, continuous waves 325A-C have a frequency 327, and continuous waves 335A-C have a frequency 337. The three frequencies 317, 327, and 337 are different from each other. Figure 3 In the embodiment of the present invention, frequency 317 is less than frequency 327, and frequency 327 is less than frequency 337. The three frequencies 317, 327, and 337 may be in the range of 50 to 200 MHz or higher.

[0053] In one embodiment, cycle 310 may be Figure 1 In another embodiment, period 310 may be an exposure period of camera assembly 120. For example, camera assembly 120 may expose during time periods of continuous waves 315A-C, 325A-C, and 335A-C, and may not expose outside of these time periods, although illuminator assembly 110 may project modulated light outside of these time periods. In another embodiment, period 310 may be a readout period of camera assembly 120. For example, camera assembly 120 may read out charge accumulated in image sensor 195 during time periods of continuous waves 315A-C, 325A-C, and 335A-C, and may not read out charge exceeding these time periods, although illuminator assembly 110 may project modulated light outside of these time periods, or image sensor 195 may expose outside of these time periods. Although Figure 3The cycle 310 in includes three continuous waves for each of the three frequencies, but cycles in other embodiments may include a different number of frequencies or a different number of continuous waves for each frequency.

[0054] exist Figure 3 , continuous waves 315A-C have different phase offsets. For example, continuous wave 315A has a phase offset of 0°, continuous wave 315B has a phase offset of 120°, and continuous wave 315C has a phase offset of 240°. Similarly, continuous waves 325A-C may have different phase offsets from each other: continuous wave 325A may have a phase offset of 0°, continuous wave 325B may have a phase offset of 120°, and continuous wave 325C may have a phase offset of 240°; and continuous waves 335A-C may start with different phase offsets from each other: continuous wave 335A may have a phase offset of 0°, continuous wave 335B may have a phase offset of 120°, and continuous wave 335C may have a phase offset of 240°. The phases of continuous waves 315A, 325A, and 335A may be between 0° and 120°, respectively, the phases of continuous waves 315B, 325B, and 335B may be between 120° and 240°, and the phases of continuous waves 315C, 325C, and 335C may be between 240° and 360°, respectively. In some embodiments, the duration of each continuous wave is about 100 μs. The time interval between two adjacent continuous waves may be in the range of 1 to 2 ms.

[0055] In other embodiments, cycle 310 may not have multiple continuous waves for each frequency. Instead, cycle 310 has a single continuous wave for a single frequency. For example, cycle 310 may include a first continuous wave for frequency 317, a second continuous wave for frequency 327, and a third continuous wave for frequency 337. The first, second, and third continuous waves may all start from 0°. There may be a time gap between two adjacent continuous waves of the first, second, and third continuous waves. Cycle 310 may generate image data for controller 130 to generate frames. Cycle 310 may be repeated so that controller 130 generates more frames. Controller 130 may perform phase unwrapping to determine depth information.

[0056] Controller Example

[0057] Figure 4 4 is a block diagram illustrating a controller 130 according to some embodiments of the present disclosure. The controller 130 includes a database 410, an illuminator module 420, a camera module 430, a depth module 440, a brightness module 450, and a depth enhancement module 460. These modules are software modules implemented on one or more processors, dedicated hardware units, or some combination thereof. Some embodiments of the controller 130 have different Figure 4Similarly, the components described together Figure 4 The functionality of the components described can be combined with Figure 4 The different ways described are distributed among other components. For example, some or all of the functions described as being performed by the controller 130 may be performed by a device incorporating a depth estimation system, such as Figure 6 System 600, Figure 7 The mobile device 700, Figure 8 Entertainment system 800, Fig. 9 The robot 900 or other device in the embodiment of the present invention.

[0058] Database 410 stores data generated and / or used by controller 130. Database 410 is a memory, such as ROM, DRAM, SRAM, or some combination thereof. Database 410 may be part of a larger digital memory of a depth estimation system, such as depth estimation system 100, or may be part of a device that includes a depth estimation system. In some embodiments, database 410 stores image data from camera assembly 120, depth images generated by depth module 440, brightness images generated by brightness module 450, enhanced depth images generated by depth enhancement module 460, baseline data describing a baseline trained or established before depth measurement from calibration module 470, and the like. In some embodiments, database 410 may store calibration data and / or other data from other components, such as depth instructions. Depth instructions include illuminator instructions generated by illuminator module 420 and camera instructions generated by camera module 430.

[0059] The illuminator module 420 controls the illuminator assembly 110 via illuminator instructions. The illuminator instructions include one or more illumination parameters that control how the illuminator assembly 110 projects light. The illumination parameters may describe, for example, a waveform, a wavelength, an amplitude, a frequency, a phase shift, a start time of each continuous wave, an end time of each continuous wave, a duration of each continuous wave, some other parameter that controls how the illuminator assembly 110 projects light, or some combination thereof. The illuminator module 420 may retrieve the illuminator instructions from the database 350. Alternatively, the illuminator module 420 generates the illuminator instructions. For example, the illuminator module 420 determines one or more illumination parameters. In an embodiment where the illuminator assembly 110 includes multiple modulated light projectors, the illuminator module 420 may determine separate illumination parameters for different light projectors.

[0060] The camera module 430 controls the camera assembly 120 via camera instructions. The camera module 430 can retrieve the camera instructions from the database 410. Alternatively, the camera module 430 generates the camera instructions based in part on the illuminator instructions generated by the illuminator module 420. The camera module 430 determines the exposure parameters (such as the start time, the end time, or the duration of the exposure interval, etc.) of the camera assembly 120 based on one or more lighting parameters (such as the duration of the continuous wave, etc.) specified in the illuminator instructions. For example, the camera module 430 determines that the duration of the exposure is equal to the duration of the continuous wave. Sometimes the camera module 430 determines that the duration of the exposure is longer than the duration of the continuous wave to avoid the inability to collect the entire continuous wave due to the delay of the incident light. The duration of the exposure can be 20% longer than the duration of the continuous wave. In some embodiments, the camera module 430 also determines the number of exposure intervals for each continuously modulated light wave projected by the lighting assembly 110.

[0061] The camera instructions may include readout instructions for controlling the readout of the camera assembly 120. The camera module 430 may determine readout parameters (e.g., a start time, an end time, or a duration of a readout interval, etc.) of the camera assembly 120. For example, the camera module 430 determines the start time of each of the one or more readout intervals, for example, based on one or more illumination parameters (e.g., phase, waveform, start time, or other parameters of the continuous wave). The imaging module 630 may also determine the duration of each readout interval, the number of readout intervals of the continuous wave, the time interval between adjacent readout intervals, the number of readout cycles, other readout parameters, or some combination thereof.

[0062] The depth module 440 is configured to generate a depth image indicating the distance to the imaged object 140, for example, based on a digital signal indicating the charge accumulated on the image sensor 195. The depth module 440 may analyze the digital signal to determine the phase shift exhibited by the light (e.g., the phase shift described above in connection with FIG. 2). ) to determine the ToF of the light (eg, the ToFδ described above in conjunction with FIG. 2 ), and further determine the depth value of the object 140 (eg, the distance described above in conjunction with FIG. 2 ).

[0063] In embodiments where illumination assembly 110 projects multiple continuous waves with different phase offsets, depth module 440 may generate a depth image by phase unwrapping. Figure 3Taking the period 310 in as an example, the depth module 440 can determine the wrapped distance, each wrapped distance corresponding to the corresponding phase. The depth module 440 can also estimate the unwrapped depth of each wrapped distance. The depth module 440 also determines the Voronoi vector corresponding to the unfolded depth and generates a lattice of Voronoi cells. Each unfolded depth corresponds to a Voronoi cell of the lattice. In an alternative embodiment, the depth module 440 is configured to determine the depth information using the charge ratio between the storage areas associated with each photodiode of the camera assembly 120.

[0064] The brightness module 450 generates a brightness image, such as an active brightness image. In some embodiments, for the depth image generated by the depth module 440, the brightness module 450 generates a corresponding brightness image. The brightness module 450 can generate a brightness image based on a request for a brightness image from the depth enhancement module 460. The brightness module 450 can generate a depth image based on the phase shift between the first captured light and the projected light, and can generate a corresponding brightness image based on the intensity or amplitude of the second captured light. In some embodiments, the first captured light and the second captured light are the same light. In other embodiments, the second captured light is different from the first captured light. For example, the first captured light may be IR, and the second captured light may be visible light.

[0065] In an embodiment where the depth module 440 generates a depth image based on the charge accumulated in a group of pixels 197 of the image sensor 195, the brightness module 450 can generate a corresponding brightness image based on the charge accumulated in all or some of the pixels 197 in the group. The corresponding brightness image includes a plurality of brightness pixels. Each brightness pixel can correspond to a depth pixel in the depth image. For example, the values ​​of the depth pixel and the corresponding brightness pixel can be determined based on the charge accumulated in the same pixel 197 in the image sensor 195. The charge accumulated in the pixel 197 can be converted from photons of modulated light reflected by the trajectory of the object 140. The value of the depth pixel can be determined by the depth module 440 based on the phase shift in the waveform of the modulated light. The value of the corresponding brightness pixel can be determined based on the accumulated charge in the pixel 197 or a different pixel 197.

[0066] The depth enhancement module 460 enhances the depth estimation performed by the depth module 440 based on the brightness image generated by the brightness module 450. For example, the depth enhancement module 460 may determine that the depth image includes one or more invalid depth pixels. An invalid depth pixel refers to a depth pixel having a value indicating an invalid depth measurement. In response to determining that the depth image includes one or more invalid depth pixels, the depth enhancement module 460 may retrieve a corresponding brightness image generated by the brightness module 450, or instruct the brightness module 450 to generate a corresponding brightness image.

[0067] The depth enhancement module 460 also uses a guided filter to fuse the depth image and the corresponding brightness image to enhance the depth estimation. The depth enhancement module 460 can use a guided filter to identify one or more invalid depth pixels in the depth image. In some embodiments, the depth enhancement module 460 uses a guided filter to identify one or more depth pixels on the boundary of the object 140, and treats the one or more depth pixels on the boundary as invalid pixels. In other embodiments, the depth enhancement module 460 can compare the depth value of the pixel in the depth image with the thickness value of the pixel determined by the guided filter, and determine whether the pixel is invalid based on the comparison result. For example, in response to determining that the difference between the depth value in the depth image and the depth value from the guided filter exceeds a threshold (e.g., a threshold absolute value or a threshold percentage), the depth enhancement module 460 can determine that the pixel is invalid.

[0068] In one example, the depth enhancement module 460 uses the depth image as the input image of the guide filter and uses the brightness image as the guide image of the filter. The depth enhancement module 460 can define the guide filter based on the assumption that there is a local linear model between the guide image and the filter output (i.e., the output of the guide filter). The local linear model can be expressed as:

[0069]

[0070] where a k and b k is a weight (also called a linear coefficient), assuming that in a box w centered at the pixel with index k and including one or more other pixels k is constant, i is the frame w k The index of the pixel in , I is the guide, and q is the filter output. This model ensures that when the guidance is bounded, the filter output is also bounded, and when the guidance is unbounded, the filter output is also unbounded.

[0071] In some embodiments, the depth enhancement module 460 identifies a target pixel with a pixel index k. The value of the target pixel k in the depth image is the input pixel index k. k The value of the target pixel k in the luminance image is the guide I k The depth enhancement module 460 also defines a box w centered at the target pixel k k , and confirm the box w k The average value of the pixels in the depth image box w k The average value of the inner pixels is p, and the average value in the brightness image is I. The depth enhancement module 460 can also determine a regularization coefficient ε. The regularization coefficient ε may penalize a larger weight a k The depth enhancement module 460 may determine the weights a and b based on the following cost function:

[0072]

[0073] The depth enhancement module 460 may also perform linear regression to obtain:

[0074]

[0075]

[0076] Where cov represents covariance and var represents variance. The weight a is calculated by using the covariance and variance of the kernel calculated using the box blur. In an embodiment, ∈=0, In an embodiment, ∈=∞, a=0.

[0077] The depth enhancement module 460 may determine that the target pixel k represents at least a trajectory of a boundary of the object 140 based on the weight a having a value of 1 or substantially similar to 1. The depth enhancement module 460 may determine that the target pixel k represents at least a trajectory of a flat plane of the object 140 based on the weight a having a value of 0 or substantially similar to 0. The plane is a portion of the object 140 that does not include any portion of any boundary of the object 140. The depth enhancement module 460 determines the new value q of the target pixel k based on the following equation k :

[0078] q k =aI k +b

[0079] In embodiments where a=1 and target pixel k is considered to represent at least one trajectory of the boundary of object 140, may be equal to or similar to I, so the new value q k Possibly with I k In embodiments where a=0 and target pixel k is considered to represent at least the trajectory of the plane of object 140, That is, the average depth value of the pixels in the box.

[0080] The depth enhancement module 460 may determine a weight a for all or some pixels in the depth image. As described above, in response to determining that a pixel represents at least a trajectory of a boundary, the depth enhancement module 460 determines a new depth value for the pixel based on the pixel value in the luminance image. In response to determining that the pixel does not represent any trajectory of any boundary, the depth enhancement module 460 generates a new depth value for the pixel based on the depth values ​​of other pixels in a box centered on the pixel. The depth enhancement module 460 may generate an enhanced depth image with a new depth value. For example, the depth enhancement module 460 may replace the value of the depth pixel with a new depth value. The enhanced depth image represents a better depth estimate, particularly for one or more boundaries of the object 140.

[0081] Depth Estimation Enhancement Example

[0082] Figure 5A An example depth image 510 according to some embodiments of the present disclosure is shown. The depth image 510 may be obtained by combining the above Figure 4 The depth module 440 generates the described. Figure 5B The present invention shows some embodiments of the present invention. Figure 5A Example brightness image 520 corresponding to the depth image 510 in FIG. The brightness image 520 can be obtained by combining the above Figure 4 The brightness module 450 described above generates. Figure 5C According to some embodiments of the present disclosure, Figure 5A The depth image 510 and Figure 5B Example depth enhanced image 530 generated from brightness image 520 in FIG.

[0083] The three images 510, 520 and 530 capture an object, an example of which is Figure 1 The object 140 has a flat surface 540 and an edge 550. Figures 5A-5C In the example, part of the edge is enclosed in a dotted ellipse. Figure 5A and 5B As shown, the brightness image 520 shows clearer edges than the depth image 510. Therefore, the brightness image 520 can be used to enhance the depth estimation of the object, especially the edge 550 of the object. The brightness image 520 is fused with the depth image 510 through the guided filter to generate the depth enhanced image 530. The depth enhanced image 530 shows a better depth estimation than the depth image 510. Figure 5A and 5C As shown, the edges in the depth enhanced image 530 are clearer than those in the depth image 510. The depth enhanced image 530 can be generated by using a guided filter, where the depth image 510 is the input image of the guided filter and the brightness image 520 is the guided image of the guided filter. The depth enhanced image 530 can be generated by the above-mentioned depth enhancement process in combination with the depth enhancement module 460. For example, the pixels representing the edge 550 are identified, and the new depth values ​​of the pixels are determined based on the pixel values ​​in the brightness image 520. The new depth values ​​of the pixels are used to generate the depth enhanced image 530.

[0084] Figure 5A and 5C It is also shown that plane 540 is smoother in depth enhanced image 530. Figures 5A-5CIn some embodiments, pixels representing plane 540 are also identified. For each of these pixels, a box is defined. The box can be centered on the pixel and include other pixels surrounding the pixel. A new depth value for the pixel is determined based on the depth values ​​of the other pixels in the box. For example, the new depth value for the pixel can be the average of the depth values ​​of the other pixels. The new depth value for the pixel is used to generate a depth enhanced image 530. Through such a depth enhancement process, the depth enhanced image 530 represents a more accurate depth estimate of the object, such as a more accurate depth estimate of the plane 540, the edge 550, or both.

[0085] Example Applications Incorporating Depth Estimation Systems

[0086] Figure 6 An example system 600 including a depth estimation system according to some embodiments of the present disclosure is shown. One embodiment of the depth estimation system is Figure 1 1. The depth estimation system 100 in FIG. 1 includes an imaging device 610, a processor 620, a memory 630, an input device 640, an output device 650, and a battery / power circuit 660. In other embodiments, the system 600 may include fewer, more, or different components. For example, the system 600 may include multiple processors, memories, display devices, input devices, or output devices.

[0087] Imaging device 610 captures depth images and brightness images. Imaging device 610 may include an illuminator assembly, such as illuminator assembly 110, for projecting light into the environment surrounding system 600. Imaging device 610 may project modulated light, such as pulsed modulated light or a continuous wave of modulated light. Imaging device 610 also includes a camera assembly, such as camera assembly 120, which captures light reflected by one or more objects in the environment and generates image data of the one or more objects.

[0088] The processor 620 may process electronic data from registers and / or memories to convert the electronic data into other electronic data that may be stored in registers and / or memories. The processor 620 may perform some or all functions of some or all components of the controller 130, such as depth estimation, depth estimation enhancement using brightness signa, etc. The processor 620 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), CPUs, GPUs, cryptographic processors (special processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices.

[0089] In some embodiments, the processor 620 may also use the depth information (e.g., an enhanced depth image) to generate content (e.g., images, audio, etc.) for display to a user of the system through one or more display devices (e.g., output device 650). The content may be used as VR, AR, or MR content. The processor 620 may also generate instructions for other components of the system 600 or another system based on the enhanced depth image. For example, the processor 620 may determine navigation instructions for a movable device, such as a robot, vehicle, or other type of movable device. The navigation instructions may include navigation parameters (e.g., navigation route, speed, direction, etc.).

[0090] The memory 630 may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile storage (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 630 may include a memory that shares a die with the processor 620. The memory 630 may store processor-executable instructions for controlling the operation of the depth estimation system 100 and / or data captured by the depth estimation system 100. In some embodiments, the memory 630 includes one or more non-transitory computer-readable media storing instructions executable to perform a depth estimation enhancement process, such as the following in conjunction with Fig.10 The method 1000 described above, or a combination thereof Figure 1 and Figure 4 The operations described are performed by the controller 130 (or some components of the controller 130 ). Instructions stored in one or more non-transitory computer-readable media may be executed by the processor 620 .

[0091] Input device 640 may include an audio input device. Audio input device 1318 may include any device that generates a signal representing sound, such as a microphone, a microphone array, a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output), etc. Input device 640 may also include one or more other types of input devices, such as an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (such as a mouse), a stylus, a touchpad, a barcode reader, a quick response (QR) reader, a sensor, a radio frequency identification (RFID) reader, etc.

[0092] Output device 650 may include one or more display devices, such as one or more visual indicators. Example visual indicators include a head-up display, a computer display, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display, etc. Output device 650 may also include an audio output device. An audio output device may include any device that produces a sound indicator, such as a speaker, a headset, or an earplug, etc. Output device 650 may also include one or more other output devices, such as an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, etc.

[0093] Battery / power circuit 660 may include one or more energy storage devices (eg, batteries or capacitors) and / or circuits for coupling components of system 600 to an energy source separate from system 600 (eg, AC line power).

[0094] Figure 7 A mobile device 700 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. An example of a depth estimation system is Figure 1 The depth estimation system 100 in FIG. 7 is a mobile device 700 that can be a mobile phone. Figure 7 As shown, mobile device 700 includes imaging component 702. The imaging device may include Figure 1 The illuminator assembly 110 and the camera assembly 120 in the embodiment of the present invention. The imaging assembly 702 can illuminate the environment around the robot 902 with modulated light (e.g., modulated IR) and capture images of one or more objects in the environment. Figure 7 Not shown, the mobile device 700 may also include one or more processors and one or more memories that may perform some or all of the functions of the controller 130. Through these components, the mobile device 700 may determine depth information of one or more objects in the environment surrounding the mobile device 700. The mobile device 700, another device, or a user of the mobile device may use the depth information for various purposes, such as VR, AR, or MR applications, navigation applications, and the like. For example, the mobile device 700 may generate and present images (two-dimensional or three-dimensional images) based on the depth information of the environment, and the images may represent virtual objects that do not exist in the real-world environment. These images may enhance the real-world objects in the environment so that the user of the mobile device 700 may have an interactive experience with the real-world environment in which real-world objects residing in the real world are enhanced by computer-generated virtual objects.

[0095] Figure 8 An entertainment system 800 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. Figure 8In the example of , user 808 can interact with the entertainment system via controller 810, such as playing a video game. Entertainment system 800 includes console 802 and display 804. Console 802 can be a video game console that is configured to generate images of a video game on display 804. In other embodiments, entertainment system 800 can include more, fewer, or different components.

[0096] The console 802 includes an imaging assembly 806. The imaging assembly 806 may include Figure 1 The imaging component 806 can illuminate the environment around the entertainment system 800 with modulated light (e.g., modulated IR) and capture the modulated light reflected by one or more objects in the environment to generate an image of the object, such as a user 808, a controller 810, or other object. Figure 8 800 . Although not shown, the console 802 may also include one or more processors and one or more memories that may perform some or all of the functions of the controller 130 . The console 802 may determine depth information of one or more objects in the environment. The depth information may be used to present images to the user on the display 804 , or to control certain other aspects of the entertainment system 800 . For example, the user 808 may control the entertainment system 800 with gestures, and the gestures may be determined at least in part by the depth information. The console 802 may generate or update display content (e.g., images, audio, etc.) based on the depth information, and may also instruct the display 804 to present the display content to the user 808 .

[0097] Fig. 9 An example robot 902 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. The robot 902 includes an imaging component 904, which may include Figure 1 The illuminator assembly 110 and the camera assembly 120 in the embodiment of the present invention. The imaging assembly 904 can illuminate the environment around the robot 902 with modulated light (e.g., modulated IR) and capture images of one or more objects in the environment. Fig. 9 Not shown, the robot 902 may also include a computing device that can perform some or all of the functions of the controller 130. The computing device may include one or more processors and one or more memories. The computing device may determine depth information of one or more objects in the environment. The robot 902 may be mobile, and the computing device may use the depth information to assist in navigation and / or motor control of the robot 902. For example, the computing device may include navigation instructions based on the depth information. The navigation instructions may include a navigation route for the robot 902. The robot 902 may navigate in the environment according to the navigation instructions.

[0098] Apart from Figure 7-9Examples of uses of the technology described herein are possible beyond those shown. For example, the depth estimation system described herein can be used in other applications such as self-driving cars, security cameras, etc.

[0099] Example method for enhancing depth estimation using guided filters

[0100] Fig.10 1 is a flow chart illustrating a method 1000 for enhancing depth estimation using a guided filter according to some embodiments of the present disclosure. The method 1000 may be performed by the controller 130. Fig.10 The flowchart shown in describes method 1000, but many other methods of using guided filters to enhance depth estimation may also be used. For example, Fig.10 The execution order of the steps in the method can be changed. As another example, some steps can be changed, eliminated or combined.

[0101] In 1010, the controller 130 determines the weight of the target pixel of the depth image of the captured object based on the depth image and the brightness image of the captured object. The depth image and the brightness image can be generated based on image data from the same image sensor. The depth image can be an input image of the guided filter, and the brightness image can be a guided image of the guided filter.

[0102] In some embodiments, the controller 130 instructs the illuminator assembly to project modulated light into a local area including an object. The modulated light may be IR modulated. The modulated light may include one or more continuous waves. In some embodiments, such as embodiments in which the modulated light includes multiple continuous waves, the continuous waves may have different frequencies, phase offsets, or both. The controller 130 also instructs the camera assembly to capture reflected light from at least a portion of the object. In some embodiments, the controller 130 instructs the camera assembly to expose within multiple exposure intervals. For example, the controller 130 instructs the camera assembly to perform multiple exposure intervals on a single continuous wave of modulated light, each exposure interval may correspond to a different phase offset. The controller 130 may generate a depth image based on the phase shift between the reflected light and the modulated light projected into the local area. The controller 130 may also generate a brightness image based on the brightness of the reflected light.

[0103] In 1020, the controller 130 determines whether the target pixel represents at least a portion of a boundary of the object based on the weight. In some embodiments, the controller 130 determines whether the weight value is equal to or substantially similar to 1, and in response to determining that the weight value is equal to or substantially similar to 1, the controller 130 determines that the target pixel represents at least a portion of the boundary of the object. In some embodiments, the controller 130 determines whether the weight value is equal to or substantially similar to 0, and in response to determining that the weight value is equal to or substantially similar to 0, the controller 130 determines that the target pixel represents at least a portion of the boundary of the object. For example, the controller 130 may determine that the target pixel represents a portion of a plane of the object.

[0104] In response to determining that the target pixel represents at least a portion of the boundary, in 1030, the controller 130 determines the depth value of the target pixel based on the brightness value of the corresponding pixel in the brightness image. The depth value of the target pixel may be independent of the value of the target pixel in the depth image, i.e., the original depth value of the target pixel. In some embodiments, in response to determining that the target pixel does not represent any part of the boundary, the controller 130 determines the depth of the target pixel based on the depth values ​​of other pixels in the depth image. The other pixels surround the target pixel. The depth value of the target pixel is the average of the depth values ​​of the other pixels. The controller 130 may identify the other pixels by defining a box in the depth image. The box may be located at the center of the target pixel. The other pixels are in the box. For example, the controller 130 may define a 3×3 box centered on the target pixel. The box also includes eight other pixels adjacent to and surrounding the target pixel. The depth value of the target pixel may be the average of the depths of the eight pixels in the depth image.

[0105] In 1040, the controller 130 updates the depth image by assigning a depth value to the target pixel. The controller 130 may generate an enhanced depth image based on the depth value. The enhanced depth image represents a better depth estimate than the depth image because the depth value of the target pixel in the enhanced depth image has better accuracy than the thickness value of the target pixel in the depth image.

[0106] Changes and implementation

[0107] Although the above reference Figure 1-10 The illustrated exemplary implementations describe embodiments of the present disclosure, but those skilled in the art will appreciate that the various teachings described above are applicable to a variety of other implementations.

[0108] In some cases, the features discussed herein may be applicable to automotive systems, safety-critical industrial applications, medical systems, scientific instrumentation, wireless and wired communications, radio, radar, industrial process control, audio and video equipment, current sensing, instrumentation (which may be high precision), and other digital processing-based systems.

[0109] In the discussion of the above embodiments, components of the system, such as filters, converters, mixers, amplifiers, digital logic circuits and / or other components, can be easily replaced, substituted or otherwise modified to suit specific circuit requirements. In addition, it should be noted that the use of complementary electronic devices, hardware, software, etc. in various communication systems provides equally viable options for implementing the teachings of the present disclosure related to fractional frequency dividers.

[0110] As described herein, portions of various systems for implementing a balanced fractional divider for duty cycle regulation may include electronic circuits that perform the functions described herein. In some cases, one or more portions of the system may be provided by a processor specifically configured to perform the functions described herein. For example, the processor may include one or more dedicated components, or may include programmable logic gates configured to perform the functions described herein. The circuit may operate in an analog domain, a digital domain, or a mixed signal domain. In some cases, the processor may be configured to perform the functions described herein by executing one or more instructions stored on a non-transient computer readable storage medium.

[0111] In an example embodiment, any number of circuits of this figure can be implemented on a board of an associated electronic device. The board can be a general circuit board that can accommodate various components of the electronic system inside the electronic device and can also provide connectors for other peripheral devices. More specifically, the board can provide an electrical connection through which other components of the system can communicate electrically. Any suitable processor (including DSP, microprocessor, supporting chipset, etc.), computer-readable non-transient storage element, etc. can be appropriately coupled to the board according to specific configuration requirements, processing requirements, computer design, etc. Other components, such as external memory, additional sensors, audio / video display controllers, and peripherals, can be connected to the board as plug-in cards via cables or integrated into the board itself. In various embodiments, the functions described herein can be implemented in simulated form as software or firmware running in one or more configurable (e.g., programmable) elements arranged in a structure that supports these functions. The software or firmware providing the simulation can be provided on a non-transient computer-readable storage medium that includes instructions that allow the processor to perform these functions.

[0112] In another example embodiment, the circuit of the present figure can be implemented as a stand-alone module (e.g., a device having related components and circuits configured to perform a specific application or function), or as a plug-in module inserted into dedicated hardware of an electronic device. Note that certain embodiments of the present disclosure can be easily included in part or in whole in a system-on-chip (SOC) package. SOC represents an IC that integrates the components of a computer or other electronic system into a single chip. It may contain digital, analog, mixed-signal, and radio frequency (RF) functions: all of which can be provided on a single chip substrate. Other embodiments may include a multi-chip module (MCM), in which multiple separate ICs are located within a single electronic package and are configured to interact closely with each other through the electronic package.

[0113] It is also important to note that all specifications, dimensions, and relationships outlined herein (e.g. Figure 1 -2, 4-5, 7 and 9-10) are for example and teaching purposes only. Such information may be significantly changed without departing from the spirit of the present disclosure or the scope of the appended claims. It should be understood that the system can be integrated in any suitable manner. Along similar design alternatives, any of the circuits, components, modules and elements shown in this figure can be combined in various possible configurations, all of which are clearly within the broad scope of this specification. In the foregoing description, example embodiments have been described with reference to specific processors and / or component arrangements. Various modifications and changes may be made to these embodiments without departing from the scope of the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0114] Note that through many of the examples provided herein, interactions may be described in terms of two, three, four or more electrical components. However, this is done for clarity and example only. It should be understood that the system may be integrated in any suitable manner. Along similar design alternatives, any components, modules, and elements shown in the figures may be combined in a variety of possible configurations, all of which are clearly within the broad scope of this specification. In some cases, it may be easier to describe one or more functions of a given set of flows by referencing only a limited number of electrical components. It should be understood that the drawings and the circuits they teach are easily expandable and can accommodate a large number of components and more complex / complex arrangements and configurations. Therefore, the examples provided should not limit the scope of the circuit or inhibit its broad teachings that may be applied to countless other architectures.

[0115] Note that in this specification, references to various features (e.g., elements, structures, modules, components, steps, operations, characteristics, etc.) included in "one embodiment," "example embodiment," "specific embodiment," "another embodiment," "some embodiments," "various embodiments," "other embodiments," or "alternative embodiments," etc., are intended to indicate that any such features are included in one or more embodiments of the present disclosure, but may or may not be combined in the same embodiment. In addition, as used herein, including in the claims, "or" used in a list of items (e.g., a list of items beginning with phrases such as "at least one" or "one or more") represents an inclusive list, for example, [at least one of A, B, or C] represents A or B or C or AB or AC or BC or ABC (i.e., A and B and C).

[0116] Various aspects of the illustrative embodiments are described using terms commonly used by those skilled in the art to convey the substance of their work to others skilled in the art. For example, the term "connected" refers to a direct electrical connection between the things being connected, without any intermediate devices / components, while the term "coupled" refers to a direct electrical connection between the connected objects, or an indirect connection through one or more passive or active intermediate devices / components. In another example, the term "circuit" refers to one or more passive and / or active components that are arranged to cooperate with each other to provide the desired functionality. In addition, as used herein, based on the context of specific values ​​described herein or known in the art, the terms "substantially", "approximately", "about", etc. may be used to generally refer to within + / -20% of a target value, such as within + / -10% of a target value.

[0117] Numerous other variations, substitutions, changes, alterations, and modifications may be determined by those skilled in the art, and the present disclosure is intended to include all such variations, substitutions, and modifications that fall within the scope of the examples and the appended claims. Note that all optional features of the above-described devices may also be implemented with respect to the methods or processes described herein, and that the details in the examples may be used anywhere in one or more embodiments.

[0118] Explanation of terms

[0119] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms. Unless the context clearly requires otherwise, throughout the specification and claims:

[0120] The words “include”, “including”, etc. shall be construed as being inclusive rather than exclusive or exhaustive; that is, in the sense of “including but not limited to”.

[0121] “Connected,” “coupled,” or any variation thereof, refers to any direct or indirect connection or coupling between two or more elements; the coupling or connection between the elements may be physical, logical, or a combination thereof.

[0122] When used to describe this Specification, the words “herein,” “above,” “hereunder,” and words of similar import shall refer to this Specification as a whole and not to any particular portion of this Specification.

[0123] “Or” refers to a list of two or more items and covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0124] The singular forms "a", "an" and "the" are intended to include any appropriate plural forms.

[0125] Words indicating directions, such as "vertical," "lateral," "horizontal," "upward," "downward," "forward," "backward," "inward," "outward," "perpendicular," "lateral," "left," "right," "front," "backward," "top," "bottom," "below," "above," "lower," etc., used in this specification and any appended claims, if any, depend on the specific orientation of the device being described and illustrated. The subject matter described herein can assume a variety of alternative orientations. Accordingly, these directional terms are not strictly defined and should not be narrowly interpreted.

[0126] Unless expressly stated to the contrary, the indefinite articles "a," "an," and "an" used in the specification and claims should be understood to mean "at least one."

[0127] The phrase "and / or" used herein in the specification and claims should be understood to mean "one or both" of the elements so coupled, i.e., elements that are present in combination in some cases and separate in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so coupled.

[0128] In addition to the elements expressly identified by the "and / or" clause, other elements may optionally be present, whether or not related to those elements expressly identified. Thus, as a non-limiting example, in one embodiment, when used in conjunction with open language such as "comprising," a reference to "A and / or B" may include only A (optionally including elements other than B); in another embodiment only B (optionally including elements other than A); in another embodiment both A and B (optionally including other elements); etc.

[0129] As used herein in the specification and claims, the phrase "at least one" refers to a list of one or more elements and should be understood to refer to at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of each element specifically listed in the list of elements, nor exclude any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than the elements specifically identified in the list of elements to which the phrase "at least one" refers, whether or not these elements are related to those elements specifically identified.

[0130] Thus, as a non-limiting example, in one embodiment, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently "at least one of A and / or B") may refer to at least one, optionally including multiple A's, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in another embodiment, to at least one (optionally including more than one), A, and at least one, optionally including more than one B (and optionally including other elements); etc.

[0131] As used herein, the term "between" is intended to be inclusive unless otherwise stated. For example, "between A and B" includes A and B unless otherwise stated.

[0132] In addition, the words and terms used herein are for description only and should not be considered as limiting. The words "include", "comprising", or "having", "containing", "involving" and their variations used herein are intended to include the items listed below and their equivalents as well as additional items.

[0133] In the claims and the above description, all transitional phrases, such as "include", "comprising", "carrying", "having", "containing", "involving", "holding", "consisting of", etc., shall be understood as open-ended, i.e., including but not limited to. Only the transitional phrases "consisting of" and "mainly composed of" shall be closed or semi-closed transitional phrases, respectively.

[0134] Numerous variations, substitutions, alterations, and modifications may be identified by those skilled in the art, and this disclosure is intended to embrace all such variations, substitutions, alterations, and modifications that fall within the scope of the appended claims.

[0135] To assist the United States Patent and Trademark Office (USPTO) and any reader of any patent that issues from this application in interpreting the claims appended hereto, applicants wish to note that applicants: (a) do not intend to invoke 35 U.S.C. § 112(f) as it existed on the date of filing this application, unless the words “for” or “for” are specifically used in a particular claim; and (b) do not intend that any statement in this disclosure should limit the present disclosure in any manner not reflected in the appended claims.

[0136] Therefore, the present invention should not be considered to be limited to the specific embodiments described above. After reading this disclosure, those skilled in the art to which the present invention is directed will easily understand various modifications, equivalent processes, and numerous structures to which the present invention may be applicable.

[0137] Select Example

[0138] Example 1 provides a method, comprising: determining a weight of a target pixel of a depth image of a captured object based on a depth image and a brightness image of the captured object; determining whether the target pixel represents at least a portion of a boundary of the object based on the weight; in response to determining that the target pixel represents at least a portion of the boundary, determining a depth value of the target pixel based on a brightness value of a corresponding pixel in the brightness image; and updating the depth image by assigning the depth value to the target pixel.

[0139] Example 2 provides the method of Example 1, further comprising: in response to determining that the target pixel does not represent any part of the boundary, determining the depth value of the target pixel based on depth values ​​of other pixels in the depth image, wherein the other pixels surround the target pixel.

[0140] Example 3 provides the method of Example 2, wherein the depth value of the target pixel is a weighted average of the depth values ​​of the other pixels.

[0141] Example 4 provides the method of Example 1, wherein the depth image is used as an input image of a guided filter, and the luminance image is used as a guided image of the guided filter.

[0142] Example 5 provides the method of Example 1, wherein the depth image and the luminance image are generated based on image data from the same image sensor.

[0143] Example 6 provides the method of Example 1, further including: instructing the illuminator component to project modulated light into a local area including an object; instructing the camera component to capture reflected light from at least a portion of the object; and generating the depth image based on a phase shift between the reflected light projected into the local area and the modulated light.

[0144] Example 7 provides the method of Example 6, further comprising: generating the brightness image based on the brightness of the reflected light.

[0145] Example 8 provides the method of Example 6, wherein the modulated light comprises a continuous infrared wave, and the continuous wave has a different modulation frequency.

[0146] Example 9 provides the method of Example 1, wherein determining whether the target pixel represents at least a portion of the boundary of the object includes: determining whether the value of the weight is equal to or substantially similar to 1; and in response to determining that the value of the weight is equal to or substantially similar to 1, determining that the target pixel represents at least a portion of the boundary of the object.

[0147] Example 10 provides the method of Example 1, wherein determining whether the target pixel represents at least a portion of the boundary of the object includes: determining whether the value of the weight is equal to or substantially similar to 0; and in response to determining that the value of the weight is equal to or substantially similar to 0, determining that the target pixel does not represent any portion of the boundary of the object.

[0148] Example 11 provides a system comprising: an illuminator assembly configured to project modulated light into a local area including an object; a camera assembly configured to capture reflected light from at least a portion of the object; and a controller configured to: generate a depth image and a brightness image based on the reflected light, determine a weight of a target pixel of the depth image by applying a guided filter to the depth image and the brightness image, determine whether the target pixel represents at least a portion of a boundary of the object based on the weight, in response to determining that the target pixel represents at least a portion of the boundary, determine a depth value of the target pixel based on a brightness value of a corresponding pixel in the brightness image, and generate an enhanced depth image based on the depth value of the target pixel.

[0149] Example 12 provides the system of Example 11, further comprising: a display configured to present one or more images generated based on the enhanced depth image.

[0150] Example 13 provides the system of Example 11, wherein the controller is further configured to generate navigation instructions for a mobile device based on the enhanced depth image.

[0151] Example 14 provides the system of Example 11, wherein the controller is further configured to: in response to determining that the target pixel does not represent any portion of the boundary, determine the depth value of the target pixel based on depth values ​​of other pixels in the depth image, wherein the other pixels surround the target pixel.

[0152] Example 15 provides the system of Example 11, wherein the controller is configured to generate the depth image and the brightness image based on the reflected light by the following steps: generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and generating the brightness image based on the brightness of the reflected light.

[0153] Example 16 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising: determining a weight of a target pixel of a depth image of a captured object based on a depth image and a luminance image of the captured object; determining whether the target pixel represents at least a portion of a boundary of the object based on the weight; in response to determining that the target pixel represents at least a portion of the boundary, determining a depth value of the target pixel based on a luminance value of a corresponding pixel in the luminance image; and updating the depth image by assigning the depth value to the target pixel.

[0154] Example 17 provides one or more non-transitory computer-readable media of Example 16, wherein the operation further includes: in response to determining that the target pixel does not represent any portion of the boundary, determining the depth value of the target pixel based on depth values ​​of other pixels in the depth image, wherein the other pixels surround the target pixel.

[0155] Example 18 provides the one or more non-transitory computer-readable media of Example 16, wherein the depth image is used as an input image for a guided filter and the luminance image is used as a guided image for the guided filter.

[0156] Example 19 provides the one or more non-transitory computer-readable media of Example 16, wherein the depth image and the luminance image are generated based on image data from the same image sensor.

[0157] Example 20 provides one or more non-transitory computer-readable media of Example 16, wherein the operation further includes: instructing the illuminator component to project modulated light into a local area including an object; instructing the camera component to capture reflected light from at least a portion of the object; generating the depth image based on a phase shift between the reflected light projected into the local area and the modulated light; and generating the brightness image based on the brightness of the reflected light.

Claims

1. A method comprising: determining a weight of a target pixel of a depth image of the captured object based on the depth image and the brightness image of the captured object; determining, based on the weight, whether the target pixel represents at least a portion of a boundary of the object; In response to determining that the target pixel represents at least a portion of a boundary, determining a depth value for the target pixel based on a luminance value of a corresponding pixel in the luminance image; and The depth image is updated by assigning the depth value to the target pixel.

2. The method according to claim 1, further comprising: In response to determining that the target pixel does not represent any portion of the boundary, a depth value of the target pixel is determined based on depth values ​​of other pixels in the depth image, wherein the other pixels surround the target pixel. The method according to claim 2 , wherein the depth value of the target pixel is a weighted average of the depth values ​​of the other pixels.

4. A method according to any one of the preceding claims, wherein the depth image is used as an input image for a guided filter and the luminance image is used as a guiding image for the guided filter. 5 . The method according to claim 1 , wherein the depth image and the brightness image are generated based on image data from the same image sensor.

6. The method according to any one of the preceding claims, further comprising: instructing the illuminator assembly to project modulated light into a local area including the object; instructing a camera assembly to capture reflected light from at least a portion of the object; and The depth image is generated based on a phase shift between the reflected light and the modulated light projected into the local area.

7. The method according to claim 6, further comprising: The brightness image is generated based on the brightness of the reflected light.

8. A method according to claim 6 or claim 7, wherein the modulated light comprises a continuous infrared wave, and the continuous wave has a different modulation frequency.

9. The method of any one of the preceding claims, wherein determining whether the target pixel represents at least a portion of a boundary of the object comprises: determining whether the value of the weight is equal to or substantially similar to 1; as well as In response to determining that the value of the weight is equal to or substantially similar to 1, it is determined that the target pixel represents at least a portion of a boundary of the object.

10. The method of any one of the preceding claims, wherein determining whether the target pixel represents at least a portion of a boundary of the object comprises: determining whether the value of the weight is equal to or substantially similar to zero; and In response to determining that the value of the weight is equal to or substantially similar to zero, it is determined that the target pixel does not represent any portion of a boundary of the object.

11. A system comprising: an illuminator assembly configured to project modulated light into a localized area including the object; a camera assembly configured to capture reflected light from at least a portion of the object; as well as The controller is configured as: generating a depth image and a brightness image based on the reflected light, determining a weight of a target pixel of the depth image by applying a guided filter to the depth image and the luminance image, determining whether the target pixel represents at least a portion of a boundary of the object based on the weight, In response to determining that the target pixel represents at least a portion of a boundary, determining a depth value for the target pixel based on a luminance value of a corresponding pixel in the luminance image, and An enhanced depth image is generated based on the depth value of the target pixel. 12 . The system of claim 11 , further comprising a display configured to present one or more images generated based on the enhanced depth image.

13. The method of claim 11 or claim 12, wherein the controller is further configured to generate navigation instructions for a mobile device based on the enhanced depth image.

14. The system according to any one of claims 11 to 13, wherein the controller is further configured to: In response to determining that the target pixel does not represent any portion of the boundary, a depth value of the target pixel is determined based on depth values ​​of other pixels in the depth image, wherein the other pixels surround the target pixel.

15. The system according to any one of claims 11 to 14, wherein the controller is configured to generate the depth image and the brightness image based on the reflected light by the following steps: generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and The brightness image is generated based on the brightness of the reflected light.

16. One or more non-transitory computer-readable media storing instructions executable to perform operations comprising: determining a weight of a target pixel of a depth image of the captured object based on the depth image and the brightness image of the captured object; determining, based on the weight, whether the target pixel represents at least a portion of a boundary of the object; In response to determining that the target pixel represents at least a portion of a boundary, determining a depth value for the target pixel based on a luminance value of a corresponding pixel in the luminance image; and The depth image is updated by assigning the depth value to the target pixel.

17. The one or more non-transitory computer-readable media of claim 16, wherein the operations further comprise: In response to determining that the target pixel does not represent any portion of the boundary, a depth value of the target pixel is determined based on depth values ​​of other pixels in the depth image, wherein the other pixels surround the target pixel.

18. The one or more non-transitory computer-readable media of claim 16 or claim 17, wherein the depth image is used as an input image for a guided filter and the luminance image is used as a guiding image for the guided filter.

19. The one or more non-transitory computer-readable media of any one of claims 16 to 18, wherein the depth image and the luminance image are generated based on image data from the same image sensor.

20. The one or more non-transitory computer-readable media of any one of claims 16 to 19, wherein the operations further comprise: instructing the illuminator assembly to project modulated light into a local area including the object; instructing a camera assembly to capture reflected light from at least a portion of the object; generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and The brightness image is generated based on the brightness of the reflected light.