Enhancing depth estimation of luminance images with energy models

Abstract

A depth estimation system may use an image energy model to enhance depth estimation using a luminance image. The light is projected onto an object. The object reflects at least a portion of the projection light. The reflected light is at least partially captured by the image sensor. The depth estimation system may generate a depth image based on a phase shift between the captured light and the projected light, and generate a brightness image based on a brightness of the captured light. The depth estimation system may determine a fusion energy based on the depth image and the luminance image and minimize the fusion energy to determine a new depth value for the pixel. The depth estimation system may assign new depth values to the pixels and generate an enhanced depth image. The enhanced depth image includes a better depth estimate than the original depth image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 310,859, filed on February 16, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to depth estimation and, more particularly, to using energy models to enhance depth estimation of luminance images. Background Art

[0004] One technique for measuring depth is to directly or indirectly calculate the time it takes for a signal to travel from a source on a sensor to a reflecting surface and back to the sensor. The time of travel is proportional to the distance from the sensor to the reflecting surface. This travel time is often referred to as time of flight (ToF). ToF sensors can use various types of signals, the most common being sound and light. Given the advantages of light in terms of speed, range, power, and weight, some sensors use light as a carrier. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] For a more complete understanding of the present disclosure and its features and advantages, please refer to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts, and in which:

[0006] Figure 1 A depth estimation system according to some embodiments of the present disclosure is shown;

[0007] Figure 2A shows a continuous wave of a projection signal 210 according to some embodiments of the present disclosure;

[0008] Figure 2B shows a continuous wave of a capture signal 220 according to some embodiments of the present disclosure;

[0009] Figure 3 shows a period of a continuous wave of modulated light according to some embodiments of the present disclosure;

[0010] Figure 4A is a block diagram illustrating a controller according to some embodiments of the present disclosure;

[0011] Figure 4B is a block diagram illustrating a depth enhancement module according to some embodiments of the present disclosure;

[0012] Figure 5A shows an example depth image according to some embodiments of the present disclosure;

[0013] Figure 5B The present invention shows some embodiments of the present invention. Figure 5AExample brightness image corresponding to the depth image in;

[0014] Figure 5C According to some embodiments of the present disclosure, Figure 5A The depth image and Figure 5B Example depth-enhanced images generated from the brightness images in ;

[0015] Figure 6 An example system including a depth estimation system according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A mobile device incorporating a depth estimation system according to some embodiments of the present disclosure is shown;

[0017] Figure 8 An entertainment system incorporating a depth estimation system according to some embodiments of the present disclosure is shown;

[0018] Fig. 9 An example robot incorporating a depth estimation system according to some embodiments of the present disclosure is shown;

[0019] Fig.10 is a flow chart illustrating a method of enhancing depth estimation of luminance images using an energy model according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] Overview

[0021] Depth estimation is a fundamental task in 3D computer vision. The high-quality and dense depth images produced by ToF camera systems play an important role in many applications, such as robotics, human-computer interaction, indoor navigation, self-driving cars, object tracking, and gesture recognition. ToF camera systems are range imaging systems. ToF camera systems typically include a light source that projects light and an imaging sensor that receives reflected light. A ToF camera system can estimate the distance between the imaging sensor and the object by measuring the round trip of light. A continuous wave ToF camera system can project multiple cycles of a continuous light wave and determine the distance based on the phase difference between the projected light and the received reflected light. A depth image can be generated based on the phase difference.

[0022] However, ToF depth maps are usually captured with low resolution, different types of noise or missing values. ToF camera systems are usually unable to accurately estimate the depth of boundaries, such as the edge of an object, reflectivity boundaries (e.g., the boundary between two areas with different reflectivity characteristics), etc. Inaccurate depth estimation may limit the use of ToF camera systems in the above applications. Therefore, there is a need to improve depth estimation techniques.

[0023] Embodiments of the present disclosure relate to a depth estimation system that can use an energy model to enhance ToF depth estimation using a brightness image. The brightness image can be an active brightness image, such as an infrared (IR) image, or an RGB (red, green, blue) image. The depth estimation system can simultaneously acquire a brightness image and a ToF depth estimate. The brightness image and the ToF depth estimate can be based on the same light source (e.g., IR) or different light sources (e.g., visible light for a brightness image and IR for a ToF depth estimate). The brightness image can better detect boundaries and can be used to enhance the depth image generated by a ToF camera system.

[0024] An example of a depth estimation system includes an illuminator assembly, a camera assembly, and a controller. Light is projected onto an object. The illuminator assembly can project light to illuminate a local area, such as an area including an object. The light can be modulated light, such as modulated IR. The illuminator assembly can project pulsed light. Alternatively, the illuminator assembly can project one or more continuous waves, such as continuous waves of different frequencies. The object can reflect at least a portion of the projected light. The camera assembly captures at least a portion of the reflected light, and can convert the captured photons into electric charge and accumulate the charge. The depth estimation system can generate a depth image and a brightness image based on the charge accumulated in the camera assembly. In some embodiments, the depth image is based on a phase shift between the captured light and the projected light, and the brightness image is based on the brightness of the captured light. The depth image may include a plurality of depth pixels, each of which may correspond to a pixel in the brightness image.

[0025] The depth estimation system can generate an energy model representing the fusion of the depth image and the brightness image. The depth estimation system also performs an optimization process to minimize the fusion energy. The fusion energy can be a collection of spatial error energy and conditional error energy. Spatial error energy can be used to eliminate local errors in depth (e.g., local changes caused by common sensor noise), and conditional error energy can be used to enhance depth discontinuities and suppress non-local noise. The depth estimation system can perform an optimization process at the pixel level. For example, the depth estimation system can determine the fusion energy of a pixel and minimize the fusion energy to determine a new depth value for the pixel. The depth estimation system can run a separate optimization process for all pixels or a subset of pixels in the depth image. The depth estimation system can generate an enhanced depth image with a new depth value.

[0026] The depth estimation system can enhance the depth estimation of boundaries using the brightness image that shows clearer boundaries and can also smooth planes. The enhanced depth image generated by the depth estimation system can show boundaries better than a regular ToF depth image. With more accurate depth estimation, the enhanced depth image can be used in a variety of applications.

[0027] Depth Estimation System Example

[0028] Figure 1A depth estimation system 100 according to some embodiments of the present disclosure is shown. The depth estimation system 100 can use ToF technology to generate a depth image. The depth estimation system 100 includes an illuminator assembly 110, a camera assembly 120, and a controller 130. The illuminator assembly 110 includes an emitter 160 and a diffuser 165. The camera assembly 120 includes a lens 190 and an image sensor 195. In alternative configurations, different and / or additional components may be included in the depth estimation system 100. In addition, the functionality attributed to one component of the depth estimation system 100 may be implemented by a different component included in the depth estimation system 100 or a different system than shown. For example, the illuminator assembly 110 may not include a diffuser or multiple diffusers. As another example, the camera assembly 120 may not include a lens or include more than one lens.

[0029] The illuminator assembly 110 projects light 170 to a local area including the object 140. The emitter 160 is a light source that emits light ("emitted light"). In some embodiments, the emitter 160 may include a laser, such as an IR or near IR (NIR) laser, an edge-emitting laser, a vertical cavity surface emitting laser (VCSEL), etc. In other embodiments, the emitter 160 may include one or more light emitting diodes (LEDs). The emitter 160 may emit light in the visible light band (i.e., about 380nm to 750nm), the near infrared band (i.e., about 750nm to 1mm), the ultraviolet band (i.e., 10nm to 380nm), the short-wave infrared (SWIR) band (e.g., about 900nm to 2200nm), some other part of the electromagnetic spectrum, or some combination thereof. In some embodiments, the illuminator assembly 110 may include a plurality of emitters 160, each of which may emit a different wavelength. For example, the illuminator assembly 110 may include a first emitter that emits IR and a second emitter that emits visible light. Before light 170 is projected into the local area, diffuser 165 spreads or scatters the emitted light. Diffuser 165 can also control the brightness of the emitted light. In some embodiments, diffuser 165 can be translucent or semi-transparent. In other embodiments, illuminator assembly 110 can include more, fewer, or different components. For example, illuminator assembly 110 can include one or more additional diffusers to direct light from emitter 160 to one or more additional objects in the local area.

[0030] The illuminator assembly 110 may project the light 170 as modulated light, for example, according to a periodic modulation waveform. An example of a periodic modulation waveform may be a sinusoidal modulation waveform. The frequency of the periodic modulation waveform is the frequency of the modulated light.

[0031] The illuminator assembly 110 can project one or more continuous waves. For a single continuous wave, the illuminator assembly 110 can project multiple cycles. Different continuous waves may have different wavelengths and frequencies. For example, the illuminator assembly 110 can project a continuous wave with a modulation frequency in the range of 50 MHz to 200 MHz. In one embodiment, the illuminator assembly 110 includes multiple (i.e., at least two) light projectors. The light projectors can project continuous waves with different frequencies. The light projectors can alternate and project continuous waves at different times. For example, a first light projector projects a first continuous wave with a first frequency during a first time period. After the first time period, a second light projector projects a second continuous wave with a second frequency during a second time period. After the second time period, a third light projector projects a third continuous wave with a third frequency during a third time period. These three continuous waves can constitute a cycle. This cycle can be repeated.

[0032] In another embodiment, the illuminator assembly 110 may include a light projector that projects all three continuous waves. In other embodiments, the illuminator assembly 110 may project a different number of continuous waves, such as two or more than three. One cycle may constitute a frame. The total time of a cycle may be 10-20ms. The illuminator assembly 110 may project light through multiple cycles to obtain multiple frames. There may be time intervals between cycles. More information about modulated light with multiple frequencies is provided below in conjunction with Figure 2.

[0033] At least a portion of object 140 is illuminated by light 170. For simplicity and illustration purposes, Figure 1 The object 140 in the embodiment has a cubic shape. In other embodiments, the object 140 may have other shapes or structures. Figure 1 Not shown, the local area may also include other objects that may be illuminated by the light. The object reflects the light, reflecting light 180, and the reflected light 180 may be captured by the camera assembly 120.

[0034] The camera assembly 120 captures image data 170 of at least a portion of the local area illuminated with light 170. For example, the camera assembly 120 captures reflected light 180 and generates image data based on the reflected light 180. The reflected light 180 may be IR. In some embodiments, the camera assembly 120 may also capture visible light reflected by the object 140. The visible light may be projected by the illuminator assembly 110, ambient light, or a combination of both.

[0035] Even in Figure 1In some embodiments, the camera assembly 120 is separate from the illuminator assembly 110. In some embodiments, the camera assembly 120 is also located in the same location as the illuminator assembly 110 (e.g., can be part of the same device). The lens 190 receives the reflected light 180 and directs the reflected light 180 to the image sensor 195. The image sensor 195 includes a plurality of pixels 197. Even if Figure 1 Although the pixels 197 shown in FIG. 1 are arranged in one column, the pixels 197 of the image sensor 195 may also be arranged in multiple columns.

[0036] In some embodiments, pixel 197 includes a photodiode that is sensitive to light and converts collected photons into electric charge (e.g., photoelectrons). Each photodiode has one or more storage areas for storing electric charge. Image sensor 195 can be a ToF sensor and a brightness sensor. Pixel 197 can be a depth sensing pixel, a brightness sensing pixel, or both. The depth sensing pixel is configured to present a depth output signal that depends on the distance from the image sensor 195 to the track of the object 140 imaged on the depth sensing pixel. Such a distance is the "depth" of the track of the object 140. Each depth sensing pixel can independently determine the distance to the object 140 seen by the pixel. The depth output signal of the depth sensing pixel in the image sensor 195 can be used to generate a depth image of a local area. The brightness sensing pixel is configured to present a brightness output signal that depends on the brightness of the light reflected from the track of the object 140 imaged on the brightness sensing pixel. The brightness output signal of the brightness sensing pixel in the image sensor 195 can be used to generate a brightness image of a local area. The brightness image can be an active brightness image. An example of a brightness image is an IR image. In some embodiments, each pixel 197 of the image sensor 195 can generate a depth output signal and a brightness output signal from the reflected light captured by the pixel 197. In other embodiments, the image sensor 195 includes two groups of pixels 197: one group for sensing depth and the other group for detecting brightness. The output signal of the image sensor 195 can be an analog signal, such as an electric charge.

[0037] In embodiments where the illuminator assembly 110 projects multiple continuous waves or multiple cycles of modulated light, the image sensor 195 may be synchronized with the projection of the illuminator assembly 110. For example, the image sensor 195 may have one or more exposure intervals during which the image sensor 195 exposes a portion of a local area and accumulates charge in the image sensor 195. Outside of the exposure interval, the image sensor 195 is not exposed. In some embodiments, the exposure interval of the image sensor 195 may be synchronized with the continuous wave or cycle projected by the illuminator assembly 110. For example, the exposure interval starts before or when the continuous wave or cycle starts and ends after or when the continuous wave or cycle ends. In other embodiments, the image sensor 195 may have multiple exposure intervals for a single continuous wave. For example, the image sensor 195 may make multiple exposures during a continuous wave, and the multiple exposures may correspond to different phase offsets. In one example, a continuous wave has three exposure intervals with different phase offsets, such as 0° (0), 120° (2π / 3), and 240° (4π / 3). There may be time intervals between the exposure intervals. The time intervals may be 1-2 milliseconds (ms). The exposure interval may have a constant duration, for example approximately 100 microseconds (μs).In alternative embodiments, the exposure interval may have a different duration.

[0038] In some embodiments, the image sensor 195 can be scanned using a global shutter. The image sensor 195 includes a global shutter that can be opened and scanned during each exposure interval and closed at the end of the exposure interval. Additionally or alternatively, the image sensor 195 can include a tunable filter. The tunable filter blocks light from reaching the detector and can be mounted anywhere in the optical path of the reflected light 180. For example, the tunable filter is attached to the top of the image sensor 195 or the front of the camera assembly 120. The tunable filter can be switched between on (active) and off (inactive). The tunable filter can be inactive during the exposure interval and active at the end of the exposure interval. When the tunable filter is inactive, light can pass through the tunable filter and reach the image sensor 195. When the tunable filter is active, light is blocked from the image sensor 195.

[0039] In some embodiments, when the tunable filter is active, it may pass light of a particular wavelength (or a particular wavelength band) but block light of other wavelengths. For example, the tunable filter may pass light of a wavelength projected by the illuminator assembly 110 (e.g., light 170) but block light of other wavelengths, which may, for example, reduce noise in image data captured by the image sensor 195. In examples where the light 170 is IR, the tunable filter may block visible light. In other embodiments, when the tunable filter is active, it may block light of all wavelengths to avoid charge accumulation in the image sensor 195. In embodiments where the tunable filter blocks all light, dark noise calibration may be performed on the image sensor 195.

[0040] The camera assembly 120 can read out the stored photoelectrons from the image sensor 195 to obtain image data, for example, from the storage area of ​​each pixel 197 of the image sensor 195. During the readout, the camera assembly 120 can convert the photoelectrons into digital signals (i.e., analog-to-digital conversion). In embodiments where the illuminator assembly 110 includes multiple light projectors, the photoelectrons corresponding to the modulated light pulses projected by the different light projectors can be stored in separate storage areas of each photodiode. The camera assembly 120 can read out the separate storage areas to obtain image data. In some embodiments, the camera assembly 120 can read out all the image data stored in the image sensor 195. In other embodiments, the camera assembly 120 can read out some of the image data stored in the image sensor 195. For example, in embodiments where the exposure interval of the image sensor 195 is synchronized with the continuous wave projected by the illuminator assembly 110, the camera assembly 120 can perform multiple readout intervals for the continuous wave. Each readout interval can correspond to a different phase offset. In one example, a continuous wave has three readout intervals at three different phase offsets (e.g., 0°, 120°, and 240°). There may be a time interval between the readout intervals. The time interval may be 1-2 milliseconds (ms). The readout interval may have a constant duration, for example about 100 microseconds (μs). In alternative embodiments, the readout interval may have a different duration.

[0041] The controller 130 controls the illuminator assembly 110 and the camera assembly 120. For example, the controller 130 provides a lighting instruction to the lighting assembly 110, and the lighting assembly 110 projects light 170 according to the lighting instruction. The controller 130 may also provide an imaging instruction to the camera assembly 120, and the camera assembly 120 performs exposure and reads out image data according to the imaging instruction.

[0042] The controller 130 also uses the image data from the camera assembly 120 to determine depth information. For example, the controller 130 can generate a depth image based on the image data. The depth image includes a plurality of depth pixels. Each depth pixel has a value corresponding to an estimated depth, for example, an estimated distance from the trajectory of the object 140 to the image sensor 195. A single depth image may also be referred to as a depth frame or a depth map. In an embodiment where the illuminator assembly 110 projects a continuous wave of modulated light, the controller 130 may determine the depth information based on the phase shift between the light 170 projected by the illuminator assembly 110 and the reflected light 180. In an embodiment where the camera assembly 120 reads out image data corresponding to different phase offsets of the modulated light, the controller 130 may perform phase unwrapping to determine the depth information. In some embodiments (e.g., an embodiment where the illuminator assembly 110 projects multiple modulated light cycles), the controller 130 may generate multiple depth frames.

[0043] The controller 130 may also generate a brightness image corresponding to the depth image. The camera assembly 120 may generate image data of the brightness image and image information of the depth image from the same light (e.g., reflected light 180). In some embodiments, the brightness image and the depth image are generated simultaneously. For example, the camera assembly 120 may read out image data of the brightness image and image information of the depth image simultaneously. In one embodiment, the image data for the brightness image and the image for the depth image are the same image data. The brightness image may include a plurality of brightness pixels. Each brightness pixel has a value corresponding to light intensity (e.g., IR intensity). The brightness pixels in the brightness image may correspond to the depth pixels in the depth image. The brightness pixels and the depth pixels may be generated by light reflected from the same track of the object 140. For example, the brightness pixels and the depth pixels may be generated based on a signal from the same pixel 197 of the image sensor 195, and the pixel 197 captures light reflected from the track of the object 140.

[0044] The controller 130 may further enhance the depth estimation in the depth image by fusing the depth image with the brightness image based on the energy model. The brightness image may display one or more boundaries of the object 140 more clearly than the depth image. For example, one or more depth pixels representing at least a portion of the boundary of the object 140 may be invalid. The boundary may be an edge of the object 140. Alternatively, the boundary is a boundary between two areas of the object 140 having different reflectivity characteristics, such as a boundary between a fluorescent strip having relatively high reflectivity and a low reflectivity surface. The controller 130 may utilize the more accurate information of the boundary of the object 140 in the brightness image to generate an enhanced depth image that includes a better depth estimate than the original depth image. The following is combined with Figure 4A and 4B Certain aspects of controller 130 are described.

[0045] Modulation Signal Example

[0046] Figure 2A 2 shows a continuous wave of a projection signal 210 according to some embodiments of the present disclosure. The projection signal 210 is a modulated signal, for example, Figure 1 The projection signal 210 has a sinusoidal modulation waveform. Figure 2A In the embodiment, the sinusoidal modulation waveform can be represented by the following equation:

[0047] s(t)=A s sin(2πft)+B s

[0048] Where t represents time, s represents the optical power of the projection signal, I is the frequency of the modulation signal (i.e., modulation frequency), π is a mathematical constant, and A s represents the amplitude of the modulated signal, B s Represents the offset of the modulated signal, which may include the attenuated original offset and / or the offset due to the presence of ambient light (such as sunlight or light from artificial sources).

[0049] Figure 2B 2 shows a continuous wave of a captured signal 220 according to some embodiments of the present disclosure. The captured signal 220 is a captured portion of the modulated light reflected by an object illuminated by the projection signal 210. The captured signal 220 may be captured by the image sensor 195 and may be Figure 1 At least a portion of the reflected light 180 in Figure 2A In the embodiment of FIG. 2 , the captured signal 220 may be represented by the following equation:

[0050]

[0051]

[0052]

[0053] Where r represents the optimal power of the captured signal 220, α represents the attenuation factor of the captured signal 200, represents the phase shift between the waveform of the captured signal 220 and the waveform of the projected signal 210, δ is the time delay between the captured signal 210 and the projected signal 220, d represents the distance from the image sensor 195 to the object 140 (ie, the depth of the object 140), and c is the speed of light.

[0054] Figure 3 A cycle 310 of continuous waves 315A-C, 325A-C, and 335A-C of modulated light is shown according to some embodiments of the present disclosure. Figure 3In the embodiment, continuous waves 315A-C, 325A-C, and 335A-C are sinusoidal waves. In other embodiments, continuous waves 315A-C, 325A-C, and 335A-C may have different waveforms. Continuous waves 315A-C have a frequency 317, continuous waves 325A-C have a frequency 327, and continuous waves 335A-C have a frequency 337. The three frequencies 317, 327, and 337 are different from each other. Figure 3 In the embodiment of the present invention, frequency 317 is less than frequency 327, and frequency 327 is less than frequency 337. The three frequencies 317, 327, and 337 may be in the range of 50 to 200 MHz or higher.

[0055] In one embodiment, cycle 310 may be Figure 1 In another embodiment, period 310 may be an exposure period of camera assembly 120. For example, camera assembly 120 may expose during time periods of continuous waves 315A-C, 325A-C, and 335A-C, and may not expose outside of these time periods, although illuminator assembly 110 may project modulated light outside of these time periods. In another embodiment, period 310 may be a readout period of camera assembly 120. For example, camera assembly 120 may read out charge accumulated in image sensor 195 during time periods of continuous waves 315A-C, 325A-C, and 335A-C, and may not read out charge exceeding these time periods, although illuminator assembly 110 may project modulated light outside of these time periods, or image sensor 195 may expose outside of these time periods. Although Figure 3 The cycle 310 in includes three continuous waves for each of the three frequencies, but cycles in other embodiments may include a different number of frequencies or a different number of continuous waves for each frequency.

[0056] exist Figure 3, continuous waves 315A-C have different phase offsets. For example, continuous wave 315A has a phase offset of 0°, continuous wave 315B has a phase offset of 120°, and continuous wave 315C has a phase offset of 240°. Similarly, continuous waves 325A-C may have different phase offsets from each other: continuous wave 325A may have a phase offset of 0°, continuous wave 325B may have a phase offset of 120°, and continuous wave 325C may have a phase offset of 240°; and continuous waves 335A-C may start with different phase offsets from each other: continuous wave 335A may have a phase offset of 0°, continuous wave 335B may have a phase offset of 120°, and continuous wave 335C may have a phase offset of 240°. The phases of continuous waves 315A, 325A, and 335A may be between 0° and 120°, respectively, the phases of continuous waves 315B, 325B, and 335B may be between 120° and 240°, and the phases of continuous waves 315C, 325C, and 335C may be between 240° and 360°, respectively. In some embodiments, the duration of each continuous wave is about 100 μs. The time interval between two adjacent continuous waves may be in the range of 1 to 2 ms.

[0057] In other embodiments, cycle 310 may not have multiple continuous waves for each frequency. Instead, cycle 310 has a single continuous wave for a single frequency. For example, cycle 310 may include a first continuous wave for frequency 317, a second continuous wave for frequency 327, and a third continuous wave for frequency 337. The first, second, and third continuous waves may all start from 0°. There may be a time gap between two adjacent continuous waves of the first, second, and third continuous waves. Cycle 310 may generate image data for controller 130 to generate frames. Cycle 310 may be repeated so that controller 130 generates more frames. Controller 130 may perform phase unwrapping to determine depth information.

[0058] Controller Example

[0059] Figure 4A 4 is a block diagram illustrating a controller 130 according to some embodiments of the present disclosure. The controller 130 includes a database 410, an illuminator module 420, a camera module 430, a depth module 440, a brightness module 450, and a depth enhancement module 460. These modules are software modules implemented on one or more processors, dedicated hardware units, or some combination thereof. Some embodiments of the controller 130 have different components than those described in conjunction with FIG. 4. Similarly, the functionality of the components described in conjunction with FIG. 4 may be distributed among other components in a manner different from that described in conjunction with FIG. 4. For example, some or all of the functions described as being performed by the controller 130 may be performed by a device incorporating a depth estimation system, such as Figure 6 System 600, Figure 7 The mobile device 700, Figure 8 Entertainment system 800, Fig. 9 The robot 900 or other device in the embodiment of the present invention.

[0060] The database 410 stores data generated and / or used by the controller 130. The database 410 is a memory, such as a ROM, DRAM, SRAM, or some combination thereof. The database 410 may be part of a larger digital memory of a depth estimation system, such as the depth estimation system 100, or may be part of a device that includes the depth estimation system. In some embodiments, the database 410 stores image data from the camera assembly 120, a depth image generated by the depth module 440, a brightness image generated by the brightness module 450, an enhanced depth image generated by the depth enhancement module 460, parameters for an energy model generated by the depth enhancement module 460, data for optimizing the energy model, and the like. In some embodiments, the database 410 may store calibration data and / or other data from other components, such as depth instructions. The depth instructions include illuminator instructions generated by the illuminator module 420 and camera instructions generated by the camera module 430.

[0061] The illuminator module 420 controls the illuminator assembly 110 via illuminator instructions. The illuminator instructions include one or more illumination parameters that control how the illuminator assembly 110 projects light. The illumination parameters may describe, for example, a waveform, a wavelength, an amplitude, a frequency, a phase shift, a start time of each continuous wave, an end time of each continuous wave, a duration of each continuous wave, some other parameter that controls how the illuminator assembly 110 projects light, or some combination thereof. The illuminator module 420 may retrieve the illuminator instructions from the database 350. Alternatively, the illuminator module 420 generates the illuminator instructions. For example, the illuminator module 420 determines one or more illumination parameters. In an embodiment where the illuminator assembly 110 includes multiple modulated light projectors, the illuminator module 420 may determine separate illumination parameters for different light projectors.

[0062] The camera module 430 controls the camera assembly 120 via camera instructions. The camera module 430 can retrieve the camera instructions from the database 410. Alternatively, the camera module 430 generates the camera instructions based in part on the illuminator instructions generated by the illuminator module 420. The camera module 430 determines the exposure parameters (such as the start time, the end time, or the duration of the exposure interval, etc.) of the camera assembly 120 based on one or more lighting parameters (such as the duration of the continuous wave, etc.) specified in the illuminator instructions. For example, the camera module 430 determines that the duration of the exposure is equal to the duration of the continuous wave. Sometimes the camera module 430 determines that the duration of the exposure is longer than the duration of the continuous wave to avoid the inability to collect the entire continuous wave due to the delay of the incident light. The duration of the exposure can be 20% longer than the duration of the continuous wave. In some embodiments, the camera module 430 also determines the number of exposure intervals for each continuously modulated light wave projected by the lighting assembly 110.

[0063] The camera instructions may include readout instructions for controlling the readout of the camera assembly 120. The camera module 430 may determine the readout parameters of 120 (e.g., the start time, the end time, or the duration of the readout interval, etc.). For example, the camera module 430 determines the start time of each of the one or more readout intervals, for example, based on one or more illumination parameters (e.g., phase, waveform, start time, or other parameters of the continuous wave). The camera module 430 may also determine the duration of each readout interval, the number of readout intervals of the continuous wave, the time interval between adjacent readout intervals, the number of readout cycles, other readout parameters, or some combination thereof.

[0064] The depth module 440 is configured to generate a depth image indicating the distance to the imaged object 140, for example, based on a digital signal indicating the charge accumulated on the image sensor 195. The depth module 440 may analyze the digital signal to determine the phase shift exhibited by the light (e.g., the phase shift described above in connection with FIG. 2). ) to determine the ToF of the light (eg, the ToFδ described above in conjunction with FIG. 2 ), and further determine the depth value of the object 140 (eg, the distance described above in conjunction with FIG. 2 ).

[0065] In embodiments where illumination assembly 110 projects multiple continuous waves with different phase offsets, depth module 440 may generate a depth image by phase unwrapping. Figure 3Taking the period 310 in as an example, the depth module 440 can determine the wrapped distance, each wrapped distance corresponding to the corresponding phase. The depth module 440 can also estimate the unwrapped depth of each wrapped distance. The depth module 440 also determines the Voronoi vector corresponding to the unfolded depth and generates a lattice of Voronoi cells. Each unfolded depth corresponds to a Voronoi cell of the lattice. In an alternative embodiment, the depth module 440 is configured to determine the depth information using the charge ratio between the storage areas associated with each photodiode of the camera assembly 120.

[0066] The brightness module 450 generates a brightness image, such as an active brightness image. In some embodiments, for the depth image generated by the depth module 440, the brightness module 450 generates a corresponding brightness image. The brightness module 450 can generate a brightness image based on a request for a brightness image from the depth enhancement module 460. The brightness module 450 can generate a depth image based on the phase shift between the first captured light and the projected light, and generate a corresponding brightness image based on the intensity or amplitude of the second captured light. In some embodiments, the first captured light and the second captured light are the same light. In other embodiments, the second captured light is different from the first captured light. For example, the first captured light may be IR, and the second captured light may be visible light.

[0067] In an embodiment where the depth module 440 generates a depth image based on the charge accumulated in a group of pixels 197 of the image sensor 195, the brightness module 450 can generate a corresponding brightness image based on the charge accumulated in all or some of the pixels 197 in the group. The corresponding brightness image includes a plurality of brightness pixels. Each brightness pixel can correspond to a depth pixel in the depth image. For example, the values ​​of the depth pixel and the corresponding brightness pixel can be determined based on the charge accumulated in the same pixel 197 in the image sensor 195. The charge accumulated in the pixel can be converted from photons of modulated light reflected by the trajectory of the object 140. The value of the depth pixel can be determined by the depth module 440 based on the phase shift in the waveform of the modulated light. The value of the corresponding brightness pixel can be determined based on the accumulated charge in the pixel 197 or a different pixel 197.

[0068] The depth enhancement module 460 enhances the depth estimation performed by the depth module 440 based on the brightness image generated by the brightness module 450. The depth enhancement module 460 may retrieve the depth image generated by the depth module 440 and the corresponding brightness image generated by the brightness module 450. In some embodiments, the depth enhancement module 460 may instruct the brightness module 450 to generate the corresponding brightness image. The depth enhancement module 460 may enhance the depth estimation of the depth module 440 by fusing the depth image and the corresponding brightness image using an energy model. The depth enhancement module 460 may generate an energy function based on the depth image and the corresponding brightness image, and then generate an enhanced depth image by optimizing the image energy function. The depth enhancement process of the depth enhancement module 460 converts the depth image into an enhanced depth image. The brightness image may be used as a guide in the depth enhancement process.

[0069] Figure 4B is a block diagram illustrating a depth enhancement module 460 according to some embodiments of the present disclosure. Figure 4B The depth enhancement module 460 in the embodiment includes a disparity generator 470, a boundary weight module 475, a fusion energy module 480, and an optimization module 485. In other embodiments, the depth enhancement module 460 may include fewer, more, or different components. In addition, the functions of the components of the depth enhancement module 460 described in conjunction with FIG. 4 may be distributed between other components in a manner different from that described in conjunction with FIG. 4.

[0070] The disparity generator 470 optionally generates a disparity image (also referred to as a "disparity map") from the depth image. In some embodiments, the disparity generator 470 converts depth pixels in the depth image into disparity pixels of the disparity image. For example, a depth value D (e.g., the value of a depth pixel in the depth image) is converted into a disparity value Dt (or disparity). The relationship between D and Dt can be expressed as:

[0071] Dt=b / D

[0072] Wherein b is a constant used to scale the disparity to a given value range. The disparity generator 470 may determine disparity values ​​of all or some depth pixels in the depth image and generate a disparity image based on the disparity values.

[0073] Boundary weight module 475 determines one or more boundary weights based on the disparity or depth image and the brightness image, for example, based on a combination of boundaries shown in the disparity map and boundaries shown in the brightness image. Although some descriptions of boundary weight module 475 relate to using a disparity image to determine boundary weights, in some embodiments, boundary weight module 475 may instead determine boundary weights based on a depth image.

[0074] In some embodiments, the boundary weight module 475 determines the boundary weight by applying a gradient to the disparity or depth image and the brightness image. The gradient of the image (e.g., the depth image, the disparity image, or the brightness image) identifies the boundary in the image. The boundary weight can avoid boundary blending and can be used to retain or enhance the boundary in the image. The boundary weight module 475 can determine the fusion gradient of the pixel of the enhanced depth image. The pixel can correspond to a depth pixel in the depth image, a disparity pixel in the disparity image, and / or a brightness pixel in the brightness image. The pixel can have a depth value (i.e., the depth value of the corresponding depth pixel in the depth image), a disparity value (i.e., the disparity value of the corresponding disparity pixel in the disparity image), a brightness value (i.e., the brightness value of the corresponding brightness pixel in the brightness image) and an enhanced depth value (i.e., the enhanced depth value of the pixel in the enhanced depth image). The pixel can be represented by an index i or a pair of (x, y) coordinates, which indicates the position of the pixel in the enhanced depth image, depth image, brightness image, or disparity image.

[0075] To determine the magnitude of the fused gradient (“fused gradient magnitude” E l ), the boundary weight module 475 can determine the magnitude of the depth gradient (“depth gradient magnitude” E ) of the pixel (x, y) in the disparity or depth image. D The depth gradient may indicate a change in depth value in the depth image (eg, a change in direction). The depth gradient may be a value having a magnitude (ie, E D ) and a two-dimensional vector in the direction of increasing depth values. In one embodiment, the boundary weight module 475 applies a gradient operator to the disparity or depth image. The gradient operator can return the depth gradient magnitude E of the pixel (x, y) D The boundary weight module 475 can also determine the magnitude of the brightness gradient of the pixel (x, y) in the brightness image (“brightness gradient magnitude” E A ), for example, by applying the same or different gradient operator to the luminance image. The boundary weight module 475 also combines the depth gradient magnitude E D and the brightness gradient amplitude E A To generate the fused gradient (E I In one embodiment, the boundary weight module 475 determines the fusion gradient E I is the depth gradient amplitude E D and the brightness gradient amplitude E A The product of:

[0076] E I =E D *E A

[0077] In some embodiments, the boundary weight module 475 may use one or more gradient operators, examples of which include a Sobel gradient operator, a Prewitt gradient operator, a center difference gradient operator, a median difference gradient operator, a Roberts gradient operator, or other suitable types of gradient operators. In some embodiments, the boundary weight module 475 determines the weighted difference between the pixel (x, y) and the value (depth value or brightness value) of the neighboring pixel as the gradient magnitude (E D or E A ). A neighboring pixel is a pixel that is adjacent to the pixel. The coordinates of the neighboring pixels may be (x+1, y), (x, y+1), (x+1, y+1), etc. For example, the boundary weight module 475 may determine the corresponding difference between the value of the pixel and each neighboring pixel and aggregate the differences of all neighboring pixels based on a weight. The weight of the neighboring pixel may be determined based on a fused depth gradient from the pixel to the neighboring pixel. The distance may be determined based on the coordinates of the pixel and the neighboring pixel. The depth gradient magnitude E D It can be a weighted sum or weighted average of the depth value differences. Similarly, the brightness gradient amplitude E A It can be a weighted sum or weighted average of the brightness value differences. In other embodiments, the boundary weight module 475 converts the gradient magnitude (E D or E A ) is determined as the difference between the value of a pixel and a single neighboring pixel: the depth gradient magnitude E D is the difference between the depth value of a pixel and the depth value of an adjacent pixel, and the brightness gradient amplitude E A It is the difference between the brightness value of the pixel and the brightness value of the adjacent pixel.

[0078] In another embodiment, the gradient operator can be expressed as:

[0079]

[0080] in is the derivative with respect to the x direction and indicates the gradient in the x direction. is the derivative with respect to the y direction and indicates the gradient in the y direction. The magnitude of the gradient can be defined as |g x |+|g y |, or based on g x and g y Determine other values.

[0081] In addition, the boundary weight module 475 is based on the fusion gradient magnitude E I Determine the boundary weight W of pixel (x, y) E :

[0082]

[0083] Where α is a scaling constant. The boundary weight module 475 may determine corresponding boundary weights for all or some pixels.

[0084] The fusion energy module 480 determines the fusion energy based on one or more boundary weights determined by the boundary weight module 475. In some embodiments, the fusion energy module 480 determines the fusion energy for the entire enhanced depth image. In other embodiments, the fusion energy module 480 determines the fusion energy for a pixel of the enhanced depth image, such as pixel (x, y). The fusion energy Q can be the spatial error energy Q S and conditional entropy error energy Q H The spatial error energy Q S It can be used to reduce spatial image noise, such as local noise. Local noise can be Gaussian noise or other sensor noise. Where applicable, the spatial error energy Q S The similarity between the depths of adjacent pixels can be enhanced. The similarity assumption allows smoothing of the depth map within continuous object surface regions with gradually changing depths. This smoothing effect may be disabled in depth transitions at object boundaries due to the neighboring pixel boundary weights. In some embodiments, the spatial error energy Q can be minimized by reducing the difference between the depth values ​​of adjacent pixels. S . Conditional entropy error energy Q H The depth discontinuity can be enhanced and other non-local noise can be suppressed by maximizing the correlation between the estimated depth image and the brightness image. H A measure of uncertainty in a pixel depth value for a given pixel brightness value may be indicated.

[0085] In some embodiments, the fusion energy module 480 is based on the level error ∈ H and vertical error ∈ V To determine the spatial error energy term Q S ,For example:

[0086]

[0087] ∈ H and ∈ V Each depth at the location It can be defined as:

[0088]

[0089]

[0090] in represents all possible solutions of the resulting depth image (ie, the enhanced depth image).

[0091] The fusion energy module 480 can determine the conditional entropy error energy Q based on the mutual information or conditional entropyH . Conditional error energy Q H It can be defined as:

[0092] Q H (x, y) = W E (x,y)*p(D(x,y),A(x,y)logp(D(x,y)|A(x,y))

[0093] Where p(D(x, y), A(x, y)) represents the joint probability value of the depth value D(x, y) and the brightness value A(x, y) of the pixel (x, y), and p(D(x, y)|A(x, y)) represents the conditional probability.

[0094] In some embodiments, the fusion energy module 480 may determine the mutual information energy Q based on the mutual information (MI) between the disparity image (or depth image) and the luminance image. M MI can measure the amount of information one variable (e.g., one of two images) contains about another variable (e.g., the other image). MI can be used to reduce the uncertainty of one variable given the information in the other variable. As a similarity or correlation measure, MI can be used in multimodal imaging. It is robust to outliers, computationally efficient, and provides a smooth cost function for optimization. Assuming D and A are discrete random variables, MI can be defined as:

[0095] MI=H(A)+H(D)-H(A,D)=H(D)-H(D / A)

[0096] Where H(A) and H(D) are the entropies of the brightness image and disparity image (or depth image), respectively, H(A, D) is the joint entropy, and H(D / A) is the conditional entropy of depth D given brightness A. Maximizing MI may be equivalent to minimizing the conditional entropy H(D / A), so Q M It can also be defined as Q H =H(D / A).

[0097] Given a depth image and a brightness image, the conditional entropy H(D / A) can be calculated, where the conditional entropy is a measure of how much uncertainty remains in the depth D given the brightness A, that is,

[0098]

[0099] where p AD (a, d) is the joint probability of brightness and depth values ​​and p AD(d / a) is the conditional probability of a depth value given a brightness value. D can be discretized and can have a finite set of values. Another option is to use disparity instead of depth, as disparity can be constrained and managed in a simpler way. In some embodiments, disparity and brightness values ​​can be discretized into a given number of intervals, such as N Dt and N IR After discretizing the disparity and brightness values, the joint probability distribution p(a, d) can be estimated by normalizing their joint histogram. The disparity can be restricted to [0, Dt max ], which is the maximum parallax Dt max The range corresponds to the minimum depth.

[0100] The fusion energy module 480 also aggregates the spatial error energy Q S and conditional error energy Q H To determine the fusion energy Q of the pixel. In some embodiments, the fusion energy Q can be expressed as:

[0101] Q(x, y) = cQ S (x,y)+(1-c)Q H (x, y)

[0102] Where c is a regularization parameter used to control the influence of each energy. MI or conditional entropy may not consider the spatial relationship between pixels. I The associated boundary weight W E Can be used to preserve borders.

[0103] The optimization module 485 performs an optimization process, wherein the optimization module 485 optimizes the fusion energy to determine the enhanced depth value. The optimization module 485 optimizes the fusion energy by minimizing the image fusion energy. The optimization process can be performed at the pixel level, that is, the optimization module 485 minimizes the fusion energy of the pixel to determine the enhanced depth value of the pixel. The optimization processes for different pixels can be independent of each other. The optimization process for pixel i=(x, y) can be expressed as:

[0104]

[0105] Where D * (x, y) is the enhanced depth value of the pixel. The optimization process can be an iterative process. The depth estimation system can run a separate optimization process on all pixels or a subset of pixels in the depth image.

[0106] In some embodiments, the optimization module 485 uses a gradient descent method to minimize the fusion energy of the pixels. The gradient descent method includes an adaptive learning rate λ, which allows for a fast approximation during the first iteration and gradually slows down during the optimization process.

[0107]

[0108] The result module 490 generates an enhanced depth image based on the enhanced depth value determined by the optimization module 485. For example, the result module 490 can replace the depth value of the pixel with the enhanced depth value. The enhanced depth image represents a better depth estimate, particularly for one or more boundaries of the object 140.

[0109] Depth Estimation Enhancement Example

[0110] Figure 5A An example depth image 510 is shown according to some embodiments of the present disclosure. The depth image 510 may be generated by the depth module 440 described above in conjunction with FIG. Figure 5B The present invention shows some embodiments of the present invention. Figure 5A 4. An example brightness image 520 corresponding to the depth image 510 in FIG. 4. The brightness image 520 may be generated by the brightness module 450 described above in conjunction with FIG. Figure 5C According to some embodiments of the present disclosure, Figure 5A The depth image 510 and Figure 5B Example depth enhanced image 530 generated from brightness image 520 in FIG.

[0111] Three images 510, 520 and 530 capture an object, an example of which is Figure 1 The object 140 has a flat surface 540 and an edge 550. Figures 5A-5C In the example, part of the edge is enclosed in a dotted ellipse. Figure 5A and 5B As shown, the brightness image 520 shows clearer edges than the depth image 510. Therefore, the brightness image 520 can be used to enhance the depth estimation of the object, especially the edge 550 of the object. The brightness image 520 is fused with the depth image 510 through the energy model to generate the depth enhanced image 530. The depth enhanced image 530 shows a better depth estimation than the depth image 510. Figure 5A and 5C As shown, the edges in the depth enhanced image 530 are clearer than those in the depth image 510. The depth enhanced image 530 may be generated by using an energy model, where the depth image 510 may be used as an input image and the brightness image 520 may be used as a guide image. The depth enhanced image 530 may be generated by the depth enhancement process described above in conjunction with the depth enhancement module 460. For example, pixels representing the edge 550 are identified, and new depth values ​​for the pixels are determined based on the pixel values ​​in the brightness image 520. The new depth values ​​for the pixels are used to generate the depth enhanced image 530.

[0112] Figure 5A and 5CIt is also shown that plane 540 is smoother in depth enhanced image 530. Figures 5A-5C In some embodiments, pixels representing plane 540 are also identified. For each of these pixels, a box is defined. The box can be centered on the pixel and include other pixels surrounding the pixel. A new depth value for the pixel is determined based on the depth values ​​of the other pixels in the box. For example, the new depth value for the pixel can be the average of the depth values ​​of the other pixels. The new depth value for the pixel is used to generate a depth enhanced image 530. Through such a depth enhancement process, the depth enhanced image 530 represents a more accurate depth estimate of the object, such as a more accurate depth estimate of the plane 540, the edge 550, or both.

[0113] Example Applications Incorporating Depth Estimation Systems

[0114] Figure 6 An example system 600 including a depth estimation system according to some embodiments of the present disclosure is shown. One embodiment of the depth estimation system is Figure 1 1. The depth estimation system 100 in FIG. 1 includes an imaging device 610, a processor 620, a memory 630, an input device 640, an output device 650, and a battery / power circuit 660. In other embodiments, the system 600 may include fewer, more, or different components. For example, the system 600 may include multiple processors, memories, display devices, input devices, or output devices.

[0115] Imaging device 610 captures depth images and brightness images. Imaging device 610 may include an illuminator assembly, such as illuminator assembly 110, for projecting light into the environment surrounding system 600. Imaging device 610 may project modulated light, such as pulsed modulated light or a continuous wave of modulated light. Imaging device 610 also includes a camera assembly, such as camera assembly 120, which captures light reflected by one or more objects in the environment and generates image data of the one or more objects.

[0116] The processor 620 may process electronic data from registers and / or memories to convert the electronic data into other electronic data that may be stored in registers and / or memories. The processor 620 may perform some or all functions of some or all components of the controller 130, such as depth estimation, depth estimation enhancement using brightness signa, etc. The processor 620 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), CPUs, GPUs, cryptographic processors (special processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices.

[0117] In some embodiments, the processor 620 may also use the depth information (e.g., an enhanced depth image) to generate content (e.g., images, audio, etc.) for display to a user of the system through one or more display devices (e.g., output device 650). The content may be used as VR, AR, or MR content. The processor 620 may also generate instructions for other components of the system 600 or another system based on the enhanced depth image. For example, the processor 620 may determine navigation instructions for a movable device, such as a robot, vehicle, or other type of movable device. The navigation instructions may include navigation parameters (e.g., navigation route, speed, direction, etc.).

[0118] The memory 630 may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile storage (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 630 may include a memory that shares a die with the processor 620. The memory 630 may store processor-executable instructions for controlling the operation of the depth estimation system 100 and / or data captured by the depth estimation system 100. In some embodiments, the memory 630 includes one or more non-transitory computer-readable media storing instructions executable to perform a depth estimation enhancement process, such as the following in conjunction with Fig.10 The method 1000 described above, or a combination thereof Figure 1 4 and the operations performed by the controller 130 (or some components of the controller 130 ). The instructions stored in one or more non-transitory computer-readable media may be executed by the processor 620 .

[0119] Input device 640 may include an audio input device. Audio input device 1318 may include any device that generates a signal representing sound, such as a microphone, a microphone array, a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output), etc. Input device 640 may also include one or more other types of input devices, such as an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (such as a mouse), a stylus, a touchpad, a barcode reader, a quick response (QR) reader, a sensor, a radio frequency identification (RFID) reader, etc.

[0120] Output device 650 may include one or more display devices, such as one or more visual indicators. Example visual indicators include a head-up display, a computer display, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display, etc. Output device 650 may also include an audio output device. An audio output device may include any device that produces a sound indicator, such as a speaker, a headset, or an earplug, etc. Output device 650 may also include one or more other output devices, such as an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, etc.

[0121] Battery / power circuit 660 may include one or more energy storage devices (eg, batteries or capacitors) and / or circuits for coupling components of system 600 to an energy source separate from system 600 (eg, AC line power).

[0122] Figure 7 A mobile device 700 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. An example of a depth estimation system is Figure 1 The depth estimation system 100 in FIG. 7 is a mobile device 700 that can be a mobile phone. Figure 7 As shown, mobile device 700 includes imaging component 702. The imaging device may include Figure 1 The illuminator assembly 110 and the camera assembly 120 in the embodiment of the present invention. The imaging assembly 702 can illuminate the environment around the robot 902 with modulated light (e.g., modulated IR) and capture images of one or more objects in the environment. Figure 7 Not shown, the mobile device 700 may also include one or more processors and one or more memories that may perform some or all of the functions of the controller 130. Through these components, the mobile device 700 may determine depth information of one or more objects in the environment surrounding the mobile device 700. The mobile device 700, another device, or a user of the mobile device may use the depth information for various purposes, such as VR, AR, or MR applications, navigation applications, and the like. For example, the mobile device 700 may generate and present images (two-dimensional or three-dimensional images) based on the depth information of the environment, and the images may represent virtual objects that do not exist in the real-world environment. These images may enhance the real-world objects in the environment so that the user of the mobile device 700 may have an interactive experience with the real-world environment in which real-world objects residing in the real world are enhanced by computer-generated virtual objects.

[0123] Figure 8 An entertainment system 800 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. Figure 8In the example of , user 808 can interact with the entertainment system via controller 810, such as playing a video game. Entertainment system 800 includes console 802 and display 804. Console 802 can be a video game console that is configured to generate images of a video game on display 804. In other embodiments, entertainment system 800 can include more, fewer, or different components.

[0124] The console 802 includes an imaging assembly 806. The imaging assembly 806 may include Figure 1 The imaging component 806 can illuminate the environment around the entertainment system 800 with modulated light (e.g., modulated IR) and capture the modulated light reflected by one or more objects in the environment to generate an image of the object, such as a user 808, a controller 810, or other object. Figure 8 800 . Although not shown, the console 802 may also include one or more processors and one or more memories that may perform some or all of the functions of the controller 130 . The console 802 may determine depth information of one or more objects in the environment. The depth information may be used to present images to the user on the display 804 , or to control certain other aspects of the entertainment system 800 . For example, the user 808 may control the entertainment system 800 with gestures, and the gestures may be determined at least in part by the depth information. The console 802 may generate or update display content (e.g., images, audio, etc.) based on the depth information, and may also instruct the display 804 to present the display content to the user 808 .

[0125] Fig. 9 An example robot 902 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. The robot 902 includes an imaging component 904, which may include Figure 1 The illuminator assembly 110 and the camera assembly 120 in the embodiment of the present invention. The imaging assembly 904 can illuminate the environment around the robot 902 with modulated light (e.g., modulated IR) and capture images of one or more objects in the environment. Fig. 9 Not shown, the robot 902 may also include a computing device that can perform some or all of the functions of the controller 130. The computing device may include one or more processors and one or more memories. The computing device may determine depth information of one or more objects in the environment. The robot 902 may be mobile, and the computing device may use the depth information to assist in navigation and / or motor control of the robot 902. For example, the computing device may include navigation instructions based on the depth information. The navigation instructions may include a navigation route for the robot 902. The robot 902 may navigate in the environment according to the navigation instructions.

[0126] Apart from Figure 7-9Examples of uses of the technology described herein are possible beyond those shown. For example, the depth estimation system described herein can be used in other applications such as self-driving cars, security cameras, etc.

[0127] Example method to enhance depth estimation using energy model

[0128] Fig.10 1 is a flow chart illustrating a method 1000 for enhancing depth estimation of a luminance image using an energy model according to some embodiments of the present disclosure. The method 1000 may be performed by the controller 130. Although reference is made to Fig.10 The flowchart shown in describes method 1000, but many other methods of using energy models to enhance depth estimation of luminance images may also be used. For example, one may change Fig.10 As another example, some steps may be changed, eliminated, or combined.

[0129] In 1010, the controller 130 converts the depth image into a disparity image. The depth image includes a plurality of depth pixels. The disparity image includes a plurality of disparity pixels. The disparity value of each of the plurality of disparity images is determined based on the depth values ​​of different depth pixels in the plurality of depth pixels. In some embodiments, the disparity value is proportional to the inverse of the depth value.

[0130] In 1020, the controller 130 determines the boundary weight of the target height pixel among the multiple depth pixels based on the gradient magnitude of the target depth pixel in the depth image and the gradient magnitude of the brightness pixel in the brightness image. The gradient magnitude of the target depth pixel in the depth image may be the difference (e.g., weighted difference) between the target depth pixel and the depth pixel adjacent to the target depth pixel in the depth image ("adjacent depth pixel"). The difference may be the difference between the depth value of the target depth pixel and the thickness value of the adjacent depth pixel. Similarly, the gradient magnitude of the brightness pixel in the brightness image may be the difference (e.g., weighted difference) between the brightness pixel and another brightness pixel adjacent to the brightness pixel in the brightness image ("adjacent brightness pixel") . The difference may be the difference between the brightness value of the brightness pixel and the intensity value of the adjacent brightness pixel.

[0131] In some embodiments, the luminance image and the depth image capture the same object. The luminance image includes a plurality of luminance pixels, and the plurality of luminance pixels include luminance pixels. Each corresponding luminance pixel in the plurality of luminance pixels corresponds to a corresponding depth pixel in the plurality of depth pixels. For example, the target depth pixel represents the same object location as the luminance pixel. In some embodiments, the depth image and the luminance image are generated based on image data from the same image sensor.

[0132] In some embodiments, the controller 130 may instruct the illuminator assembly to project modulated light into a local area including an object. The controller 130 may also instruct the camera assembly to capture reflected light from at least a portion of the object. The controller 130 may generate a depth image based on a phase shift between the reflected light and the modulated light projected into the local area. The controller 130 may generate a brightness image based on the brightness of the reflected light. The reflected light may be a first reflected light, and the controller 130 may instruct the camera assembly to capture a second reflected light from at least a portion of the object, and generate a brightness image based on the brightness of the second reflected light. The second reflected light has a different wavelength from the first reflected light.

[0133] In 1030, the controller 130 determines the energy of the target depth pixel based on the boundary weight. In 1040, the controller 130 determines a new depth value of the target depth pixel by optimizing the energy. In some embodiments, the controller 130 determines the spatial error energy of the target thickness pixel based on the boundary weight. The controller 130 can optimize the energy, including optimizing the spatial error energy by reducing the difference between the depth value of the target depth pixel and the thickness value of another depth pixel adjacent to the target depth pixel in the depth image. The controller 130 can also determine the conditional error energy of the target depth pixel based on the depth value of the target depth pixel in the depth image and the brightness value of the brightness pixel in the brightness image. The conditional error energy indicates a measure of the uncertainty of the depth value of the target depth pixel given the brightness value of the brightness pixel.

[0134] In 1040, the controller 130 updates the depth image by assigning the depth value to the target depth pixel. The controller 130 may generate an enhanced depth image based on the depth value. The enhanced depth image represents a better depth estimate than the depth image because the depth value of the target depth pixel in the enhanced depth image has better accuracy than the target depth pixel in the depth image.

[0135] Changes and implementation

[0136] Although the above reference Figure 1-10 The illustrated exemplary implementations describe embodiments of the present disclosure, but those skilled in the art will appreciate that the various teachings described above are applicable to a variety of other implementations.

[0137] In some cases, the features discussed herein may be applicable to automotive systems, safety-critical industrial applications, medical systems, scientific instrumentation, wireless and wired communications, radio, radar, industrial process control, audio and video equipment, current sensing, instrumentation (which may be high precision), and other digital processing-based systems.

[0138] In the discussion of the above embodiments, components of the system, such as filters, converters, mixers, amplifiers, digital logic circuits and / or other components, can be easily replaced, substituted or otherwise modified to suit specific circuit requirements. In addition, it should be noted that the use of complementary electronic devices, hardware, software, etc. in various communication systems provides equally viable options for implementing the teachings of the present disclosure related to fractional frequency dividers.

[0139] As described herein, portions of various systems for implementing a balanced fractional divider for duty cycle regulation may include electronic circuits that perform the functions described herein. In some cases, one or more portions of the system may be provided by a processor specifically configured to perform the functions described herein. For example, the processor may include one or more dedicated components, or may include programmable logic gates configured to perform the functions described herein. The circuit may operate in an analog domain, a digital domain, or a mixed signal domain. In some cases, the processor may be configured to perform the functions described herein by executing one or more instructions stored on a non-transient computer readable storage medium.

[0140] In an example embodiment, any number of circuits of this figure can be implemented on a board of an associated electronic device. The board can be a general circuit board that can accommodate various components of the electronic system inside the electronic device and can also provide connectors for other peripheral devices. More specifically, the board can provide an electrical connection through which other components of the system can communicate electrically. Any suitable processor (including DSP, microprocessor, supporting chipset, etc.), computer-readable non-transient storage element, etc. can be appropriately coupled to the board according to specific configuration requirements, processing requirements, computer design, etc. Other components, such as external memory, additional sensors, audio / video display controllers, and peripherals, can be connected to the board as plug-in cards via cables or integrated into the board itself. In various embodiments, the functions described herein can be implemented in simulated form as software or firmware running in one or more configurable (e.g., programmable) elements arranged in a structure that supports these functions. The software or firmware providing the simulation can be provided on a non-transient computer-readable storage medium that includes instructions that allow the processor to perform these functions.

[0141] In another example embodiment, the circuit of the present figure can be implemented as a stand-alone module (e.g., a device having related components and circuits configured to perform a specific application or function) or as a plug-in module inserted into dedicated hardware of an electronic device. Note that certain embodiments of the present disclosure can be easily included in part or in whole in a system-on-chip (SOC) package. SOC represents an integrated circuit that integrates the components of a computer or other electronic system into a single chip. It may contain digital, analog, mixed-signal, and radio frequency (RF) functions: all of which can be provided on a single chip substrate. Other embodiments may include a multi-chip module (MCM), in which multiple separate ICs are located within a single electronic package and are configured to interact closely with each other through the electronic package.

[0142] It is also important to note that all specifications, dimensions, and relationships outlined herein (e.g. Figure 1 -2, 4-5, 7 and 9-10) are for example and teaching purposes only. Such information may be significantly changed without departing from the spirit of the present disclosure or the scope of the appended claims. It should be understood that the system can be integrated in any suitable manner. Along similar design alternatives, any of the circuits, components, modules and elements shown in this figure can be combined in various possible configurations, all of which are clearly within the broad scope of this specification. In the foregoing description, example embodiments have been described with reference to specific processors and / or component arrangements. Various modifications and changes may be made to these embodiments without departing from the scope of the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0143] Note that through many of the examples provided herein, interactions may be described in terms of two, three, four or more electrical components. However, this is done for clarity and example only. It should be understood that the system may be integrated in any suitable manner. Along similar design alternatives, any components, modules, and elements shown in the figures may be combined in a variety of possible configurations, all of which are clearly within the broad scope of this specification. In some cases, it may be easier to describe one or more functions of a given set of flows by referencing only a limited number of electrical components. It should be understood that the drawings and the circuits they teach are easily expandable and can accommodate a large number of components and more complex / complex arrangements and configurations. Therefore, the examples provided should not limit the scope of the circuit or inhibit its broad teachings that may be applied to countless other architectures.

[0144] Note that in this specification, references to various features (e.g., elements, structures, modules, components, steps, operations, characteristics, etc.) included in "one embodiment," "example embodiment," "specific embodiment," "another embodiment," "some embodiments," "various embodiments," "other embodiments," or "alternative embodiments," etc., are intended to indicate that any such features are included in one or more embodiments of the present disclosure, but may or may not be combined in the same embodiment. In addition, as used herein, including in the claims, "or" used in a list of items (e.g., a list of items beginning with phrases such as "at least one" or "one or more") represents an inclusive list, for example, [at least one of A, B, or C] represents A or B or C or AB or AC or BC or ABC (i.e., A and B and C).

[0145] Various aspects of the illustrative embodiments are described using terms commonly used by those skilled in the art to convey the substance of their work to others skilled in the art. For example, the term "connected" refers to a direct electrical connection between the things being connected, without any intermediate devices / components, while the term "coupled" refers to a direct electrical connection between the connected objects, or an indirect connection through one or more passive or active intermediate devices / components. In another example, the term "circuit" refers to one or more passive and / or active components that are arranged to cooperate with each other to provide the desired functionality. In addition, as used herein, based on the context of specific values ​​described herein or known in the art, the terms "substantially", "approximately", "about", etc. may be used to generally refer to within + / -20% of a target value, such as within + / -10% of a target value.

[0146] Numerous other variations, substitutions, changes, alterations, and modifications may be determined by those skilled in the art, and the present disclosure is intended to include all such variations, substitutions, and modifications that fall within the scope of the examples and the appended claims. Note that all optional features of the above-described devices may also be implemented with respect to the methods or processes described herein, and that the details in the examples may be used anywhere in one or more embodiments.

[0147] Explanation of terms

[0148] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms. Unless the context clearly requires otherwise, throughout the specification and claims:

[0149] The words “include”, “including”, etc. shall be construed as being inclusive rather than exclusive or exhaustive; that is, in the sense of “including but not limited to”.

[0150] “Connected,” “coupled,” or any variation thereof, refers to any direct or indirect connection or coupling between two or more elements; the coupling or connection between the elements may be physical, logical, or a combination thereof.

[0151] When used to describe this Specification, the words “herein,” “above,” “hereunder,” and words of similar import shall refer to this Specification as a whole and not to any particular portion of this Specification.

[0152] “Or” refers to a list of two or more items and covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0153] The singular forms "a", "an" and "the" are intended to include any appropriate plural forms.

[0154] Words indicating directions, such as "vertical," "lateral," "horizontal," "upward," "downward," "forward," "backward," "inward," "outward," "perpendicular," "lateral," "left," "right," "front," "backward," "top," "bottom," "below," "above," "lower," etc., used in this specification and any appended claims, if any, depend on the specific orientation of the device being described and illustrated. The subject matter described herein can assume a variety of alternative orientations. Accordingly, these directional terms are not strictly defined and should not be narrowly interpreted.

[0155] Unless expressly stated to the contrary, the indefinite articles "a," "an," and "an" used in the specification and claims should be understood to mean "at least one."

[0156] The phrase "and / or" used herein in the specification and claims should be understood to mean "one or both" of the elements so coupled, i.e., elements that are present in combination in some cases and separate in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so coupled.

[0157] In addition to the elements expressly identified by the "and / or" clause, other elements may optionally be present, whether or not related to those elements expressly identified. Thus, as a non-limiting example, in one embodiment, when used in conjunction with open language such as "comprising," a reference to "A and / or B" may include only A (optionally including elements other than B); in another embodiment only B (optionally including elements other than A); in another embodiment both A and B (optionally including other elements); etc.

[0158] As used herein in the specification and claims, the phrase "at least one" refers to a list of one or more elements and should be understood to refer to at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of each element specifically listed in the list of elements, nor exclude any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than the elements specifically identified in the list of elements to which the phrase "at least one" refers, whether or not these elements are related to those elements specifically identified.

[0159] Thus, as a non-limiting example, in one embodiment, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently "at least one of A and / or B") may refer to at least one, optionally including multiple A's, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in another embodiment, to at least one (optionally including more than one), A, and at least one, optionally including more than one B (and optionally including other elements); etc.

[0160] As used herein, the term "between" is intended to be inclusive unless otherwise stated. For example, "between A and B" includes A and B unless otherwise stated.

[0161] In addition, the words and terms used herein are for description only and should not be considered as limiting. The words "include", "comprising", or "having", "containing", "involving" and their variations used herein are intended to include the items listed below and their equivalents as well as additional items.

[0162] In the claims and the above description, all transitional phrases, such as "include", "comprising", "carrying", "having", "containing", "involving", "holding", "consisting of", etc., shall be understood as open-ended, i.e., including but not limited to. Only the transitional phrases "consisting of" and "mainly composed of" shall be closed or semi-closed transitional phrases, respectively.

[0163] Numerous variations, substitutions, alterations, and modifications may be identified by those skilled in the art, and this disclosure is intended to embrace all such variations, substitutions, alterations, and modifications that fall within the scope of the appended claims.

[0164] To assist the United States Patent and Trademark Office (USPTO) and any reader of any patent that issues from this application in interpreting the claims appended hereto, applicants wish to note that applicants: (a) do not intend to invoke 35 U.S.C. § 112(f) as it existed on the date of filing this application, unless the words “for” or “for” are specifically used in a particular claim; and (b) do not intend that any statement in this disclosure should limit the present disclosure in any manner not reflected in the appended claims.

[0165] Therefore, the present invention should not be considered as limited to the specific embodiments described above. After reading this disclosure, those skilled in the art to which the present invention is directed will easily understand various modifications, equivalent processes, and numerous structures to which the present invention may be applicable.

[0166] Select Example

[0167] Example 1 provides a method, comprising: converting a depth image including multiple depth pixels into a disparity image including multiple disparity pixels, wherein a disparity value of each of the multiple disparity images is determined based on depth values ​​of different depth pixels among the multiple depth pixels; determining a boundary weight of a target depth pixel among the multiple depth pixels based on a gradient magnitude of the target depth pixel in the depth image and a gradient magnitude of a luminance pixel in a luminance image; determining an energy of the target depth pixel based on the boundary weight; determining a new depth value of the target depth pixel by optimizing the energy; and updating the depth image by assigning the new depth value to the target depth pixel.

[0168] Example 2 provides the method of Example 1, wherein: the luminance image and the depth image capture the same object, the luminance image includes a plurality of luminance pixels, the plurality of luminance pixels include luminance pixels, and each corresponding luminance pixel in the plurality of luminance pixels corresponds to a corresponding depth pixel in the plurality of depth pixels.

[0169] Example 3 provides the method of Example 1, wherein: the target depth pixel represents the same object location as the brightness pixel.

[0170] Example 4 provides the method of Example 1, wherein: determining the energy of the target depth pixel based on the boundary weight includes: determining the spatial error energy of the target depth pixel based on the boundary weight, wherein optimizing the energy includes optimizing the spatial error energy by reducing the difference between the depth value of the target depth pixel and the depth value of another depth pixel adjacent to the target depth pixel in the depth image.

[0171] Example 5 provides the method of Example 1, wherein: determining the energy of the target depth pixel based on the boundary weight also includes: determining the conditional error energy of the target depth pixel based on the depth value of the target depth pixel in the depth image and the luminance value of the luminance pixel in the luminance image, wherein the conditional error energy indicates a measure of the uncertainty of the depth value of the target depth pixel given the luminance value of the luminance pixel.

[0172] Example 6 provides the method of Example 1, wherein: the disparity value is proportional to the inverse of the depth value.

[0173] Example 7 provides the method of Example 1, wherein: the depth image and the luminance image are generated based on image data from the same image sensor.

[0174] Example 8 provides the method of Example 1, further including: instructing the illuminator component to project modulated light into a local area including an object; instructing the camera component to capture reflected light from at least a portion of the object; and generating the depth image based on a phase shift between the reflected light projected into the local area and the modulated light.

[0175] Example 9 provides the method of Example 8, further comprising: generating the brightness image based on the brightness of the reflected light.

[0176] Example 10 provides the method of Example 8, wherein the reflected light is a first reflected light, and the method further includes: instructing the camera assembly to capture a second reflected light from at least a portion of the object; and generating the brightness image based on the brightness of the second reflected light, wherein the second reflected light has a different wavelength from the first reflected light.

[0177] Example 11 provides a system comprising: an illuminator component configured to project modulated light into a local area including an object; a camera component configured to capture reflected light from at least a portion of the object; and a controller configured to: generate a depth image from the reflected light, the depth image comprising a plurality of depth pixels and capturing at least a portion of the object, generate a brightness image comprising a plurality of brightness pixels and capturing at least a portion of the object, each brightness pixel corresponding to a different depth pixel, for each corresponding depth pixel in the plurality of depth pixels, determine a corresponding energy based on a gradient magnitude of the corresponding depth pixel in the depth image and a gradient magnitude of the brightness pixel in the brightness image, and generate an enhanced depth image by fusing the depth image with the brightness image based on the respective energies of the plurality of depth pixels.

[0178] Example 12 provides the system of Example 11, wherein fusing the depth image with the luminance image based on respective energies of the plurality of depth pixels comprises: for each respective depth pixel in the plurality of depth pixels, optimizing the respective energy.

[0179] Example 13 provides the system of Example 12, wherein the controller is configured to determine the corresponding energy based on the gradient amplitude of the corresponding depth pixel in the depth image and the gradient amplitude of the luminance pixel in the luminance image by the following steps: determining the spatial error energy of the corresponding depth pixel based on the gradient amplitude of the corresponding depth pixel in the depth image and the gradient amplitude of the luminance pixel in the luminance image, wherein optimizing the corresponding energy includes optimizing the spatial error energy by reducing the difference between the depth value of the corresponding depth pixel and the depth value of another depth pixel adjacent to the corresponding depth pixel in the depth image.

[0180] Example 14 provides the system of Example 11, wherein the controller is configured to further determine the corresponding energy based on the gradient magnitude of the corresponding depth pixel in the depth image and the gradient magnitude of the luminance pixel in the luminance image by the following steps: determining the conditional error energy of the corresponding depth pixel based on the depth value of the corresponding depth pixel in the depth image and the luminance value of the luminance pixel in the luminance image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the corresponding depth pixel given the luminance value of the luminance pixel.

[0181] Example 15 provides the system of Example 11, wherein the controller is configured to generate the depth image and the brightness image based on the reflected light by the following steps: generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and generating the brightness image based on the brightness of the reflected light.

[0182] Example 16 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising: converting a depth image including multiple depth pixels into a disparity image including multiple disparity pixels, wherein a disparity value of each of the multiple disparity images is determined based on depth values ​​of different depth pixels among the multiple depth pixels; determining a boundary weight of a target depth pixel among the multiple depth pixels based on a gradient magnitude of the target depth pixel in the depth image and a gradient magnitude of a luminance pixel in a luminance image; determining an energy of the target depth pixel based on the boundary weight; determining a new depth value of the target depth pixel by optimizing the energy; and updating the depth image by assigning the new depth value to the target depth pixel.

[0183] Example 17 provides one or more non-transitory computer-readable media of Example 16, wherein the operation further includes: the luminance image and the depth image capture the same object, the luminance image includes a plurality of luminance pixels, the plurality of luminance pixels include luminance pixels, and each corresponding luminance pixel in the plurality of luminance pixels corresponds to a corresponding depth pixel in the plurality of depth pixels.

[0184] Example 18 provides one or more non-transitory computer-readable media of Example 16, wherein determining the energy of the target depth pixel based on the boundary weight includes: determining a spatial error energy of the target depth pixel based on the boundary weight, wherein optimizing the energy includes optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel adjacent to the target depth pixel in the depth image.

[0185] Example 19 provides one or more non-transitory computer-readable media of Example 18, wherein determining the energy of the target depth pixel based on the boundary weight also includes: determining a conditional error energy of the target depth pixel based on a depth value of the target depth pixel in the depth image and a luminance value of a luminance pixel in the luminance image, wherein the conditional error energy indicates a measure of uncertainty in the depth value of the target depth pixel given the luminance value of the luminance pixel.

[0186] Example 20 provides the one or more non-transitory computer-readable media of Example 16, wherein the depth image and the luminance image are generated based on image data from the same image sensor.

Claims

1. A method comprising: converting a depth image including a plurality of depth pixels into a disparity image including a plurality of disparity pixels, wherein a disparity value of each of the plurality of disparity images is determined based on depth values ​​of different depth pixels among the plurality of depth pixels; Determining a boundary weight of a target depth pixel among the plurality of depth pixels based on a gradient magnitude of the target depth pixel in the depth image and a gradient magnitude of a luminance pixel in the luminance image; determining an energy of the target depth pixel based on the boundary weight; determining a new depth value of the target depth pixel by optimizing the energy; and The depth image is updated by assigning new depth values ​​to the target depth pixels.

2. The method according to claim 1, wherein: The brightness image and the depth image capture the same object, The luminance image includes a plurality of luminance pixels, the plurality of luminance pixels include luminance pixels, and Each respective luma pixel of the plurality of luma pixels corresponds to a respective depth pixel of the plurality of depth pixels.

3. A method according to any one of the preceding claims, wherein the target depth pixel represents the same object location as the brightness pixel.

4. The method according to any one of the preceding claims, wherein determining the energy of the target depth pixel based on the boundary weight comprises: determining a spatial error energy of the target depth pixel based on the boundary weight, The optimizing energy includes optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel adjacent to the target depth pixel in the depth image.

5. The method according to claim 4, wherein determining the energy of the target depth pixel based on the boundary weight further comprises: determining a conditional error energy of the target depth pixel based on a depth value of the target depth pixel in the depth image and a luminance value of the luminance pixel in the luminance image, Wherein the conditional error energy indicates a measure of uncertainty of a depth value of the target depth pixel given a luma value of the luma pixel.

6. A method according to any preceding claim, wherein the disparity value is proportional to the inverse of the depth value.

7. The method according to any of the preceding claims, wherein the depth image and the brightness image are generated based on image data from the same image sensor.

8. The method according to any one of the preceding claims, further comprising: instructing the illuminator assembly to project modulated light into a local area including the object; instructing a camera assembly to capture reflected light from at least a portion of the object; and The depth image is generated based on a phase shift between the reflected light and the modulated light projected into the local area.

9. The method according to claim 8, further comprising: The brightness image is generated based on the brightness of the reflected light.

10. The method according to claim 8 or claim 9, wherein the reflected light is a first reflected light, and the method further comprises: instructing the camera assembly to capture second reflected light from at least a portion of the object; and generating the brightness image based on the brightness of the second reflected light, The second reflected light has a wavelength different from that of the first reflected light.

11. A system comprising: an illuminator assembly configured to project modulated light into a localized area including the object; a camera assembly configured to capture reflected light from at least a portion of the object; as well as The controller is configured as: generating a depth image from the reflected light, the depth image comprising a plurality of depth pixels and capturing at least a portion of the object, generating a luminance image comprising a plurality of luminance pixels and capturing at least a portion of the object, each luminance pixel corresponding to a different depth pixel, For each corresponding depth pixel of the plurality of depth pixels, determining a corresponding energy based on a gradient magnitude of the corresponding depth pixel in the depth image and a gradient magnitude of a luma pixel in the luma image, and An enhanced depth image is generated by fusing the depth image with the luminance image based on respective energies of a plurality of depth pixels.

12. The system of claim 11, wherein fusing the depth image with the luminance image based on respective energies of the plurality of depth pixels comprises: For each respective depth pixel in the plurality of depth pixels, a respective energy is optimized.

13. The system of claim 12, wherein the controller is configured to determine the corresponding energy based on the gradient magnitude of the corresponding depth pixel in the depth image and the gradient magnitude of the luminance pixel in the luminance image by the following steps: Determining a spatial error energy of a corresponding depth pixel based on a gradient magnitude of a corresponding depth pixel in the depth image and a gradient magnitude of a luminance pixel in the luminance image, The optimizing the corresponding energy comprises optimizing the spatial error energy by reducing a difference between a depth value of the corresponding depth pixel and a depth value of another depth pixel adjacent to the corresponding depth pixel in the depth image.

14. The system according to any one of claims 11 to 13, wherein the controller is configured to determine the corresponding energy based on the gradient magnitude of the corresponding depth pixel in the depth image and the gradient magnitude of the luminance pixel in the luminance image by further following the steps of: determining a conditional error energy of the corresponding depth pixel based on a depth value of the corresponding depth pixel in the depth image and a luminance value of the luminance pixel in the luminance image, The conditional error energy indicates a measure of uncertainty of a depth value of a corresponding depth pixel given a luma value of the luma pixel.

15. The system according to any one of claims 11 to 14, wherein the controller is configured to generate the depth image and the brightness image based on the reflected light by the following steps: generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area; and The brightness image is generated based on the brightness of the reflected light.

16. One or more non-transitory computer-readable media storing instructions executable to perform operations comprising: converting a depth image including a plurality of depth pixels into a disparity image including a plurality of disparity pixels, wherein a disparity value of each of the plurality of disparity images is determined based on depth values ​​of different depth pixels among the plurality of depth pixels; Determining a boundary weight of a target depth pixel among the plurality of depth pixels based on a gradient magnitude of the target depth pixel in the depth image and a gradient magnitude of a luminance pixel in the luminance image; determining an energy of the target depth pixel based on the boundary weight; determining a new depth value of the target depth pixel by optimizing the energy; and The depth image is updated by assigning new depth values ​​to the target depth pixels.

17. The one or more non-transitory computer-readable media of claim 16, wherein the operations further comprise: The brightness image and the depth image capture the same object, The luminance image includes a plurality of luminance pixels, the plurality of luminance pixels include luminance pixels, and Each respective luma pixel of the plurality of luma pixels corresponds to a respective depth pixel of the plurality of depth pixels.

18. The one or more non-transitory computer-readable media of claim 16 or claim 17, wherein determining the energy of the target depth pixel based on the boundary weight comprises: determining a spatial error energy of the target depth pixel based on the boundary weight, The optimizing energy includes optimizing the spatial error energy by reducing a difference between a depth value of the target depth pixel and a depth value of another depth pixel adjacent to the target depth pixel in the depth image.

19. The one or more non-transitory computer-readable media of claim 18, wherein determining the energy of the target depth pixel based on the boundary weight further comprises: determining a conditional error energy of the target depth pixel based on a depth value of the target depth pixel in the depth image and a brightness value of the brightness pixel in the brightness image, Wherein the conditional error energy indicates a measure of uncertainty of a depth value of the target depth pixel given a luma value of the luma pixel.

20. The one or more non-transitory computer-readable media of any one of claims 16 to 19, wherein the depth image and the luminance image are generated based on image data from the same image sensor.

Images