Enhancing depth estimation with luminance images
By combining brightness images and depth images processing technology, the problem of inaccurate depth estimation at the boundary and reflectivity boundaries of ToF camera system is solved, and clearer and more accurate depth-enhanced images are generated, which enhances the application potential of ToF camera system.
Patent Information
- Application Number
- CN202380028457.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-12-16
- Filing Date
- 2023-02-16
- Publication Date
- 2024-11-19
AI Technical Summary
ToF camera systems often have low resolution and noise problems when capturing depth images, especially when it is difficult to accurately estimate depth at boundaries and reflectivity boundaries, limiting their use in applications such as robots and autonomous driving.
Brightness images are used to enhance ToF depth estimation, and the controller processes the fusion of brightness images and depth images at the pixel level, identify boundaries and adjust the values of depth pixels to generate clearer depth enhancement images.
Improve the resolution and accuracy of depth images, especially in boundary and planar areas, enhance the depth estimation capability of ToF camera systems, and is suitable for a variety of application scenarios.
Smart Images

Figure BDA0005048323060000093 
Figure HDA0005048323070000011 
Figure HDA0005048323070000021
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 310,859, filed on February 16, 2022, the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present disclosure relates generally to depth estimation and, more particularly, to enhancing depth estimation using brightness images. Background Art
[0004] One technique for measuring depth is to directly or indirectly calculate the time it takes for a signal to travel from a source on a sensor to a reflecting surface and back to the sensor. The time of travel is proportional to the distance from the sensor to the reflecting surface. This travel time is often referred to as time of flight (ToF). ToF sensors can use various types of signals, the most common being sound and light. Given the advantages of light in terms of speed, range, power, and weight, some sensors use light as a carrier. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] For a more complete understanding of the present disclosure and its features and advantages, please refer to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts, and in which:
[0006] Figure 1 A depth estimation system according to some embodiments of the present disclosure is shown;
[0007] Figure 2A shows a continuous wave of a projection signal according to some embodiments of the present disclosure;
[0008] Figure 2B shows a continuous wave of a captured signal according to some embodiments of the present disclosure;
[0009] Figure 3 shows a period of a continuous wave of modulated light according to some embodiments of the present disclosure;
[0010] Figure 4 An example image sensor according to some embodiments of the present disclosure is shown;
[0011] Figure 5 Some embodiments of the present disclosure are shown Figure 4 Example pixels of an image sensor in;
[0012] Figure 6 is a block diagram illustrating a controller according to some embodiments of the present disclosure;
[0013] Fig. 7A shows an example depth image according to some embodiments of the present disclosure;
[0014] Figure 7B The present invention shows some embodiments of the present invention. Figure 5 Example brightness image corresponding to the depth image in A;
[0015] Figure 7C According to some embodiments of the present disclosure, Figure 5 The depth image in A and Figure 5 Example depth-enhanced images generated from the brightness image in B;
[0016] Figure 8 An example system including a depth estimation system according to some embodiments of the present disclosure is shown;
[0017] Fig. 9 A mobile device incorporating a depth estimation system according to some embodiments of the present disclosure is shown;
[0018] Fig.10 An entertainment system incorporating a depth estimation system according to some embodiments of the present disclosure is shown;
[0019] Fig.11 An example robot incorporating a depth estimation system according to some embodiments of the present disclosure is shown;
[0020] Fig.12 is a flowchart illustrating a method of enhancing depth estimation using a brightness image according to some embodiments of the present disclosure. DETAILED DESCRIPTION
[0021] Overview
[0022] Depth estimation is a fundamental task in 3D computer vision. The high-quality and dense depth images produced by ToF camera systems play an important role in many applications, such as robotics, human-computer interaction, indoor navigation, self-driving cars, object tracking, and gesture recognition. ToF camera systems are range imaging systems. ToF camera systems typically include a light source that projects light and an imaging sensor that receives reflected light. A ToF camera system can estimate the distance between the imaging sensor and the object by measuring the round trip of light. A continuous wave ToF camera system can project multiple cycles of a continuous light wave and determine the distance based on the phase difference between the projected light and the received reflected light. A depth image can be generated based on the phase difference.
[0023] However, ToF depth maps are usually captured with low resolution, different types of noise or missing values. ToF camera systems are usually unable to accurately estimate the depth of boundaries, such as the edge of an object, reflectivity boundaries (e.g., the boundary between two areas with different reflectivity characteristics), etc. Inaccurate depth estimation may limit the use of ToF camera systems in the above applications. Therefore, there is a need to improve depth estimation techniques.
[0024] Embodiments of the present disclosure relate to a depth estimation system that can use a brightness image to enhance ToF depth estimation. The brightness image can be an active brightness image, such as an infrared (IR) image, or an RGB (red, green, blue) image. The depth estimation system can simultaneously acquire a brightness image and a ToF depth estimate. The brightness image and the ToF depth estimate can be based on the same light source (e.g., IR) or different light sources (e.g., visible light for a brightness image and IR for a ToF depth estimate). The brightness image can better detect boundaries and can be used to enhance the depth image generated by a ToF camera system.
[0025] An example of a depth estimation system includes an illuminator assembly, an image sensor, and a controller. The illuminator assembly can project light to illuminate a local area, such as an area including an object. The light can be modulated light, such as modulated IR. The illuminator assembly can project pulsed light. Alternatively, the illuminator assembly can project one or more continuous waves, such as continuous waves of different frequencies. The object can reflect at least a portion of the projected light. The image sensor captures at least a portion of the reflected light and generates image data. The image sensor can include a pixel array and multiple analog-to-digital converters (ADCs). The pixel array can convert photons into charges that constitute an analog signal. The ADC can convert an analog signal into a digital signal. Each pixel can be associated with two ADCs, each ADC receiving a different analog signal from a pixel and generating a different digital signal. In some embodiments, the pixel array includes pixel columns. Each column can be associated with two ADCs.
[0026] The controller receives the digital signal generated by the ADC and can generate a depth image and a corresponding brightness image based on the digital signal. In some embodiments, the depth image is based on the phase shift between the captured light and the projected light, and the brightness image is based on the brightness of the captured light. The depth image may include a plurality of depth pixels, each of which may correspond to a pixel in the brightness image. The controller uses the brightness image to enhance the depth image. The controller can perform enhancement processing at the pixel level. For example, the controller can identify a target depth pixel in the depth image and change the depth value of the target depth pixel based on one or more brightness pixels in the brightness image. The controller can change the depth value of some or all of the depth pixels in the depth image and generate a new depth image.
[0027] The depth estimation system in the present disclosure can enhance the depth estimation of the boundary using the brightness image showing clearer boundaries, and can also smooth the plane. The enhanced depth image generated by the depth estimation system can show the boundary better than the conventional ToF depth image. With more accurate depth estimation, the enhanced depth image can be used in various applications.
[0028] Depth Estimation System Example
[0029] Figure 1 A depth estimation system 100 is shown in accordance with some embodiments of the present disclosure. The depth estimation system 100 may use ToF technology to generate a depth image. The depth estimation system 100 includes an illuminator assembly 110, an image sensor 120, and a controller 130. The illuminator assembly 110 includes an emitter 160 and a diffuser 165. In alternative configurations, different and / or additional components may be included in the depth estimation system 100. In addition, the functionality attributed to one component of the depth estimation system 100 may be implemented by a different component included in the depth estimation system 100 or a different system than that shown. For example, the illuminator assembly 110 may not include a diffuser or multiple diffusers. As another example, the image sensor 120 may not include a lens or include more than one lens.
[0030] The illuminator assembly 110 projects light 170 to a local area including the object 140. The emitter 160 is a light source that emits light ("emitted light"). In some embodiments, the emitter 160 may include a laser, such as an IR or near IR (NIR) laser, an edge-emitting laser, a vertical cavity surface emitting laser (VCSEL), etc. In other embodiments, the emitter 160 may include one or more light emitting diodes (LEDs). The emitter 160 may emit light in the visible light band (i.e., about 380nm to 750nm), the near infrared band (i.e., about 750nm to 1mm), the ultraviolet band (i.e., 10nm to 380nm), the short-wave infrared (SWIR) band (e.g., about 900nm to 2200nm), some other part of the electromagnetic spectrum, or some combination thereof. In some embodiments, the illuminator assembly 110 may include a plurality of emitters 160, each of which may emit a different wavelength. For example, the illuminator assembly 110 may include a first emitter that emits IR and a second emitter that emits visible light. Before light 170 is projected into the local area, diffuser 165 spreads or scatters the emitted light. Diffuser 165 can also control the brightness of the emitted light. In some embodiments, diffuser 165 can be translucent or semi-transparent. In other embodiments, illuminator assembly 110 can include more, fewer, or different components. For example, illuminator assembly 110 can include one or more additional diffusers to direct light from emitter 160 to one or more additional objects in the local area.
[0031] The illuminator assembly 110 may project the light 170 as modulated light, for example, according to a periodic modulation waveform. An example of a periodic modulation waveform may be a sinusoidal modulation waveform. The frequency of the periodic modulation waveform is the frequency of the modulated light.
[0032] The illuminator assembly 110 can project one or more continuous waves. For a single continuous wave, the illuminator assembly 110 can project multiple cycles. Different continuous waves may have different wavelengths and frequencies. For example, the illuminator assembly 110 can project a continuous wave with a modulation frequency in the range of 50 MHz to 200 MHz. In one embodiment, the illuminator assembly 110 includes multiple (i.e., at least two) light projectors. The light projectors can project continuous waves with different frequencies. The light projectors can alternate and project continuous waves at different times. For example, a first light projector projects a first continuous wave with a first frequency during a first time period. After the first time period, a second light projector projects a second continuous wave with a second frequency during a second time period. After the second time period, a third light projector projects a third continuous wave with a third frequency during a third time period. These three continuous waves can constitute a cycle. This cycle can be repeated.
[0033] In another embodiment, the illuminator assembly 110 may include a light projector that projects all three continuous waves. In other embodiments, the illuminator assembly 110 may project a different number of continuous waves, such as two or more than three. One cycle may constitute a frame. The total time of a cycle may be 10-20ms. The illuminator assembly 110 may project light through multiple cycles to obtain multiple frames. There may be time intervals between cycles. More information about modulated light with multiple frequencies is provided below in conjunction with Figure 2.
[0034] At least a portion of object 140 is illuminated by light 170. For simplicity and illustration purposes, Figure 1 The object 140 in the embodiment has a cubic shape. In other embodiments, the object 140 may have other shapes or structures. Figure 1 Not shown in the figure, the local area may also include other objects that may be illuminated by the light. The object reflects the light, reflecting light 180, and the reflected light 180 may be captured by the image sensor 120.
[0035] Image sensor 120 captures light from at least a portion of the local area illuminated with light 170. For example, image sensor 120 captures reflected light 180 and generates image data based on reflected light 180. Reflected light 180 may be IR. In some embodiments, image sensor 120 may also capture visible light reflected by object 140. Visible light may be projected by illuminator assembly 110, ambient light, or a combination of both. Even in Figure 1In some embodiments, the image sensor 120 is separate from the illuminator assembly 110. In some embodiments, the image sensor 120 is also located at the same location as the illuminator assembly 110 (for example, it can be part of the same device). One embodiment of the image sensor 120 is Figure 4 Image sensor 400 in.
[0036] The image sensor 120 includes a lens 190 and a pixel array 195. The lens 190 receives the reflected light 180 and directs the reflected light 180 to the pixel array 195. The pixel array 195 includes a plurality of pixels 197. Figure 1 Although the pixels 197 shown in FIG. 1 are arranged in one column, the pixels 197 of the pixel array 195 may also be arranged in multiple columns.
[0037] In some embodiments, the pixel includes a photodiode that is sensitive to light and converts collected photons into electric charge (e.g., photoelectrons). Each photodiode has one or more storage areas for storing electric charge. The image sensor 120 can be a ToF sensor and a brightness sensor. The pixel can be a depth sensing pixel, a brightness sensing pixel, or both. The depth sensing pixel is configured to present a depth output signal that depends on the distance from the pixel array 195 to the trajectory of the object 140 imaged on the depth sensing pixel. Such a distance is the "depth" of the trajectory of the object 140. Each depth sensing pixel can independently determine the distance to the object 140 seen by the pixel. The depth output signal of the depth sensing pixel in the pixel array 195 can be used to generate a depth image of a local area. The brightness sensing pixel is configured to present a brightness output signal that depends on the brightness of the light reflected from the trajectory of the object 140 imaged on the brightness sensing pixel. The brightness output signal of the brightness sensing pixel in the pixel array 195 can be used to generate a brightness image of a local area. The brightness image can be an active brightness image. An example of a brightness image is an IR image. In some embodiments, each pixel of the pixel array 195 can generate a depth output signal and a brightness output signal from the reflected light captured by the pixel. In other embodiments, the pixel array 195 includes two groups of pixels 197: one group for sensing depth and the other group for detecting brightness. The output signal of the pixel array 195 can be an analog signal, such as an electric charge.
[0038] In embodiments where the illuminator assembly 110 projects multiple continuous waves or multiple cycles of modulated light, the image sensor 120 may be synchronized with the projection of the illuminator assembly 110. For example, the image sensor 120 may have one or more exposure intervals during which the image sensor 120 exposes a portion of a local area and accumulates charge in the pixel array 195. Outside of the exposure interval, the image sensor 120 does not expose. In some embodiments, the exposure interval of the image sensor 120 may be synchronized with the continuous wave or cycle projected by the illuminator assembly 110. For example, the exposure interval begins before or when the continuous wave or cycle begins and ends after or when the continuous wave or cycle ends. In other embodiments, the image sensor 120 may have multiple exposure intervals for a single continuous wave. For example, the image sensor 120 may perform multiple exposures during a continuous wave, and the multiple exposures may correspond to different phase offsets. In one example, a continuous wave has three exposure intervals with different phase offsets, such as 0° (0), 120° (2π / 3), and 240° (4π / 3). There may be time intervals between the exposure intervals. The time intervals may be 1-2 milliseconds (ms). The exposure interval may have a constant duration, for example approximately 100 microseconds (μs).In alternative embodiments, the exposure interval may have a different duration.
[0039] In some embodiments, the image sensor 120 can be scanned using a global shutter. The image sensor 120 includes a global shutter that can be opened and scanned during each exposure interval and closed at the end of the exposure interval. Additionally or alternatively, the image sensor 120 can include an adjustable filter. The tunable filter blocks light from reaching the detector and can be installed at any position in the optical path of the reflected light 180. For example, the tunable filter is attached to the top of the pixel array 195 or the front of the image sensor 120. The tunable filter can be switched between on (active) and off (inactive). The tunable filter can be inactive during the exposure interval and active at the end of the exposure interval. When the tunable filter is inactive, light can pass through the tunable filter and reach the pixel array 195. When the tunable filter is active, light is blocked outside the pixel array 195.
[0040] In some embodiments, when the tunable filter is active, it may pass light of a particular wavelength (or a particular wavelength band) but block light of other wavelengths. For example, the tunable filter may pass light of a wavelength projected by illuminator assembly 110 (e.g., light 170) but block light of other wavelengths, which may, for example, reduce noise in image data captured by image sensor 120. In examples where light 170 is IR, the tunable filter may block visible light. In other embodiments, when the tunable filter is active, it may block light of all wavelengths to avoid charge accumulation in pixel array 195. In embodiments where the tunable filter blocks all light, dark noise calibration may be performed on pixel array 195.
[0041] The image sensor 120 may read out photoelectrons from the pixel array 195 to obtain image data. The photoelectrons may be analog signals stored in a storage area of each pixel of the pixel array 195. During readout, the image sensor 120 may convert the analog signal into a digital signal, for example, by an ADC. The analog signal may be amplified by an amplifier before being converted into a digital signal.
[0042] In embodiments where the illuminator assembly 110 includes multiple light projectors, photoelectrons corresponding to modulated light pulses projected by different light projectors may be stored in separate storage areas of each photodiode. The image sensor 120 may read out separate storage areas to obtain image data. In some embodiments, the image sensor 120 may read out all image data stored in the pixel array 195. In other embodiments, the image sensor 120 may read out some image data stored in the pixel array 195. For example, in embodiments where the exposure interval of the pixel array 195 is synchronized with the continuous wave projected by the illuminator assembly 110, the image sensor 120 may perform multiple readout intervals on the continuous wave. Each readout interval may correspond to a different phase offset. In one example, a continuous wave has three readout intervals at three different phase offsets (e.g., 0°, 120°, and 240°). There may be a time interval between the readout intervals. The time interval may be 1-2 milliseconds (ms). The readout interval may have a constant duration, such as approximately 100 microseconds (μs). In alternative embodiments, the readout intervals may have different durations.
[0043] The controller 130 controls the illuminator assembly 110 and the image sensor 120. For example, the controller 130 provides an illumination instruction to the illumination assembly 110, and the illumination assembly 110 projects light 170 according to the illumination instruction. The controller 130 may also provide an imaging instruction to the image sensor 120, and the image sensor 120 performs exposure and reads out image data according to the imaging instruction.
[0044] The controller 130 also uses the image data from the image sensor 120 to determine depth information. For example, the controller 130 can generate a depth image based on the image data. The depth image includes a plurality of depth pixels. Each depth pixel has a value corresponding to an estimated depth, for example, an estimated distance from the trajectory of the object 140 to the pixel array 195. A single depth image may also be referred to as a depth frame or a depth map. In an embodiment where the illuminator assembly 110 projects a continuous wave of modulated light, the controller 130 may determine the depth information based on the phase shift between the light 170 projected by the illuminator assembly 110 and the reflected light 180. In an embodiment where the image sensor 120 reads out image data corresponding to different phase offsets of the modulated light, the controller 130 may perform phase unwrapping to determine the depth information. In some embodiments (e.g., an embodiment where the illuminator assembly 110 projects multiple cycles of modulated light), the controller 130 may generate multiple depth frames.
[0045] The controller 130 may also generate a brightness image corresponding to the depth image. The image sensor 120 may generate image data of the brightness image and image information of the depth image from the same light (e.g., reflected light 180). In some embodiments, the brightness image and the depth image are generated simultaneously. For example, the image sensor 120 may read out image data of the brightness image and image information of the depth image simultaneously. In one embodiment, the image data for the brightness image and the image for the depth image are the same image data. The brightness image may include a plurality of brightness pixels. Each brightness pixel has a value corresponding to light intensity (e.g., IR intensity). The brightness pixels in the brightness image may correspond to the depth pixels in the depth image. The brightness pixels and the depth pixels may be generated by light reflected from the same track of the object 140. For example, the brightness pixels and the depth pixels may be generated based on a signal from the same pixel of the pixel array 195, and the pixel captures the light reflected from the track of the object 140.
[0046] Controller 130 may further enhance the depth estimation in the depth image by fusing the depth image with the brightness image. The brightness image may display one or more boundaries of object 140 more clearly than the depth image. For example, one or more depth pixels representing at least a portion of the boundary of object 140 may be invalid. The boundary may be an edge of object 140. Alternatively, the boundary is a boundary between two areas of object 140 having different reflectivity characteristics, such as a boundary between a fluorescent strip having relatively high reflectivity and a low reflectivity surface. Controller 130 may utilize the more accurate information of the boundary of object 140 in the brightness image to generate an enhanced depth image that includes a better depth estimate than the original depth image. Figure 6 Certain aspects of controller 130 are described.
[0047] Modulation Signal Example
[0048] Figure 2A 2 shows a continuous wave of a projection signal 210 according to some embodiments of the present disclosure. The projection signal 210 is a modulated signal, for example, Figure 1 The projection signal 210 has a sinusoidal modulation waveform. Figure 2A In the embodiment, the sinusoidal modulation waveform can be represented by the following equation:
[0049] S(t)=A s sin(2πft)+B s
[0050] Where t represents time, s represents the optical power of the projection signal, f is the frequency of the modulation signal (i.e., modulation frequency), π is a mathematical constant, and A s represents the amplitude of the modulated signal, B s Represents the offset of the modulated signal, which may include the attenuated original offset and / or the offset due to the presence of ambient light (such as sunlight or light from artificial sources).
[0051] Figure 2B 2 shows a continuous wave of a captured signal 220 according to some embodiments of the present disclosure. The captured signal 220 is a captured portion of the modulated light reflected by an object illuminated by the projection signal 210. The captured signal 220 may be captured by the pixel array 195 and may be Figure 1 At least a portion of the reflected light 180 in Figure 2A In the embodiment of FIG. 2 , the captured signal 220 may be represented by the following equation:
[0052]
[0053]
[0054]
[0055] Where r represents the optimal power of the captured signal 220, α represents the attenuation factor of the captured signal 200, represents the phase shift between the waveform of the captured signal 220 and the waveform of the projected signal 210, δ is the time delay between the captured signal 210 and the projected signal 220, d represents the distance from the pixel array 195 to the object 140 (ie, the depth of the object 140), and c is the speed of light.
[0056] Figure 3 A cycle 310 of continuous waves 315A-C, 325A-C, and 335A-C of modulated light is shown according to some embodiments of the present disclosure. Figure 3In the embodiment, continuous waves 315A-C, 325A-C, and 335A-C are sinusoidal waves. In other embodiments, continuous waves 315A-C, 325A-C, and 335A-C may have different waveforms. Continuous waves 315A-C have a frequency 317, continuous waves 325A-C have a frequency 327, and continuous waves 335A-C have a frequency 337. The three frequencies 317, 327, and 337 are different from each other. Figure 3 In the embodiment of the present invention, frequency 317 is less than frequency 327, and frequency 327 is less than frequency 337. The three frequencies 317, 327, and 337 may be in the range of 50 to 200 MHz or higher.
[0057] In one embodiment, cycle 310 may be Figure 1 310 may be a period during which the illuminator assembly 110 projects modulated light. In another embodiment, the period 310 may be an exposure period of the image sensor 120. For example, the image sensor 120 may be exposed during the time periods of the continuous waves 315A-C, 325A-C, and 335A-C, and not exposed outside these time periods, although the illuminator assembly 110 may project modulated light outside these time periods. In another embodiment, the period 310 may be a readout period of the image sensor 120. For example, the image sensor 120 may read out the charge accumulated in the pixel array 195 during the time periods of the continuous waves 315A-C, 325A-C, and 335A-C, and not read out the charge exceeding these time periods, although the illuminator assembly 110 may project modulated light outside these time periods, or the pixel array 195 may be exposed outside these time periods. Although Figure 3 The cycle 310 in includes three continuous waves for each of the three frequencies, but cycles in other embodiments may include a different number of frequencies or a different number of continuous waves for each frequency.
[0058] exist Figure 3, continuous waves 315A-C have different phase offsets. For example, continuous wave 315A has a phase offset of 0°, continuous wave 315B has a phase offset of 120°, and continuous wave 315C has a phase offset of 240°. Similarly, continuous waves 325A-C may have different phase offsets from each other: continuous wave 325A may have a phase offset of 0°, continuous wave 325B may have a phase offset of 120°, and continuous wave 325C may have a phase offset of 240°; and continuous waves 335A-C may start with different phase offsets from each other: continuous wave 335A may have a phase offset of 0°, continuous wave 335B may have a phase offset of 120°, and continuous wave 335C may have a phase offset of 240°. The phases of continuous waves 315A, 325A, and 335A may be between 0° and 120°, respectively, the phases of continuous waves 315B, 325B, and 335B may be between 120° and 240°, and the phases of continuous waves 315C, 325C, and 335C may be between 240° and 360°, respectively. In some embodiments, the duration of each continuous wave is about 100 μs. The time interval between two adjacent continuous waves may be in the range of 1 to 2 ms.
[0059] In other embodiments, cycle 310 may not have multiple continuous waves for each frequency. Instead, cycle 310 has a single continuous wave for a single frequency. For example, cycle 310 may include a first continuous wave for frequency 317, a second continuous wave for frequency 327, and a third continuous wave for frequency 337. The first, second, and third continuous waves may all start from 0°. There may be a time gap between two adjacent continuous waves of the first, second, and third continuous waves. Cycle 310 may generate image data for controller 130 to generate frames. Cycle 310 may be repeated so that controller 130 generates more frames. Controller 130 may perform phase unwrapping to determine depth information.
[0060] Image Sensor Examples
[0061] Figure 4 An example image sensor 400 according to some embodiments of the present disclosure is shown. The image sensor 400 is configured to read out two signals from two readout terminals of a single pixel, respectively. Since the pixel provides two signals, the pixel is also called a differential pixel. The image sensor 400 is Figure 1 An embodiment of the image sensor 120 in FIG. Figure 4As shown, image sensor 400 includes pixel array 401 including a plurality of differential pixels 405 (individually referred to as "differential pixels 405"), amplifier array 402 including a plurality of amplifiers 407 (individually referred to as "amplifiers 407"), and ADC array 403 including a plurality of ADCs 409 (individually referred to as "ADC 409"). In alternative configurations, different and / or additional components may be included in image sensor 400. For example, image sensor 400 may include a buffer array including a plurality of buffers in place of or in addition to amplifier array 402. Furthermore, the functionality attributed to one component of image sensor 400 may be implemented by a different component included in image sensor 400 or by a different system than that shown.
[0062] The pixel array 401 includes pixels arranged in eight columns 410, 420, 430, 440, 450, 460, 470, and 480. Each column includes six differential pixels 405 (individually referred to as "differential pixels 405"). In other embodiments, the pixel array 401 may include a different number of columns, or a column may include a different number of differential pixels 405. Each differential pixel 405 can receive photons and convert the photons into electric charge. The charge can be stored in one or more storage areas in the differential pixel 405. Each differential pixel 405 can output two signals through two readout terminals. Figure 4 In the example, the two ends are represented by "A" and "B". Figure 4 In the embodiment of FIG. 4 , differential pixels 405 in the same column share the same readout terminal, and each column is associated with two readout terminals.
[0063] Each column of the pixel array 401 corresponds to two adjacent amplifiers 407 in the amplifier array 402. The two adjacent amplifiers 407 are respectively coupled to the two readout terminals of the column. Each amplifier 407 can amplify the analog signal from the corresponding readout terminal. Then, the amplified analog signal is provided to the ADC 409 in the ADC array 403. The ADC 409 is coupled to the amplifier 407 and can convert the amplified analog signal into a digital signal. Figure 4 As shown, each column of the pixel array 401 corresponds to two adjacent ADCs 409 in the ADC array 403 , and each of the two ADCs 409 corresponds to a different one of the two readout ends of the column. The image sensor 400 can read out two digital signals from each differential pixel 405 .
[0064] As an initial example, Figure 4, each pixel column has two single-ended amplifiers 407 and two ADCs 409. In an alternative, for example, if the signal read out from the image sensor 400 is large enough, the amplifiers can be omitted entirely. In another alternative, each pixel column can be provided with a single-ended ADC (optionally with one or two single-ended amplifiers), which can be multiplexed to convert signals A and B. Providing a single-ended ADC per column can reduce the size and cost of the depth estimation system, but can also reduce the readout speed.
[0065] Figure 5 An example differential pixel 500 is shown according to some embodiments of the present disclosure. The differential pixel 500 may be Figure 4 4. The differential pixel 500 includes a photodiode 510, switches 515, 520A, 525A, 520B, and 525B, clocks 530A and 530B, and capacitors 540A and 540B. In alternative configurations, different and / or additional components may be included in the differential pixel 500. Furthermore, the functionality attributed to one component of the image sensor 400 may be implemented by different components included in the differential pixel 500 or by a system different from that shown.
[0066] Photodiode 510 receives photons and converts the photons into electric charge. The photons may include photons of light projected by illuminator assembly 110, photons of ambient light, or a combination of the two. When switches 515 and 520A are closed, the charge may be stored in capacitor 540A. Switch 520A is controlled by clock 530A. When switch 525A is closed, the charge stored in capacitor 540A may be read out through readout terminal A. Similarly, when switches 515 and 520B are closed, the charge may be stored in capacitor 540B. Switch 520B is controlled by clock 530B. When switch 525B is closed, the charge stored in capacitor 540B may be read out through readout terminal B.
[0067] Readout terminal A is coupled to amplifier 510A and ADC 560A, through which the charge from capacitor 540A can be amplified and converted into a digital signal. Readout terminal B is coupled to amplifier 510B and ADC 560B, through which the charge from capacitor 540B can be amplified and converted into a digital signal.
[0068] The digital signal can be received and processed by other devices or components (e.g., controller 130) to generate an image, such as a brightness image. In some embodiments, the controller 130 may include a memory for storing digital signals. The digital signals from the two readout terminals can be read out independently and digitally calculated. The controller 130 can determine the sum of the two digital signals, i.e., A+B, and generate a common mode (CM) image. The CM image combines image data from ambient light reflected from the object. Alternatively or additionally, the controller 130 can determine the difference between the two digital signals (e.g., AB) and cancel the ambient light from the signal. The image generated based on the difference between the two digital signals does not contain image data from ambient light.
[0069] Controller Example
[0070] Figure 6 6 is a block diagram illustrating controller 130 according to some embodiments of the present disclosure. Controller 130 includes database 610, illuminator module 620, imaging module 630, depth module 640, brightness module 650, and depth enhancement module 660. These modules are software modules implemented on one or more processors, dedicated hardware units, or some combination thereof. Some embodiments of controller 130 have different Figure 6 Similarly, the components described together Figure 6 The functionality of the components described can be combined with Figure 6 The different ways described are distributed among other components. For example, some or all of the functions described as being performed by the controller 130 may be performed by a device incorporating a depth estimation system, such as Figure 8 System 800, Fig. 9 The mobile device 900, Fig.10 Entertainment system 1000, Fig.11 The robot 1100 or other device in.
[0071] The database 610 stores data generated and / or used by the controller 130. The database 610 is a memory, such as a ROM, DRAM, SRAM, or some combination thereof. The database 610 may be part of a larger digital memory of a depth estimation system, such as the depth estimation system 100, or may be part of a device that includes the depth estimation system. In some embodiments, the database 610 stores image data from the image sensor 120, a depth image generated by the depth module 640, a brightness image generated by the brightness module 650, an enhanced depth image generated by the depth enhancement module 660, parameters for an energy model generated by the depth enhancement module 660, data for optimizing the energy model, and the like. In some embodiments, the database 610 may store calibration data and / or other data from other components, such as depth instructions. The depth instructions include illuminator instructions generated by the illuminator module 620 and imaging instructions generated by the imaging module 630.
[0072] The illuminator module 620 controls the illuminator assembly 110 via illuminator instructions. The illuminator instructions include one or more illumination parameters that control how the illuminator assembly 110 projects light. The illumination parameters may describe, for example, a waveform, a wavelength, an amplitude, a frequency, a phase shift, a start time of each continuous wave, an end time of each continuous wave, a duration of each continuous wave, some other parameter that controls how the illuminator assembly 110 projects light, or some combination thereof. The illuminator module 620 may retrieve the illuminator instructions from the database 350. Alternatively, the illuminator module 620 generates the illuminator instructions. For example, the illuminator module 620 determines one or more illumination parameters. In an embodiment where the illuminator assembly 110 includes multiple modulated light projectors, the illuminator module 620 may determine separate illumination parameters for different light projectors.
[0073] The imaging module 630 controls the image sensor 120 via the imaging instructions. The imaging module 630 can retrieve the imaging instructions from the database 610. Alternatively, the imaging module 630 generates the imaging instructions based in part on the illuminator instructions generated by the illuminator module 620. The imaging module 630 determines the exposure parameters (such as the start time, the end time, or the duration of the exposure interval, etc.) of the image sensor 120 based on one or more illumination parameters (such as the duration of the continuous wave, etc.) specified in the illuminator instructions. For example, the imaging module 630 determines that the duration of the exposure is equal to the duration of the continuous wave. Sometimes the imaging module 630 determines that the duration of the exposure is longer than the duration of the continuous wave to avoid the inability to collect the entire continuous wave due to the delay of the incident light. The duration of the exposure can be 20% longer than the duration of the continuous wave. In some embodiments, the imaging module 630 also determines the number of exposure intervals for each continuously modulated light wave projected by the lighting assembly 110.
[0074] The imaging instructions may include readout instructions for controlling the readout of the image sensor 120. The imaging module 630 may determine readout parameters (e.g., a start time, an end time, or a duration of a readout interval, etc.) of the image sensor 120. For example, the imaging module 630 determines the start time of each of the one or more readout intervals, for example, based on one or more illumination parameters (e.g., phase, waveform, start time, or other parameters of the continuous wave). The imaging module 630 may also determine the duration of each readout interval, the number of readout intervals of the continuous wave, the time interval between adjacent readout intervals, the number of readout cycles, other readout parameters, or some combination thereof.
[0075] The depth module 640 is configured to generate a depth image indicating the distance to the imaged object 140 based on, for example, a digital signal indicating the charge accumulated on the pixel array 195. The depth module 640 may analyze the digital signal to determine the phase shift exhibited by the light (e.g., the phase shift described above in connection with FIG. 2). ) to determine the ToF of the light (eg, the ToFδ described above in conjunction with FIG. 2 ), and further determine the depth value of the object 140 (eg, the distance described above in conjunction with FIG. 2 ).
[0076] In embodiments where the illumination assembly 110 projects multiple continuous waves with different phase offsets, the depth module 640 can generate a depth image by phase unwrapping. Figure 3 Taking the period 310 in as an example, the depth module 640 can determine the wrapping distance, each wrapping distance corresponding to the corresponding phase. The depth module 640 can also estimate the unwrapped depth of each wrapping distance. The depth module 640 also determines the Voronoi vector corresponding to the unfolded depth and generates a lattice of Voronoi cells. Each unfolded depth corresponds to a Voronoi cell of the lattice. In an alternative embodiment, the depth module 640 is configured to determine the depth information using a charge ratio between storage areas associated with each photodiode of the image sensor 120.
[0077] The brightness module 650 generates a brightness image, such as an active brightness image. In some embodiments, for the depth image generated by the depth module 640, the brightness module 650 generates a corresponding brightness image. The brightness module 650 can generate a brightness image based on a request for a brightness image from the depth enhancement module 660. The depth module 640 can generate a depth image based on a phase shift between the first captured light and the projected light, and the brightness module 650 can generate a corresponding brightness image based on the intensity or amplitude of the second captured light. In some embodiments, the first captured light and the second captured light are the same light. In other embodiments, the second captured light is different from the first captured light. For example, the first captured light may be IR, and the second captured light may be visible light.
[0078] In an embodiment where the depth module 640 generates a depth image based on the accumulated charge in the pixel array 195, the brightness module 650 can generate a corresponding brightness image based on the accumulated charge in all or some of the pixels 197 in the group. The corresponding brightness image includes a plurality of brightness pixels. Each brightness pixel can correspond to a depth pixel in the depth image. For example, the values of the depth pixel and the corresponding brightness pixel can be determined based on the accumulated charge in the same pixel in the pixel array 195. The accumulated charge in the pixel can be converted from photons of modulated light reflected by the trajectory of the object 140. The value of the depth pixel can be determined by the depth module 640 based on the phase shift in the waveform of the modulated light. The value of the corresponding brightness pixel can be determined based on the accumulated charge in the pixel or in different pixels.
[0079] The depth enhancement module 660 enhances the depth estimation performed by the depth module 640 based on the luminance image generated by the luminance module 650. The depth enhancement module 660 may retrieve the depth image generated by the depth module 640 and the corresponding luminance image generated by the luminance module 650. In some embodiments, the depth enhancement module 660 may instruct the luminance module 650 to generate the corresponding luminance image.
[0080] In some embodiments, the depth enhancement module 660 may use the brightness image to enhance one or more boundaries in the depth image. The brightness image may show one or more boundaries of the object 140 more clearly than the depth image. For example, one or more depth pixels representing at least a portion of the boundary of the object 140 may be invalid. The boundary may be the edge of the object 140. Alternatively, the boundary is the boundary between two regions of the object 140 with different reflectivity characteristics, such as the boundary between a fluorescent strip with relatively high reflectivity and a low reflectivity surface. The invalid depth pixel is the result of an invalid depth estimate, and the difference between the value of the invalid depth pixel and the ground truth depth value may exceed a threshold, such as 5%, 10%, or other percentage of the ground truth depth. The depth enhancement module 660 can generate an enhanced depth image using more accurate information of the boundary of the object 140 in the brightness image, which includes a better depth estimate than the original depth image.
[0081] The depth enhancement module 660 may fuse the depth image and the corresponding brightness image, for example, by a filter. The filter may be a box filter, a Gaussian filter, a joint bilateral filter, a guide filter, etc. The depth enhancement module 660 may, for example, identify one or more invalid depth pixels in the depth image by using a filter. In one embodiment, the depth enhancement module 660 identifies one or more depth pixels on the boundary of the object 140, for example, by using a filter, and regards the one or more depth pixels on the boundary as invalid pixels. The depth pixels on the boundary may be depth pixels representing boundary tracks. In another embodiment, the depth enhancement module 660 may compare the depth value of the pixel in the depth image with the thickness value of the pixel determined by the guide filter, and determine whether the pixel is invalid based on the comparison result. For example, in response to determining that the difference between the depth value in the depth image and the depth value from the guide filter exceeds a threshold value (e.g., a threshold absolute value or a threshold percentage), the depth enhancement module 660 may determine that the pixel is invalid. In another embodiment, the depth enhancement module 660 may determine the noise level of the depth pixel, and determine that the depth pixel is invalid in response to the noise level exceeding the threshold.
[0082] The depth enhancement module 660 may determine a new depth value for the invalid depth pixel based on the brightness value of the corresponding brightness pixel in the brightness image. In one example, the new depth value is equal to or similar to the brightness value. For a depth pixel that is not on any boundary, the depth enhancement module 660 may determine that the depth pixel is on a plane. The depth enhancement module 660 may not change the depth value of the depth pixel, or the depth enhancement module 660 may determine the average depth value of the depth pixels in a box centered on the depth pixel and use the average as the new depth value of the depth pixel.
[0083] The depth enhancement module 660 can generate an enhanced depth image with new depth values. For example, the depth enhancement module 660 can replace the value of the depth pixel with the new depth value. The enhanced depth image represents a better depth estimate, especially for one or more boundaries of the object 140.
[0084] Depth Estimation Enhancement Example
[0085] Fig. 7A An example depth image 710 according to some embodiments of the present disclosure is shown. The depth image 710 may be obtained by combining the above Figure 4 The depth module 640 generates the described. Figure 7B The present invention shows some embodiments of the present invention. Fig. 7A Example brightness image 720 corresponding to the depth image 710 in FIG. The brightness image 720 can be obtained by combining the above Figure 4 The brightness module 650 described above generates. Figure 7C According to some embodiments of the present disclosure, Fig. 7A The depth image 710 and Figure 7B Example depth enhanced image 730 generated from brightness image 720 in FIG.
[0086] The three images 710, 720 and 730 capture an object, an example of which is Figure 1 The object 140 has a flat surface 740 and an edge 750. Figure 7A-5 In C, part of the edge is enclosed in a dotted ellipse. Fig. 7A and 7B As shown, the brightness image 720 shows clearer edges than the depth image 710. Therefore, the brightness image 720 can be used to enhance the depth estimation of the object, especially the edge 750 of the object. The brightness image 720 is fused with the depth image 710 through the energy model to generate the depth enhanced image 730. The depth enhanced image 730 shows a better depth estimation than the depth image 710. Fig. 7A and 7C As shown, the edges in the depth enhanced image 730 are clearer than the depth image 710. The depth enhanced image 730 can be generated by using an energy model, where the depth image 710 can be used as an input image and the brightness image 720 can be used as a guide image. The depth enhanced image 730 can be generated by the above-mentioned depth enhancement process in combination with the depth enhancement module 660. For example, the pixels representing the edge 750 are identified, and the new depth value of the pixel is determined based on the pixel value in the brightness image 720. The new depth value of the pixel is used to generate the depth enhanced image 730.
[0087] Fig. 7A and 7C It is also shown that the plane 740 is smoother in the depth enhanced image 730. Figure 7A-5 In some embodiments of C, pixels representing plane 740 are also identified. For each of these pixels, a box is defined. The box can be centered on the pixel and include other pixels surrounding the pixel. The new depth value of the pixel is determined based on the depth values of the other pixels in the box. For example, the new depth value of the pixel can be the average of the depth values of the other pixels. The new depth value of the pixel is used to generate a depth-enhanced image 730. Through such a depth enhancement process, the depth-enhanced image 730 represents a more accurate depth estimate of the object, such as a more accurate depth estimate of the plane 740, the edge 750, or both.
[0088] Example Applications Incorporating Depth Estimation Systems
[0089] Figure 8 An example system 800 including a depth estimation system according to some embodiments of the present disclosure is shown. One embodiment of the depth estimation system is Figure 1The depth estimation system 100 in FIG. 800 includes an imaging device 810, a processor 820, a memory 830, an input device 840, an output device 850, and a battery / power supply circuit 860. In other embodiments, the system 800 may include fewer, more, or different components. For example, the system 800 may include multiple processors, memories, display devices, input devices, or output devices.
[0090] The imaging device 810 captures a depth image and a brightness image. The imaging device 810 may include an illuminator assembly, such as the illuminator assembly 110, for projecting light into the environment surrounding the system 800. The imaging device 810 may project modulated light, such as pulsed modulated light or a continuous wave of modulated light. The imaging device 810 also includes an image sensor, such as the image sensor 120, which captures light reflected by one or more objects in the environment and generates image data of the one or more objects.
[0091] The processor 820 may process electronic data from registers and / or memories to convert the electronic data into other electronic data that may be stored in registers and / or memories. The processor 820 may perform some or all functions of some or all components of the controller 130, such as depth estimation, depth estimation enhancement using brightness signa, etc. The processor 820 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), CPUs, GPUs, cryptographic processors (special processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices.
[0092] In some embodiments, the processor 820 may also use the depth information (e.g., an enhanced depth image) to generate content (e.g., images, audio, etc.) for display to a user of the system through one or more display devices (e.g., output device 850). The content may be used as VR, AR, or MR content. The processor 820 may also generate instructions for other components of the system 800 or another system based on the enhanced depth image. For example, the processor 820 may determine navigation instructions for a movable device, such as a robot, vehicle, or other type of movable device. The navigation instructions may include navigation parameters (e.g., navigation route, speed, direction, etc.).
[0093] The memory 830 may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile storage (e.g., read-only memory (ROM)), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 830 may include a memory that shares a die with the processor 820. The memory 830 may store processor-executable instructions for controlling the operation of the depth estimation system 100 and / or data captured by the depth estimation system 100. In some embodiments, the memory 830 includes one or more non-transitory computer-readable media that store instructions executable to perform a depth estimation enhancement process, such as the following in conjunction with Fig.10 The method 1200 described above, or in combination with Figure 1 and Figure 4 The operations described are performed by the controller 130 (or some components of the controller 130 ). Instructions stored in one or more non-transitory computer-readable media may be executed by the processor 820 .
[0094] Input device 840 may include an audio input device. Audio input device 1318 may include any device that generates a signal representing sound, such as a microphone, a microphone array, a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output), etc. Input device 840 may also include one or more other types of input devices, such as an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (such as a mouse), a stylus, a touchpad, a barcode reader, a quick response (QR) reader, a sensor, a radio frequency identification (RFID) reader, etc.
[0095] Output device 850 may include one or more display devices, such as one or more visual indicators. Example visual indicators include a head-up display, a computer display, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display, etc. Output device 850 may also include an audio output device. An audio output device may include any device that produces a sound indicator, such as a speaker, a headset, or an earplug, etc. Output device 850 may also include one or more other output devices, such as an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, etc.
[0096] Battery / power circuitry 860 may include one or more energy storage devices (eg, batteries or capacitors) and / or circuitry for coupling components of system 800 to an energy source separate from system 800 (eg, AC line power).
[0097] Fig. 9 A mobile device 900 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. An example of a depth estimation system is Figure 1 The depth estimation system 100 in FIG. 9 is a mobile device 900 that can be a mobile phone. Fig. 9 As shown, mobile device 900 includes imaging component 902. The imaging device may include Figure 1 The illuminator assembly 110 and the image sensor 120 in the mobile device 900. The imaging assembly 902 can illuminate the environment around the mobile device 900 with modulated light (e.g., modulated IR) and capture images of one or more objects in the environment. Fig. 9 Not shown, the mobile device 900 may also include one or more processors and one or more memories that may perform some or all of the functions of the controller 130. Through these components, the mobile device 900 may determine depth information of one or more objects in the environment surrounding the mobile device 900. The mobile device 900, another device, or a user of the mobile device may use the depth information for various purposes, such as VR, AR, or MR applications, navigation applications, and the like. For example, the mobile device 900 may generate and present images (two-dimensional or three-dimensional images) based on the depth information of the environment, and the images may represent virtual objects that do not exist in the real-world environment. These images may enhance the real-world objects in the environment so that the user of the mobile device 900 may have an interactive experience with the real-world environment in which real-world objects residing in the real world are enhanced by computer-generated virtual objects.
[0098] Fig.10 An entertainment system 1000 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. Fig.10 In the example of FIG. 1 , a user 1008 can interact with the entertainment system via a controller 1010, such as to play a video game. The entertainment system 1000 includes a console 1002 and a display 1004. The console 1002 can be a video game console configured to generate images of a video game on the display 1004. In other embodiments, the entertainment system 1000 can include more, fewer, or different components.
[0099] Console 1002 includes imaging assembly 1006. Imaging assembly 1006 may include Figure 1 The illuminator assembly 110 and the image sensor 120 in the embodiment of the present invention can be configured to illuminate the environment around the entertainment system 1000 with modulated light (e.g., modulated IR) and capture the modulated light reflected by one or more objects in the environment to generate an image of the object, such as a user 1008, a controller 1010, or other object. Fig.101002, the console 1002 may also include one or more processors and one or more memories that may perform some or all of the functions of the controller 130. The console 1002 may determine depth information of one or more objects in the environment. The depth information may be used to present images to the user on the display 1004, or to control certain other aspects of the entertainment system 1000. For example, the user 1008 may control the entertainment system 1000 with gestures, and the gestures may be determined at least in part by the depth information. The console 1002 may generate or update display content (e.g., images, audio, etc.) based on the depth information, and may also instruct the display 1004 to present the display content to the user 1008.
[0100] Fig.11 An example robot 1102 incorporating a depth estimation system according to some embodiments of the present disclosure is shown. The robot 1102 includes an imaging component 1104, which may include Figure 1 The illuminator assembly 110 and the image sensor 120 in the embodiment of the present invention. The imaging assembly 1104 can illuminate the environment around the robot 1102 with modulated light (e.g., modulated IR) and capture images of one or more objects in the environment. Fig.11 102. Not shown, the robot 1102 may also include a computing device that can perform some or all of the functions of the controller 130. The computing device may include one or more processors and one or more memories. The computing device may determine depth information of one or more objects in the environment. The robot 1102 may be mobile, and the computing device may use the depth information to assist in navigation and / or motor control of the robot 1102. For example, the computing device may include navigation instructions based on the depth information. The navigation instructions may include a navigation route for the robot 1102. The robot 1102 may navigate in the environment according to the navigation instructions.
[0101] Examples of uses of the techniques described herein are possible in addition to those shown in FIGS. 7-9. For example, the depth estimation system described herein can be used in other applications such as self-driving cars, security cameras, etc.
[0102] Example method to enhance depth estimation using energy model
[0103] Fig.12 1 is a flow chart illustrating a method 1200 for enhancing depth estimation of a luminance image using an energy model according to some embodiments of the present disclosure. The method 1200 may be performed by the controller 130. Although reference is made to Fig.12 The flowchart shown in describes method 1200, but many other methods of using energy models to enhance depth estimation of luminance images may also be used. For example, one may change Fig.12As another example, some steps may be changed, eliminated, or combined.
[0104] The controller 130 generates 1210 a brightness image based on image data from an image sensor. The image sensor includes a plurality of pixels that may be arranged in columns. The brightness image includes a plurality of brightness pixels. Each of the plurality of brightness pixels corresponds to a corresponding pixel in a plurality of pixels and is generated based on signals from two ADCs associated with the corresponding pixel. Each of the two ADCs converts a different analog signal from the corresponding pixel into a different digital signal in the image data. The two ADCs are associated with one or more other pixels in the plurality of pixels. The one or more other pixels and the individual pixels may be arranged in the same column in the image sensor.
[0105] The controller 130 also generates 1220 a depth image based on the image data. The depth image includes a plurality of depth pixels. Each depth pixel is associated with a depth value. The brightness image and the depth image may capture the same object. The target depth pixel may represent the same object trajectory as the brightness pixel.
[0106] The controller 130 determines 1230 a new depth value of a target depth pixel among the plurality of depth pixels based on the luma pixel among the plurality of luma pixels. The target depth pixel and the luma pixel may correspond to the same pixel among the plurality of pixels. The controller 130 updates 1240 the depth image by assigning the new depth value to the target depth pixel.
[0107] In some embodiments, the controller 130 instructs the illuminator assembly to project modulated light into a local area including an object. The modulated light includes a continuous infrared wave, and the continuous wave has different frequencies. The controller 130 also instructs the image sensor to capture reflected light from at least a portion of the object. The controller 130 generates a depth image based on a phase shift between the reflected light and the modulated light projected into the local area. In some embodiments, the controller 130 may generate a brightness image based on the brightness of the reflected light. In other embodiments, the reflected light is a first reflected light, and the controller 130 instructs the image sensor to capture a second reflected light from at least a portion of the object.
[0108] Changes and implementation
[0109] Although the above reference Figure 1-10 The illustrated exemplary implementations describe embodiments of the present disclosure, but those skilled in the art will appreciate that the various teachings described above are applicable to a variety of other implementations.
[0110] In some cases, the features discussed herein may be applicable to automotive systems, safety-critical industrial applications, medical systems, scientific instrumentation, wireless and wired communications, radio, radar, industrial process control, audio and video equipment, current sensing, instrumentation (which may be high precision), and other digital processing-based systems.
[0111] In the discussion of the above embodiments, components of the system, such as filters, converters, mixers, amplifiers, digital logic circuits and / or other components, can be easily replaced, substituted or otherwise modified to suit specific circuit requirements. In addition, it should be noted that the use of complementary electronic devices, hardware, software, etc. in various communication systems provides equally viable options for implementing the teachings of the present disclosure related to fractional frequency dividers.
[0112] As described herein, portions of various systems for implementing a balanced fractional divider for duty cycle regulation may include electronic circuits that perform the functions described herein. In some cases, one or more portions of the system may be provided by a processor specifically configured to perform the functions described herein. For example, the processor may include one or more dedicated components, or may include programmable logic gates configured to perform the functions described herein. The circuit may operate in an analog domain, a digital domain, or a mixed signal domain. In some cases, the processor may be configured to perform the functions described herein by executing one or more instructions stored on a non-transient computer readable storage medium.
[0113] In an example embodiment, any number of circuits of this figure can be implemented on a board of an associated electronic device. The board can be a general circuit board that can accommodate various components of the electronic system inside the electronic device and can also provide connectors for other peripheral devices. More specifically, the board can provide an electrical connection through which other components of the system can communicate electrically. Any suitable processor (including DSP, microprocessor, supporting chipset, etc.), computer-readable non-transient storage element, etc. can be appropriately coupled to the board according to specific configuration requirements, processing requirements, computer design, etc. Other components, such as external memory, additional sensors, audio / video display controllers, and peripherals, can be connected to the board as plug-in cards via cables or integrated into the board itself. In various embodiments, the functions described herein can be implemented in simulated form as software or firmware running in one or more configurable (e.g., programmable) elements arranged in a structure that supports these functions. The software or firmware providing the simulation can be provided on a non-transient computer-readable storage medium that includes instructions that allow the processor to perform these functions.
[0114] In another example embodiment, the circuit of the present figure can be implemented as a stand-alone module (e.g., a device having related components and circuits configured to perform a specific application or function) or as a plug-in module inserted into dedicated hardware of an electronic device. Note that certain embodiments of the present disclosure can be easily included in part or in whole in a system-on-chip (SOC) package. SOC represents an integrated circuit that integrates the components of a computer or other electronic system into a single chip. It may contain digital, analog, mixed-signal, and radio frequency (RF) functions: all of which can be provided on a single chip substrate. Other embodiments may include a multi-chip module (MCM), in which multiple separate ICs are located within a single electronic package and are configured to interact closely with each other through the electronic package.
[0115] It must also be noted that all specifications, dimensions, and relationships outlined herein are for example and teaching purposes only. Such information may be significantly changed without departing from the spirit of the present disclosure or the scope of the appended claims. It should be understood that the system may be integrated in any suitable manner. Along similar design alternatives, any of the circuits, components, modules, and elements shown in this figure may be combined in various possible configurations, all of which are clearly within the broad scope of this specification. In the foregoing description, example embodiments have been described with reference to specific processors and / or component arrangements. Various modifications and changes may be made to these embodiments without departing from the scope of the appended claims. Therefore, the specification and drawings should be regarded as illustrative, not restrictive.
[0116] Note that through many of the examples provided herein, interactions may be described in terms of two, three, four or more electrical components. However, this is done for clarity and example only. It should be understood that the system may be integrated in any suitable manner. Along similar design alternatives, any components, modules, and elements shown in the figures may be combined in a variety of possible configurations, all of which are clearly within the broad scope of this specification. In some cases, it may be easier to describe one or more functions of a given set of flows by referencing only a limited number of electrical components. It should be understood that the drawings and the circuits they teach are easily expandable and can accommodate a large number of components and more complex / complex arrangements and configurations. Therefore, the examples provided should not limit the scope of the circuit or inhibit its broad teachings that may be applied to countless other architectures.
[0117] Note that in this specification, references to various features (e.g., elements, structures, modules, components, steps, operations, characteristics, etc.) included in "one embodiment," "example embodiment," "specific embodiment," "another embodiment," "some embodiments," "various embodiments," "other embodiments," or "alternative embodiments," etc., are intended to indicate that any such features are included in one or more embodiments of the present disclosure, but may or may not be combined in the same embodiment. In addition, as used herein, including in the claims, "or" used in a list of items (e.g., a list of items beginning with phrases such as "at least one" or "one or more") represents an inclusive list, for example, [at least one of A, B, or C] represents A or B or C or AB or AC or BC or ABC (i.e., A and B and C).
[0118] Various aspects of the illustrative embodiments are described using terms commonly used by those skilled in the art to convey the substance of their work to others skilled in the art. For example, the term "connected" refers to a direct electrical connection between the things being connected, without any intermediate devices / components, while the term "coupled" refers to a direct electrical connection between the connected objects, or an indirect connection through one or more passive or active intermediate devices / components. In another example, the term "circuit" refers to one or more passive and / or active components that are arranged to cooperate with each other to provide the desired functionality. In addition, as used herein, based on the context of specific values described herein or known in the art, the terms "substantially", "approximately", "about", etc. may be used to generally refer to within + / -20% of a target value, such as within + / -10% of a target value.
[0119] Numerous other variations, substitutions, changes, alterations, and modifications may be determined by those skilled in the art, and the present disclosure is intended to include all such variations, substitutions, and modifications that fall within the scope of the examples and the appended claims. Note that all optional features of the above-described devices may also be implemented with respect to the methods or processes described herein, and that the details in the examples may be used anywhere in one or more embodiments.
[0120] Explanation of terms
[0121] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms. Unless the context clearly requires otherwise, throughout the specification and claims:
[0122] The words “include”, “including”, etc. shall be construed as being inclusive rather than exclusive or exhaustive; that is, in the sense of “including but not limited to”.
[0123] “Connected,” “coupled,” or any variation thereof, refers to any direct or indirect connection or coupling between two or more elements; the coupling or connection between the elements may be physical, logical, or a combination thereof.
[0124] When used to describe this Specification, the words “herein,” “above,” “hereunder,” and words of similar import shall refer to this Specification as a whole and not to any particular portion of this Specification.
[0125] “Or” refers to a list of two or more items and covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.
[0126] The singular forms "a", "an" and "the" are intended to include any appropriate plural forms.
[0127] Words indicating directions, such as "vertical," "lateral," "horizontal," "upward," "downward," "forward," "backward," "inward," "outward," "perpendicular," "lateral," "left," "right," "front," "backward," "top," "bottom," "below," "above," "lower," etc., used in this specification and any appended claims, if any, depend on the specific orientation of the device being described and illustrated. The subject matter described herein can assume a variety of alternative orientations. Accordingly, these directional terms are not strictly defined and should not be narrowly interpreted.
[0128] Unless expressly indicated to the contrary, the indefinite articles "a," "an," and "a" used in the specification and claims should be understood to mean "at least one."
[0129] The phrase "and / or" used herein in the specification and claims should be understood to mean "one or both" of the elements so coupled, i.e., elements that are present in combination in some cases and separate in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so coupled.
[0130] In addition to the elements expressly identified by the "and / or" clause, other elements may optionally be present, whether or not related to those elements expressly identified. Thus, as a non-limiting example, in one embodiment, when used in conjunction with open language such as "comprising," a reference to "A and / or B" may include only A (optionally including elements other than B); in another embodiment only B (optionally including elements other than A); in another embodiment both A and B (optionally including other elements); etc.
[0131] As used herein in the specification and claims, the phrase "at least one" refers to a list of one or more elements and should be understood to refer to at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of each element specifically listed in the list of elements, nor exclude any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than the elements specifically identified in the list of elements to which the phrase "at least one" refers, whether or not these elements are related to those elements specifically identified.
[0132] Thus, as a non-limiting example, in one embodiment, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently "at least one of A and / or B") may refer to at least one, optionally including multiple A's, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in another embodiment, to at least one (optionally including more than one), A, and at least one, optionally including more than one B (and optionally including other elements); etc.
[0133] As used herein, the term "between" is intended to be inclusive unless otherwise stated. For example, "between A and B" includes A and B unless otherwise stated.
[0134] In addition, the words and terms used herein are for description only and should not be considered as limiting. The words "include", "comprising", or "having", "containing", "involving" and their variations used herein are intended to include the items listed below and their equivalents as well as additional items.
[0135] In the claims and the above description, all transitional phrases, such as "include", "comprising", "carrying", "having", "containing", "involving", "holding", "consisting of", etc., shall be understood as open-ended, i.e., including but not limited to. Only the transitional phrases "consisting of" and "mainly composed of" shall be closed or semi-closed transitional phrases, respectively.
[0136] Numerous variations, substitutions, alterations, and modifications may be identified by those skilled in the art, and this disclosure is intended to embrace all such variations, substitutions, alterations, and modifications that fall within the scope of the appended claims.
[0137] To assist the United States Patent and Trademark Office (USPTO) and any reader of any patent that issues from this application in interpreting the claims appended hereto, applicants wish to note that applicants: (a) do not intend to invoke 35 U.S.C. § 112(f) as it existed on the date of filing this application, unless the words “for” or “for” are specifically used in a particular claim; and (b) do not intend that any statement in this disclosure should limit the present disclosure in any manner not reflected in the appended claims.
[0138] Therefore, the present invention should not be considered as limited to the specific embodiments described above. After reading this disclosure, those skilled in the art to which the present invention is directed will easily understand various modifications, equivalent processes, and numerous structures to which the present invention may be applicable.
[0139] Select Example
[0140] Example 1 provides a method, comprising: generating a luminance image comprising a plurality of luminance pixels based on image data from an image sensor comprising a plurality of pixels, wherein each of the plurality of luminance images corresponds to a corresponding pixel among the plurality of pixels and is generated based on signals from two ADCs associated with the corresponding pixel; generating a depth image comprising a plurality of depth pixels based on the image data, each depth pixel being associated with a depth value; determining a new depth value of a target depth pixel among the plurality of depth pixels based on the luminance pixels among the plurality of luminance pixels; and updating the depth image by assigning the new depth value to the target depth pixel.
[0141] Example 2 provides the method of Example 1, wherein the target depth pixel and the luma pixel correspond to a same pixel among the plurality of pixels.
[0142] Example 3 provides the method of Example 1 or 2, wherein each of the two ADCs converts a different analog signal from a corresponding pixel into a different digital signal in the image data.
[0143] Example 4 provides the method of any of the preceding examples, wherein the two ADCs are associated with one or more other pixels of the plurality of pixels.
[0144] Example 5 provides the method of Example 4, wherein the one or more other pixels and the corresponding pixel are arranged in a same column in the image sensor.
[0145] Example 6 provides the method of any of the preceding examples, wherein the luminance image and the depth image capture the same object, and the target depth pixel represents the same object trajectory as the luminance pixel.
[0146] Example 7 provides the method of any of the preceding examples, further comprising: instructing the illuminator assembly to project modulated light into a local area including an object; instructing the image sensor to capture reflected light from at least a portion of the object; and generating the depth image based on a phase shift between the reflected light and the modulated light projected into the local area.
[0147] Example 8 provides the method of Example 7, wherein the modulated light comprises continuous infrared waves, and the continuous waves have different frequencies.
[0148] Example 9 provides the method of Example 7 or 8, further comprising: generating the brightness image based on the brightness of the reflected light.
[0149] Example 10 provides the method of any one of Examples 7-9, wherein the reflected light is a first reflected light, and the method further includes: instructing the image sensor to capture a second reflected light from at least a portion of the object; and generating the brightness image based on the brightness of the second reflected light, wherein the second reflected light has a different wavelength from the first reflected light.
[0150] Example 11 provides a system comprising: an illuminator assembly configured to project modulated light into a local area including an object; an image sensor comprising a plurality of pixels, the image sensor being configured to: capture reflected light from at least a portion of the object, and generate image data from the reflected light; and a controller configured to: generate a luminance image comprising a plurality of luminance pixels based on the image data, wherein each of the plurality of luminance images corresponds to a corresponding pixel among the plurality of pixels and is generated based on signals from two ADCs associated with the corresponding pixels; generate a depth image comprising a plurality of depth pixels based on the image data, each depth pixel being associated with a depth value; determine a new depth value of a target depth pixel among the plurality of depth pixels based on the luminance pixels of the plurality of luminance pixels; and update the depth image by assigning the new depth value to the target depth pixel.
[0151] Example 12 provides the system of Example 11, wherein the target depth pixel and the luma pixel correspond to a same pixel among the plurality of pixels.
[0152] Example 13 provides the system of Example 11 or 12, wherein each of the two ADCs converts a different analog signal from a corresponding pixel into a different digital signal in the image data.
[0153] Example 14 provides the system of any of Examples 11-13, wherein the two ADCs are associated with one or more other pixels of the plurality of pixels.
[0154] Example 15 provides the system of Example 14, wherein the one or more other pixels and the corresponding pixel are arranged in a same column in the image sensor.
[0155] Example 16 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising: generating a luminance image comprising a plurality of luminance pixels based on image data from an image sensor comprising a plurality of pixels, wherein each of the plurality of luminance images corresponds to a corresponding pixel among the plurality of pixels and is generated based on signals from two ADCs associated with the corresponding pixel; generating a depth image comprising a plurality of depth pixels based on the image data, each depth pixel being associated with a depth value; determining a new depth value of a target depth pixel among the plurality of depth pixels based on the luminance pixels among the plurality of luminance pixels; and updating the depth image by assigning the new depth value to the target depth pixel.
[0156] Example 17 provides the one or more non-transitory computer-readable media of Example 16, wherein the target depth pixel and the luma pixel correspond to a same pixel of the plurality of pixels.
[0157] Example 18 provides one or more non-transitory computer-readable media of Example 16 or 17, wherein each of the two ADCs converts a different analog signal from a corresponding pixel into a different digital signal in the image data.
[0158] Example 19 provides the one or more non-transitory computer-readable media of any of Examples 16-18, wherein the two ADCs are associated with one or more other pixels of the plurality of pixels.
[0159] Example 20 provides the one or more non-transitory computer-readable media of Example 19, wherein the one or more other pixels and the corresponding pixel are arranged in a same column in the image sensor.
Claims
1. A method comprising: generating a luminance image including a plurality of luminance pixels based on image data from an image sensor including a plurality of pixels, wherein each of the plurality of luminance images corresponds to a respective pixel of the plurality of pixels and is generated based on signals from two analog-to-digital converters (ADCs) associated with the respective pixels; generating a depth image comprising a plurality of depth pixels based on the image data, each depth pixel being associated with a depth value; determining, based on a luma pixel in the plurality of luma pixels, a new depth value of a target depth pixel in the plurality of depth pixels; and The depth image is updated by assigning new depth values to target depth pixels. 2 . The method of claim 1 , wherein the target depth pixel and the luma pixel correspond to a same pixel among the plurality of pixels.
3. A method according to any one of the preceding claims, wherein each of the two ADCs converts a different analog signal from a corresponding pixel into a different digital signal in the image data.
4. A method according to any preceding claim, wherein the two ADCs are associated with one or more other pixels of the plurality of pixels. 5 . The method of claim 4 , wherein the one or more other pixels and the corresponding pixel are arranged in a same column in the image sensor.
6. A method according to any one of the preceding claims, wherein: The brightness image and the depth image capture the same object, and The target depth pixel represents the same object trajectory as the brightness pixel.
7. The method according to any one of the preceding claims, further comprising: instructing the illuminator assembly to project modulated light into a local area including the object; instructing the image sensor to capture reflected light from at least a portion of the object; and The depth image is generated based on a phase shift between the reflected light and the modulated light projected into the local area.
8. The method of claim 7, wherein the modulated light comprises continuous infrared waves, and the continuous waves have different frequencies.
9. The method according to claim 7 or claim 8, further comprising: The brightness image is generated based on the brightness of the reflected light.
10. The method according to any one of claims 7 to 9, wherein the reflected light is a first reflected light, and the method further comprises: instructing the image sensor to capture second reflected light from at least a portion of the object; and generating the brightness image based on the brightness of the second reflected light, The second reflected light has a wavelength different from that of the first reflected light.
11. A system comprising: an illuminator assembly configured to project modulated light into a localized area including the object; An image sensor includes a plurality of pixels, wherein the image sensor is configured to: capturing reflected light from at least a portion of the object, and generating image data from the reflected light; as well as The controller is configured as: generating a luminance image comprising a plurality of luminance pixels based on the image data, wherein each of the plurality of luminance images corresponds to a respective pixel of the plurality of pixels and is generated based on signals from two analog-to-digital converters (ADCs) associated with the respective pixels; generating a depth image comprising a plurality of depth pixels based on the image data, each depth pixel being associated with a depth value; determining, based on a luma pixel of the plurality of luma pixels, a new depth value of a target depth pixel in the plurality of depth pixels; and The depth image is updated by assigning new depth values to the target depth pixels. 12 . The system of claim 11 , wherein the target depth pixel and the luma pixel correspond to a same pixel among the plurality of pixels.
13. A system according to claim 11 or claim 12, wherein each of the two ADCs converts a different analog signal from a corresponding pixel into a different digital signal in the image data.
14. The system of any one of claims 11 to 13, wherein the two ADCs are associated with one or more other pixels of the plurality of pixels.
15. The system of claim 14, wherein the one or more other pixels and the corresponding pixel are arranged in a same column in the image sensor.
16. One or more non-transitory computer-readable media storing instructions executable to perform operations comprising: generating a luminance image including a plurality of luminance pixels based on image data from an image sensor including a plurality of pixels, wherein each of the plurality of luminance images corresponds to a respective pixel of the plurality of pixels and is generated based on signals from two analog-to-digital converters (ADCs) associated with the respective pixels; generating a depth image comprising a plurality of depth pixels based on the image data, each depth pixel being associated with a depth value; determining, based on a luma pixel in the plurality of luma pixels, a new depth value of a target depth pixel in the plurality of depth pixels; and The depth image is updated by assigning new depth values to target depth pixels. 17 . The one or more non-transitory computer-readable media of claim 16 , wherein the target depth pixel and the luma pixel correspond to a same pixel in the plurality of pixels.
18. The one or more non-transitory computer-readable media of claim 16 or claim 17, wherein each of the two ADCs converts a different analog signal from a corresponding pixel into a different digital signal in the image data.
19. The one or more non-transitory computer readable media of any one of claims 16 to 18, wherein the two ADCs are associated with one or more other pixels of the plurality of pixels.
20. The one or more non-transitory computer-readable media of claim 19, wherein the one or more other pixels and the corresponding pixel are arranged in a same column in the image sensor.
