Active depth sensing

Abstract

Systems and techniques for depth sensing are described. For example, a method may include obtaining a first depth image of a scene. The first depth image of the scene is associated with a first illumination configuration that includes illuminating the scene with a first type of illumination. The method may include obtaining a second depth image of the scene, where the second depth image is associated with a second illumination configuration, the second illumination configuration being different from the first illumination configuration. The second illumination configuration includes illuminating the scene with a second type of illumination. The method may include determining multipath interference (MPI) associated with the first depth image based on the second depth image. The method may also include generating an adjusted depth image based on determining the MPI associated with the first depth image, the adjusted depth image including one or more pixels from the first depth image and including one or more adjusted pixels.

Description

Technical Field

[0001] The present disclosure relates generally to depth sensing systems, and particularly to improving the resolution of depth information. Background Art

[0002] A device may use various active or passive depth sensing techniques to determine the distance of its surroundings. Passive depth sensing systems measure reflected ambient light. Active depth sensing systems emit waves from a non-ambient light source (e.g., an illuminator, a light emitting diode (LED), a laser, or another suitable light source) and measure the corresponding reflected energy. For example, an active depth sensing system or device may generate depth information illustrating or otherwise indicating the depth from the device to a scene or one or more objects in the scene. An active depth sensing system may emit one or more light pulses and measure reflections of those light pulses from an object or scene. Example active depth technologies include time of flight (ToF) and structured light (SL).

[0003] In a ToF system, light is emitted from a transmitter (or "projector"), and the reflection of the light is received at a receiver (or "sensor"). In some implementations, the round-trip time of light from the transmitter to the receiver is determined, and the distance or depth of the object reflecting the emitted light can be determined based on the round-trip time. In some cases, a ToF system that directly measures the round-trip time may be referred to as a direct time-of-flight (dToF) system. In some implementations, dToF depth sensing may require high-speed detectors and high-gain photosensors. For example, some dToF system receivers may include avalanche photodiodes. In some cases, high-resolution image sensor arrays with avalanche photodiodes may not be commercially available. In some cases, indirect time-of-flight (iToF) depth sensing may be used. In some cases, iToF measures the distance from the scene to each pixel of the camera by measuring the phase difference between the emitted light and its reflection from the scene. Based on size, accuracy, performance indicators, and cost, ToF systems are useful for many applications. For example, compared to stereo multi-camera systems that require more complex processing, ToF systems provide depth information faster and more reliably at the hardware level.

[0004] The SL system may include a light emitter (or "transmitter" or "projector") to project an infrared (IR) light distribution (such as an IR light spot distribution) onto a scene. The system or device may also include a sensor (or "receiver") that senses reflections of the light distribution to determine the distance of objects in the scene. The emitter and sensor are separated by a distance, so spatially distributed displacements and distortions occur at the sensor. The SL system uses, for example, triangulation of the spatially distributed displacements and distortions and the distance between the transmitter and receiver to determine the distance or depth of an object that reflects the emitted light back to the system. Summary of the invention

[0005] Disclosed are systems, devices, methods, and computer-readable media for depth sensing. According to at least one example, a method for depth sensing is provided. The method includes: obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration includes illuminating the scene with a first type of illumination; obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, the second illumination configuration being different from the first illumination configuration, wherein the second illumination configuration includes illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determining multipath interference (MPI) associated with the first depth image based on the second depth image; and generating an adjusted depth image based on determining that the MPI is associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0006] In another example, a device for depth sensing is provided, the device comprising at least one memory and at least one processor, the at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration includes illuminating the scene with a first type of illumination; obtain a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, the second illumination configuration is different from the first illumination configuration, wherein the second illumination configuration includes illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determine multipath interference (MPI) associated with the first depth image based on the second depth image; and generate an adjusted depth image based on determining that the MPI is associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0007] In another example, a non-transitory computer-readable medium is provided, on which instructions are stored that, when executed by one or more processors, cause the one or more processors to perform the following operations: obtain a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration includes illuminating the scene with a first type of illumination; obtain a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, which is different from the first illumination configuration, wherein the second illumination configuration includes illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determine multipath interference (MPI) associated with the first depth image based on the second depth image; and generate an adjusted depth image based on determining that the MPI is associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0008] In another example, an apparatus for depth sensing is provided. The apparatus includes: a component for obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration includes illuminating the scene with a first type of illumination; a component for obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, the second illumination configuration being different from the first illumination configuration, wherein the second illumination configuration includes illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determining multipath interference (MPI) associated with the first depth image based on the second depth image; and a component for generating an adjusted depth image based on determining that the MPI is associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0009] In some aspects, one or more of the devices described above are, are part of, or include a mobile device (e.g., a mobile phone or so-called "smart phone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a computing device of a vehicle), or other device. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors may be used to determine the position and / or posture of the device, the state of the device, and / or for other purposes.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various aspects of the disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings and in which like reference numerals refer to similar elements.

[0013] Figure 1 A block diagram showing an example device including a hybrid depth sensing system according to some examples of the present disclosure;

[0014] Figure 2 An example time of flight (ToF) system according to some examples of the present disclosure is illustrated;

[0015] Figure 3A An example ToF system including a transmitter and a sensor according to some examples of the present disclosure is illustrated;

[0016] Figure 3B illustrates example sensor pixels according to some examples of the present disclosure;

[0017] Figure 3C illustrates a pulse diagram illustrating a pulse signal from a transmitter and a corresponding reflected signal received at a sensor according to some examples of the present disclosure;

[0018] Figure 4is a block diagram illustrating an example ToF system according to some examples of the present disclosure;

[0019] Figure 5A illustrates an example environment in which multipath interference (MPI) may affect ToF depth sensing according to some examples of the present disclosure;

[0020] Figure 5B Another example environment in which MPI may affect ToF depth sensing according to some examples of the present disclosure is illustrated;

[0021] Figure 5C Another example environment in which MPI may affect ToF depth sensing according to some examples of the present disclosure is illustrated;

[0022] Fig. 6A An example scene including a corner of a room having a vertex, wherein two of the walls intersect with a ceiling, according to some examples of the present disclosure is illustrated;

[0023] Figure 6B 1 illustrates some examples of the present disclosure as generated by a conventional ToF system. Figure 5A An example depth map of a scene;

[0024] Figure 7 Some examples of the present disclosure are illustrated. Fig. 6A A graph of the corresponding X distance and Z distance of the scene;

[0025] Figure 8 An example structured light (SL) system according to some examples of the present disclosure is illustrated;

[0026] Fig. 9A illustrates an example hybrid depth sensing system operating in a first mode according to some examples of the present disclosure;

[0027] Fig. 9B Illustrate some examples of the present disclosure operating in the second mode Fig. 10A An example hybrid depth sensing system of;

[0028] Fig. 9C Illustrate some examples of the present disclosure operating in the third mode Fig. 10A An example hybrid depth sensing system of;

[0029] Fig. 10A Some examples of the present disclosure are illustrated. Fig. 9A A simplified illustration of a hybrid depth sensing system is shown in;

[0030] Fig. 10B Some examples of the present disclosure are illustrated. Fig. 9B A simplified illustration of a hybrid depth sensing system is shown in;

[0031] Fig. 10C Some examples of the present disclosure are illustrated. Fig. 9C A simplified illustration of a hybrid depth sensing system is shown in FIG.

[0032] Fig.11 is a diagram illustrating an adjusted depth image based on a ToF depth image and a SL depth image according to some examples of the present disclosure.

[0033] Fig.12 is a flowchart illustrating an example of a process for processing one or more frames according to some examples of the present disclosure; and

[0034] Fig.13 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0035] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0036] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present application as set forth in the appended claims.

[0037] Various aspects of the present disclosure relate to active depth sensing systems and techniques. In some aspects, active depth sensing systems and techniques may include systems that incorporate time of flight (ToF), sparse ToF (sToF), and structured light (SL) technologies into a single device. As used herein, the terms "ToF system" and "traditional ToF system" may refer to an indirect time of flight (iToF) system. In some cases, an iToF system may include illuminating a scene with active illumination of diffuse light. ToF systems generally produce higher resolution depth information compared to SL systems and ToF systems. In some cases, an sToF system may include illuminating a scene with points, which may be a random grid, a pseudo-random grid, a structured grid, or a periodic grid. However, the depth information generated by a conventional ToF system is often affected by image artifacts caused by multipath interference (MPI). On the other hand, the depth information generated by the SL system is relatively unaffected by MPI. In some cases, relative to a conventional ToF system, the depth information degraded by sToF may be affected by less MPI and / or a correctable amount of MPI.

[0038] In some aspects of the present disclosure, an active depth sensing system may combine the advantages of ToF (e.g., higher resolution), SL (e.g., no MPI), and / or point ToF (e.g., reduced MPI with medium resolution). For example, embodiments of the present disclosure disclose a hybrid depth sensing system configured to operate in a hybrid depth sensing mode. In one example, the system may determine SL information of an object or scene, and utilize the SL information to mitigate the effects of MPI when determining ToF depth information (e.g., ToF information, point ToF information, and / or any combination thereof) of the object or scene. In another example, the hybrid depth sensing system may determine point ToF information of an object or scene, and utilize the point ToF information to mitigate the effects of MPI when determining ToF depth information of the object or scene. The system may include a composite sensor, a hybrid transmitter, and / or a programmable architecture, thereby saving device space and requiring fewer device components compared to a conventional ToF system (or a system supporting ToF, point ToF, and SL modes), while also improving the speed and accuracy of the system in generating depth information.

[0039] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or being coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0040] Figure 1A block diagram of an example hybrid depth sensing system 100 configured for active depth sensing using ToF, point ToF, and / or SL techniques is shown. It should be understood that ToF, point ToF, and SL are example active depth techniques, and in some implementations, the hybrid depth sensing system 100 may use other active depth techniques. In some examples, the hybrid depth sensing system 100 may be configured to use ToF techniques to generate high-resolution depth information while using point ToF and / or SL techniques to mitigate the effects of MPI in the depth information. The hybrid depth sensing system 100 may include or be coupled to one or more emitters 101, one or more sensors 102, one or more processors 104, a memory 106 storing instructions 108, and an active depth controller 110 (which may include one or more signal processors 112). One or more emitters 101 may include or be coupled to a diffractive optical element (DOE) 105. The DOE 105 may optionally be included in or coupled to the hybrid depth sensing system 100. One or more emitters 101 may include or be coupled to a diffuser 107. Diffuser 107 may optionally be included in or coupled to hybrid depth sensing system 100. For purposes of discussion herein, hybrid depth sensing system 100 may be referred to as a "hybrid depth sensing system". Further for purposes of discussion herein, "hybrid depth sensing system" may alternatively refer to only one or more components of hybrid depth sensing system 100 (e.g., active depth controller 110, one or more transmitters 101, sensor 102, DOE 105, and / or diffuser 107) and / or any other components that may be used for active depth sensing. In some implementations, the device may be a wireless communication device.

[0041] In some embodiments, one or more emitters 101 may be a single hybrid laser projector that is capable of switching between projecting a first light distribution (e.g., using diffuser 107) during a first projection mode (e.g., ToF projection mode) of one or more emitters 101, projecting a second light distribution (e.g., using DOE 105) during a second projection mode (e.g., SL projection mode) of one or more emitters 101, and / or projecting a third light distribution (e.g., point ToF projection mode) of one or more emitters 101. When operating in the SL projection mode, the DOE 105 may enable the one or more emitters 101 to transmit the second light distribution, which may be, for example, a known DOE dot pattern, a codeword DOE projection, or a random dot projection or distribution, etc. Similarly, when operating in the point ToF projection mode, the DOE 105 may enable the one or more emitters 101 to transmit a third light distribution, which may be, for example, a known DOE dot pattern, a codeword DOE projection, or a random dot projection or distribution, etc. The diffuser 107 can be switchable so that the diffuser is "off" (or "disabled" or "off") when the hybrid depth sensing system 100 operates in the SL projection mode and / or the point ToF projection mode, and the diffuser is "on" (or "enabled" or "connected") when the hybrid depth sensing system 100 operates in the ToF projection mode. More specifically, when operating in the ToF projection mode, the diffuser 107 is turned on, which causes one or more emitters 101 to transmit a second light distribution (e.g., a diffuse distribution). In some cases, one or more emitters 101 can be synchronized to project a first light distribution (e.g., a diffuse distribution) during the ToF projection mode, a second light distribution (e.g., a sparse distribution) during the SL projection mode, and a third light distribution (an additional sparse distribution) during the point ToF projection mode.

[0042] In some embodiments, one or more emitters 101 may include multiple projectors. For example, one or more emitters 101 may include a first emitter that can be used to operate the hybrid depth sensing system 100 in a ToF projection mode and a point ToF projection mode. In some aspects, the first emitter (e.g., using DOE 105) may transmit a third light distribution when operating in the point ToF projection mode. In some specific implementations, when operating in the ToF projection mode, the first emitter may transmit a first light distribution. In some examples, one or more emitters 101 may include a second emitter that is used to operate in the SL projection mode. In some cases, when the hybrid depth sensing system 100 operates in the SL projection mode, a second transmitter (e.g., using DOE 105 or an additional DOE (not shown)) may transmit a second light distribution.

[0043] In some embodiments, one or more sensors 102 may be a single hybrid hybrid depth sensor that is used to receive reflected light according to ToF, point ToF, and / or SL (or "readout") modes. Sensor 102 may be configured to switch between operating in a first sensing mode (e.g., ToF sensing mode), a second sensing mode (e.g., SL sensing mode), and a third sensing mode (e.g., point ToF sensing mode). For example, sensor 102 may be a composite CMOS image sensor that is configured to switch between operating in a first sensing mode, a second sensing mode, and / or a third sensing mode (or alternate between the first sensing mode, the second sensing mode, and / or the third sensing mode). In some cases, one or more sensors 102 may include a first sensor that is used to receive reflected light for a first sensing mode (e.g., ToF sensing mode) and a third sensing mode (e.g., point ToF sensing mode). In some specific implementations, one or more sensors 102 may include a second sensor that is used to receive light for a second sensing mode (e.g., SL sensing mode). The sensing mode may depend on the distribution (e.g., SL, point ToF, or diffuse) that one or more emitters 101 are projecting. In some aspects, sensor 102 may be based on a monolithic pixel array architecture, for example, with time division multiplexing readout (TDMR) capability. In other embodiments, one or more sensors 102 may include one or more general ToF sensors operating in conjunction with multiple projectors.

[0044] In some embodiments, the active depth controller 110 may be a computing element for calculating depth information. The active depth controller 110 may be configured to calculate depth information using ToF technology (e.g., for ToF and / or point ToF projection mode), and to calculate depth information using SL technology (e.g., for SL projection mode). For the purposes of discussion herein, depth information calculated using SL technology may also be referred to as "SL depth information" or "SL information", etc. For the purposes of discussion herein, depth information calculated using ToF technology may also be referred to as "ToF depth information" or "ToF information", etc. For the purposes of discussion herein, depth information calculated using point ToF technology may also be referred to as "point ToF depth information" or "point ToF information", etc. In some aspects, the active depth controller 110 may use SL depth information and / or point ToF depth information as a reference for calculating or supplementing ToF depth information, which may help compensate for MPI errors in ToF depth information.

[0045] For example, in some specific implementations, the active depth controller 110 may be configured to combine the ToF depth information with the point ToF depth information and / or the SL depth information to compensate for the MPI error in the ToF depth information. In an illustrative example, the active depth controller 110 may determine the numerical correspondence between the ToF depth information and the point ToF depth information. As a non-limiting example, the numerical correspondence may be determined by calculating the difference between the ToF depth information and the point ToF depth information. In some cases, if the difference exceeds the MPI threshold, the ToF depth information may be adjusted based on the point ToF depth information. In an illustrative example, adjusting the depth value may include interpolation between the depth information and the point ToF information. Similarly, in some cases, the active depth controller 110 may determine the numerical correspondence between the ToF depth information and the SL depth information. In some aspects, the ToF depth information may be adjusted based on the SL depth information. In another illustrative example, the active depth controller 110 may determine the numerical correspondence between the point ToF depth information and the SL depth information. In some cases, the point ToF depth information may be adjusted based on the SL depth information.

[0046] In some cases, one or more image fusion techniques may be used to adjust depth information obtained using two different illumination configurations. In some cases, interpolation may be used to determine the amount of adjustment. For example, depth information acquired with a sparse illumination pattern (e.g., point ToF, SL) may include depth information of a subset of pixels corresponding to the illumination pattern. In some examples, interpolation may be used to estimate depth values ​​in areas of the scene that are not represented in the depth information. In some specific implementations, a window technique may be used to determine which pixel(s) of the depth information from the sparse illumination pattern will be used as a reference for adjusting depth information for different illumination configurations (e.g., adjusting ToF based on point ToF and / or SL, and adjusting point ToF based on SL). In some examples, a fixed grid technique may be used to determine which pixel(s) of the depth information from the sparse illumination pattern will be used as a reference for adjusting depth information for different illumination configurations (e.g., adjusting ToF based on point ToF and / or SL, and adjusting point ToF based on SL). Although specific examples of image techniques are provided herein, other image fusion techniques may be used without departing from the scope of the present disclosure.

[0047] In some embodiments, the sensor 102 may be a reconfigurable instruction cell array (RICA), which is a proprietary, real-time, low-power, (re)programmable image signal processing (ISP) and active sensing processing engine. In some aspects, stacking the RICA programmable implementation with the hybrid NIR sensor described herein may enable the active depth controller 110 to switch programming "on the fly" to calculate SL depth information, point ToF depth information, and / or ToF depth information while reducing the number of components of the sensor (e.g., sensor 102). In other embodiments, the active depth controller 110 may be a general purpose sensor.

[0048] In some aspects, the active depth controller 110 may be configured to control (or otherwise operate) at least one of the one or more transmitters 101 and the one or more sensors 102 to coordinate their respective operating modes, so that the one or more transmitters 101 and the one or more sensors 102 operate in their respective SL mode, point ToF mode, and / or ToF mode. In some aspects, the active depth controller 110 may be controlled, coordinated, or otherwise operated by one or more other components of the hybrid depth sensing system 100 (such as one or more processors 104 and / or memory 106).

[0049] The hybrid depth sensing system 100 may optionally include or be coupled to a display 114 and a plurality of input / output (I / O) components 116. The one or more sensors 102 may be or otherwise be coupled to a camera, such as a single camera, a dual camera module, or a module having any number of other camera sensors (not shown). The signal processor 112 may be configured to process captures from the sensors 102. The hybrid depth sensing system 100 may also include one or more optional sensors 120 (such as gyroscopes, magnetometers, inertial sensors, NIR sensors, etc.) coupled to the one or more processors 104. The hybrid depth sensing system 100 may include additional features or components not shown.

[0050] The memory 106 may be a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 for performing all or part of one or more operations described in the present disclosure. The one or more processors 104 may be one or more suitable processors capable of executing scripts or instructions (such as the instructions 108) of one or more software programs stored within the memory 106. In some aspects, the one or more processors 104 may be one or more general-purpose processors that execute the instructions 108 to cause the hybrid depth sensing system 100 to perform any number of functions or operations. In additional or alternative aspects, the one or more processors 104 may include integrated circuits or other hardware for performing functions or operations without the use of software. Although in Figure 8 104, but one or more processors 104, memory 106, active depth controller 110, optional display 114, optional I / O components 116, and optional sensors 120 may be coupled to each other in various arrangements. For example, one or more processors 104, memory 106, active depth controller 110, optional display 114, optional I / O components 116, and / or optional sensors 120 may be coupled to each other via one or more local buses (not shown for simplicity).

[0051] The display 114 may be any suitable display or screen that allows user interaction and / or presents items (such as depth information or preview images of a scene) for viewing by the user. In some aspects, the display 114 may be a touch-sensitive display. The I / O component 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user. For example, the I / O component 116 may include (but is not limited to) a graphical user interface, a keyboard, a mouse, a microphone and a speaker, a squeezable border or border of the hybrid depth sensing system 100, a physical button located on the hybrid depth sensing system 100, and the like. The display 114 and / or the I / O component 116 may provide a preview image or depth information of a scene to the user, and / or receive user input for adjusting one or more settings of the hybrid depth sensing system 100 (such as adjusting the emission intensity of one or more emitters 101, determining or switching one or more operating modes of the hybrid depth sensing system 100, adjusting the emission field of one or more emitters 101, and the like).

[0052] The active depth controller 110 may include or may be otherwise coupled to a signal processor 112, which may be one or more processors for processing depth information from the one or more sensors 102. The active depth controller 110 may be configured to switch at least one or more of the one or more transmitters 101 or at least one of the one or more sensors 102 between one or more operating modes. The active depth controller 110 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

[0053] One or more emitters 101 may vary the emission field for different modes of operation. In some example implementations, one or more emitters 101 may include a focusing device for adjusting the size of the emission / transmission field. In one example, a mirror attached to an actuator (such as a micro-electromechanical system (MEMS) actuator) may adjust the focus of light emission from one or more emitters 101. In another example, an adjustable holographic optical element (HOE) may adjust the focus of light emission from one or more emitters 101. In another example, a shapeable DOE (such as a piezoelectric material for adjusting the shape) may be adjusted to focus the diffraction spot of the emitted light.

[0054] In some other example implementations, the hybrid depth sensing system 100 may use multiple light emitters (not shown) instead of one or more emitters 101 or in combination with the one or more emitters to emit light. The emitters may include a first group of light emitters (e.g., in a first light emitter array) for emitting light having a first transmission field. The emitters may also include a second or different group of light emitters (e.g., in a second light emitter array) for emitting light having a second transmission field. At a common depth from the one or more emitters 101, the first field may be greater than the second field. In some example implementations, the first group of light emitters may be active for the first mode of the one or more emitters 101 and the third mode of the one or more emitters 101, and the second group of light emitters may be active for the second mode of the one or more emitters 101.

[0055] Figure 2 An example ToF system 200 is shown. The ToF system 200 may be used to generate depth information for a scene including a surface 206, or may be used for other applications where the surface or other portions of the scene are measured. The ToF system 200 may include a transmitter 202 and a receiver 208. The transmitter 202 may be referred to as a "light projector," "transmitter," "projector," "emitter," etc., and should not be limited to a particular transmitting component. Similarly, the receiver 208 may be referred to as a "light sensor," "detector," "sensor," "sensing element," "photodetector," etc., and should not be limited to a particular receiving component.

[0056] The transmitter 202 may be configured to transmit, emit, or project a signal, such as a light field, onto a scene. Although the ToF system is described in various examples as emitting light, which may include near infrared light (NIR), other frequency signals may be used, such as short wavelength infrared light (SWIR), microwaves, radio frequency signals, sound, etc. The present disclosure should not be limited to a specific frequency range of the emitted signal.

[0057] In some cases, the configuration of the ToF system 200 can be used for direct time of flight (dToF) measurement. In some examples, a dToF transmitter 202 transmits light 204 toward a scene including a surface 206. In some cases, the transmitted light 204 includes light pulses 214 at known time intervals (such as at regular time intervals). In some cases, a dToF receiver 208 includes a sensor 210 for sensing a reflection 212 of the transmitted light 204. In some cases, the reflection 212 includes a reflected light pulse 216, and the ToF system 200 determines a round trip time 222 of the light by comparing the timing 218 of the transmitted light pulse to the timing 220 of the reflected light pulse 216. In a dToF measurement, the distance of the surface 206 from the ToF system 200 can be calculated as half of the round trip time multiplied by a transmission speed (such as the speed of light for light transmission).

[0058] In some cases, the ToF system 200 (e.g., transmitter 202, receiver 208, sensor 210) can be used for indirect time of flight (iToF) measurement. In some cases, the round trip time 222 for iToF measurement can be derived by measuring the phase shift between reflected light and transmitted light, as shown below with respect to FIG. 3A to FIG. 3C exemplified.

[0059] The sensor 210 may include a photodiode array to measure or sense reflections. Alternatively, the sensor 210 may include a CMOS sensor or other suitable light-sensitive sensor including a plurality of pixels or regions for sensing. The ToF system 200 identifies a reflected light pulse 216 sensed by the sensor 210 when the magnitude of the pulse is greater than a certain value. For example, the ToF system 200 measures the magnitude of ambient light and other interference in the absence of a signal and determines whether a further measurement is greater than a previous measurement by a certain value.

[0060] Figure 3A An example ToF system 320 is shown that includes an emitter 325 and a sensor 330. The emitter 325 transmits a light pulse toward an object 335, and the light pulse is reflected by the object 335 back to the sensor 330. The light reflected back to the sensor 330 may have a different phase than the light emitted from the emitter 325. Figure 3BAn example sensor pixel 340 is shown that may be included, for example, in sensor 230. Sensor pixel 340 includes a photodiode 342 for converting photons from reflected light into an electric current. Sensor pixel 340 may also include one or more capacitors (e.g., C1 and C2) to store energy from the electric current. ToF system 320 may calculate the distance between ToF system 320 and object 335 in part by comparing voltages (e.g., V0 and V180) with their corresponding phases (e.g., Φ0 and Φ180, respectively).

[0061] Figure 3C 3 shows an example pulse signal transmitted from transmitter 325 and the corresponding reflection of the pulse signal received at sensor 330 (e.g., by a sensor such as Figure 3A 235). The reflected signal is phase delayed relative to the pulse signal. The ToF system 320 can open and close the shutter at multiple specific phase offsets (e.g., phase 0°, phase 180°, phase 90°, and phase 270°) relative to the pulse signal to expose the sensor 330. During each exposure cycle (grey shading), charge can be stored by one or more storage elements, such as by Figure 3B Capacitors C1 and C2 store. For example, during a first exposure cycle, C1 may store charge (Q1) and C2 may store charge (Q2), where Q1 is the accumulated charge from the reflected signal when the shutter is opened with a 0° phase offset, and where Q2 is the accumulated charge from the reflected signal when the shutter is opened with a 180° phase offset. During a second exposure cycle, C1 may store charge (Q3) and C2 may store charge (Q4), where Q3 is the accumulated charge from the reflected signal when the shutter is opened with a 90° phase offset, and where Q4 is the accumulated charge from the reflected signal when the shutter is opened with a 270° phase offset. For each of these exposure cycles, the ToF system 320 may calculate a phase offset (φ) between the pulse signal and the reflected signal based on the charges stored on C1 and C2:

[0062]

[0063] The calculated phase shift between the pulse signal and the reflected signal is proportional to the distance between the corresponding sensor pixel (such as sensor pixel 340) and the object 335:

[0064]

[0065] Where c is the speed of light, and f is the frequency of the modulated signal. Based on the determined distance from each pixel of sensor 330 to object 335, ToF system 320 may generate depth information for objects in the scene.

[0066] Figure 4 An example ToF system 400 is shown. The ToF system 400 includes a transmitter 415 and a sensor 435. The transmitter 415 emits a pulse signal 420 toward an object 425, and the sensor 435 receives a corresponding reflected signal 430. The pulse signal 420 may be an AMCW pulse of light. Figure 3C As described, the ToF system 400 can determine the distance to one or more pixels of the sensor 435 based on the ratio of the phase difference between the pulse signal 420 and the reflected signal 430. The pulse signal 420 and the reflected signal 430 can be along a direct path 440. In some implementations, the ToF system 400 can include a bandpass filter 431 positioned between the path of the reflected signal 430 and the sensor 435. In some cases, the bandpass filter 431 can filter out (e.g., attenuate) ambient light while allowing light at the wavelength of the emitter 415 to pass through. In an illustrative example, the bandpass filter 431 can have a passband width between 10 nm and 50 nm.

[0067] Some environments (e.g., with corners, convex areas, and / or reflective surfaces) may cause different light pulses to arrive at the ToF system sensor along multiple reflection paths and be recombined at the sensor, which is referred to as MPI. For the purposes of this discussion, MPI may also be referred to as a "multipath effect" or an "MPI effect." MPI may cause the ToF system to overestimate the amount of charge accumulated for one or more phase shifts of the corresponding pulse signal. This overestimation may cause the ToF system to inaccurately calculate the corresponding phase shift between the pulse signal and the reflected signal. As a result, the ToF system may inaccurately calculate the corresponding distance from one or more of the sensor pixels to the object or scene, which may cause distortion (or "bumps") in the corresponding depth information.

[0068] Figure 5A An example environment 500 is shown in which MPI may affect ToF depth sensing. The ToF system includes an emitter 515 and a sensor 535. The scene includes an object 525 and an object 527. Object 527 may have a mirror-like surface. The emitter 515 sends a pulse signal 520 and a pulse signal 522 toward the object 525. The sensor 535 receives the corresponding reflection signal 530 and the reflection signal 532, respectively. Figure 4 4 and 5. In contrast, pulse signal 522 and reflection signal 532 follow an indirect path 550 (e.g., reflected from object 527), such that reflection signal 532 may arrive at sensor 535 at the same time as reflection signal 530, thereby causing MPI.

[0069] Figure 5B Another example environment 560 is shown in which MPI may affect ToF depth sensing. The ToF system includes a transmitter 545 and a sensor 565. The scene includes an object 555 and an object 557. Object 557 may have a semi-transparent surface. Transmitter 545 sends a pulse signal 521 and a pulse signal 523. Sensor 565 receives corresponding reflection signals 531 and 533, respectively. Pulse signal 521 and reflection signal 531 follow path 541 (e.g., reflected from object 555). Pulse signal 523 and reflection signal 533 follow path 551 (e.g., reflected from object 557). Reflection signal 533 may arrive at sensor 565 at the same time as reflection signal 531, thereby causing MPI.

[0070] Figure 5C Another example environment 570 is shown in which MPI may affect ToF depth sensing. The ToF system includes a transmitter 575 and a sensor 595. The scene includes an object 585 and an object 587, which may represent two walls that meet at a corner point. The transmitter 575 sends a pulse signal 591 toward the object 585, and the sensor 595 receives a corresponding reflection signal 592. In addition, possibly due to the reflective properties of the object 585 or the object 587, the reflection signal 593 and the reflection signal 594 may arrive at the sensor 595 at the same time as the reflection signal 592, thereby causing MPI.

[0071] Fig. 6A An example scene 600 is shown including a corner of a room with a vertex 610 where two of the walls intersect the ceiling. It should be understood that the vertex 610 is the point farthest from the sensor of the ToF system (eg, the point with the highest z distance). Figure 6B shows a graph from a conventional ToF system (“conventional ToF”) Fig. 6A Example depth map 650 of a scene. However, similar to Figure 5C In the example environment 570 of FIG. 5 , multiple reflection signals may reach the sensor (eg, due to the corner of the room having vertex 660 ), thereby causing MPI. A conventional ToF system may superimpose multiple reflection signals, ultimately causing the corresponding area in the depth map 650 to appear to have a uniform depth.

[0072] For illustration purposes, Figure 7 Shown for Fig. 6AGraph 700 of corresponding x distances as measured by a ToF system and corresponding z distances as measured by the ToF system for scene 600. The x distance may represent the horizontal distance in meters (m) from the center of the scene at 0 m. The z distance may represent the depth distance from the ToF system's sensor to an object in the scene, such as a surface of a wall. The lower polyline represents the actual ("true") x distance and z distance of the corner, while the upper polyline represents the distance measured by the ToF system. The true distance polyline accurately shows the vertex of the corner of the room, which is a sharp point. The measured distance polyline depicts the vertex as a bowl-shaped curve. This inaccurate measurement results in incorrect distance calculations, such as Figure 6B As shown in depth map 650.

[0073] Figure 8 An example SL system 800 is shown. As described above, SL is relatively unaffected by MPI and generally produces better results than ToF systems (e.g., Figure 2 The SL system can project light in a point distribution (or another suitable shape of focused light). For the purposes of this discussion, the point distribution may be referred to as a "pattern," "SL pattern," or "point pattern," etc., and the pattern may be predefined or random. Points of light may be projected onto a scene, and reflections of the points of light may be received by the SL system. The depth of objects in the scene may be determined by comparing the pattern of received light to the pattern of transmitted light, or by comparing the pattern of received light to the pattern of received light reflected from a known distance via a calibration process. When comparing the patterns, a portion of a predefined distribution of transmitted light may be identified in the received light. The SL system may use a structured light projector to project a light distribution (such as a distribution of points of light or other shapes).

[0074] The light distribution emitted by the SL projector may not change. A denser light distribution (such as additional light spots or more instances of focused light in an area compared to a sparser light distribution) can result in depth measurements with higher lateral resolution. For example, a denser light distribution can resolve closer objects than a sparser light distribution. However, the intensity of each light spot of a denser distribution is lower than the intensity of each light spot of a sparser distribution, where the total intensity between the distributions is similar. Therefore, ambient light interference may make it more difficult to identify reflections from a denser light distribution than from a sparser light distribution. For example, an SL projector may project IR light (such as NIR light) having a wavelength of 850 nanometers (nm), 905nm, or 940nm (or other suitable wavelengths). The SL receiver may receive reflections of the IR light as well as sunlight and other ambient light. Ambient light may cause interference from the IR light spots. Therefore, because the SL receiver captures additional ambient light, brightly illuminated scenes (such as outdoor scenes during the day) may cause more interference than darker scenes (such as indoor scenes or night scenes).

[0075] The SL system 800 (which may also be referred to herein as the SL system) may be used to generate depth information for a scene 806. For example, the scene 806 may include a face, and the SL system 800 may be used to identify or authenticate the face. The SL system 800 includes a transmitter 802 and a receiver 808. The transmitter 802 may be referred to as a "transmitter," "projector," "emitter," etc., and should not be limited to a specific transmitting component. In the following disclosure, the terms "projector" and "transmitter" may be used interchangeably. The receiver 808 may be referred to as a "detector," "sensor," "sensing element," "photodetector," etc., and should not be limited to a specific receiving component.

[0076] Although the present disclosure refers to the distribution as a light distribution, any suitable signal of other frequencies (such as radio frequency waves, sound waves, etc.) may be used. In addition, although the present disclosure refers to the distribution as including multiple light spots, the light may be focused into any suitable size and dimension. For example, the light may be projected in a straight line, a square, or any other suitable size. In addition, the present disclosure may refer to the distribution as a codeword distribution, where a defined part of the distribution (such as a predefined patch of light spots) is called a codeword. If the distribution of the light spots is known, it is possible to know the codeword of the distribution. However, the distribution may be organized in any way, and the present disclosure should not be limited to a specific type of distribution, or a specific type of signal or pulse.

[0077] The transmitter 802 may be configured to project or transmit a distribution of light spots 804 onto a scene 806. White circles in the distribution 804 may indicate locations where no light is projected for possible spot locations, and black circles in the distribution 804 may indicate locations where light is projected for possible spot locations. In some example implementations, the transmitter 802 may include one or more light sources 824 (such as one or more lasers), a lens 826, and a light modulator 828. The transmitter 802 may also include an aperture 822 from which the transmitted light escapes the transmitter 802. In some implementations, the transmitter 802 may also include a diffractive optical element (DOE) for diffracting emissions from the one or more light sources 824 into additional emissions. In some aspects, the light modulator 828 (for adjusting the intensity of the emissions) may include a DOE. The transmitter 802 may transmit one or more laser beams from a light source 824 through a lens 826 (and / or through a DOE or light modulator 828) onto the scene 806 while projecting the light spot distribution 804 onto the scene 806. The transmitter 802 may be positioned on the same reference plane as the receiver 808, and the transmitter 802 and the receiver 808 may be separated by a distance, which is referred to as a baseline (812).

[0078] In some example implementations, the light projected by the transmitter 802 may be IR light. The IR light may include portions of the visible spectrum and / or portions of the spectrum that are not visible to the naked eye. In one example, the IR light may include NIR light (which may or may not include light within the visible spectrum) and / or IR light (such as far infrared (FIR) light) (which is outside the visible spectrum). The term "IR light" should not be limited to light having a specific wavelength within or near the wavelength range of IR light. In addition, IR light is provided as an example emission from the transmitter. In the following description, light of other suitable wavelengths may be used, for example, light outside the wavelength range of IR light or ultraviolet light in portions of the visible spectrum. Alternatively, other signals having different wavelengths may be used, such as microwaves, radio frequency signals, and other suitable signals.

[0079] The scene 806 may include objects at different depths from the SL system (such as from the transmitter 802 and the receiver 808). For example, objects 806A and 806B in the scene 806 may be at different depths. The receiver 808 may be configured to receive reflections 810 of the transmitted light spot distribution 804 from the scene 806. To receive the reflections 810, the receiver 808 may capture an image. When capturing the image, the receiver 808 may receive the reflections 810, as well as (i) other reflections from other portions of the light spot distribution 804 at different depths from the scene 806 and (ii) ambient light. Noise may also be present in the captured image.

[0080] In some example implementations, the receiver 808 may include a lens 830 to focus or direct the received light (including reflections 810 from objects 806A and 806B) onto a sensor 832 of the receiver 808. The receiver 808 may also include an aperture 820. In some cases, the receiver 808 may include a bandpass filter 831. For example, Figure 8A bandpass filter 831 positioned between the lens 830 and the sensor 832 is illustrated. In some cases, the bandpass filter 831 may be placed between the aperture 820 and the lens 830. In some cases, the bandpass filter 831 may filter out (e.g., attenuate) ambient light while allowing light at the wavelength of the light source 824 to pass through. In one illustrative example, the bandpass filter 831 may have a passband width between 10 nm and 50 nm. Assuming an example of receiving only the reflection 810, the depth of objects 806A and 806B may be determined based on the baseline 812, the displacement and distortion of the light distribution 804 (such as in the form of a codeword) in the reflection 810, and the intensity of the reflection 810. For example, the distance 834 from the position 816 to the center 814 along the sensor 832 may be used to determine the depth of the object 806B in the scene 806. Similarly, the distance 836 from the position 818 to the center 814 along the sensor 832 may be used to determine the depth of the object 806A in the scene 806. The distance along the sensor 832 may be measured in terms of the number of pixels of the sensor 832 or in distances such as millimeters.

[0081] In some example implementations, the sensor 832 may include an array of photodiodes (such as avalanche photodiodes) for capturing an image. To capture the image, each photodiode in the array may capture light that strikes the photodiode and may provide a value (capture value) indicating the intensity of the light. Thus, the image may be a capture value provided by the photodiode array.

[0082] In addition to or as an alternative to the sensor 832 including a photodiode array, the sensor 832 may include a complementary metal oxide semiconductor (CMOS) sensor. To capture the image via a light-sensitive CMOS sensor, each pixel of the sensor may capture light impinging on the pixel and may provide a value indicative of the intensity of the light. In some example implementations, the photodiode array may be coupled to the CMOS sensor. In this way, the electrical pulses generated by the photodiode array may trigger the corresponding pixels of the CMOS sensor to provide the captured values.

[0083] Sensor 832 may include at least a number of pixels that is equal to the number of possible light points in distribution 804. For example, a photodiode array or a CMOS sensor may include a number of photodiodes or a number of pixels, respectively, that corresponds to the number of possible light points in distribution 804. In some cases, sensor 832 may include a number of photodiodes or a number of pixels that is greater than the number of possible light points of distribution 804 (e.g., 10 times greater, 100 times greater, 1000 times greater, or any other suitable number of photodiodes or pixels). In some cases, a light point reflected by a scene may be detected by multiple photodiodes or pixels of sensor 832. Sensor 832 may be logically divided into groups of pixels or photodiodes corresponding to the size of a bit of a codeword (such as 4x4 groups). A pixel or photodiode group may also be referred to as a bit, and a portion of a captured image of a bit from sensor 832 may also be referred to as a bit. In some example implementations, sensor 832 may include the same number of bits as distribution 804.

[0084] If light source 824 transmits IR light (such as NIR light having a wavelength of, for example, 940 nm), sensor 832 may be an IR sensor for receiving reflections of the NIR light. Sensor 832 may also be configured to capture images using a flood illuminator (not illustrated). As illustrated, distance 834 (corresponding to reflection 810 from object 806B) is less than distance 836 (corresponding to reflection 810 from object 806A). Using triangulation based on baseline 812 and distances 834 and 836, different depths of objects 806A and 806B in scene 806 may be determined when generating depth information for scene 806. Determining depth may also include determining a displacement or distortion of distribution 804 in reflection 810.

[0085] Despite Figure 8 , but one or more of these components may be implemented together or include additional functionality. The SL system 800 may not require all of the described components, or the functionality of the components may be separated into separate components. There may also be additional components that are not illustrated. For example, the receiver 808 may include a bandpass filter to allow signals having a certain wavelength range to pass to the sensor 832 (thereby filtering out signals having wavelengths outside of this range). In this way, some incidental signals (such as ambient light) can be prevented from interfering with the capture of the sensor 832. The range of the bandpass filter can be centered on the transmission wavelength of the transmitter 802. For example, if the transmitter 802 is configured to transmit NIR light having a wavelength of 940nm, the receiver 808 may include a bandpass filter that is configured to allow NIR light having a wavelength in the range of, for example, 800nm ​​to 960nm. Therefore, with respect to Figure 7The described example is for illustrative purposes, and the present disclosure should not be limited to the example SL system 800 .

[0086] For a light projector such as transmitter 802, the light source may be any suitable light source. In some example implementations, the light source 824 may include one or more distributed feedback (DFB) lasers. In some other example implementations, the light source 824 may include one or more vertical cavity surface emitting lasers (VCSELs).

[0087] A DOE is a material located in the projection path of light from a light source. The DOE can be configured to split a single point of light into multiple points of light. For example, the material of the DOE can be a translucent or transparent polymer with a known refractive index. The surface of the DOE can include peaks and valleys (changing the depth of the DOE) so that when light passes through the DOE, a single point of light is split into multiple points of light. For example, the DOE can be configured to receive one or more points of light from one or more lasers and project an expected distribution of more points of light than the number of points of light emitted by the one or more lasers. Although the accompanying drawings may illustrate that the depth of the DOE varies only along one axis of the DOE, these drawings are only used to help describe various aspects of the present disclosure. The peaks and valleys of the surface of the DOE can be located at any part of the surface of the DOE and cause any suitable changes in the depth of the various parts of the DOE, and the present disclosure should not be limited to a specific surface configuration of the DOE.

[0088] If the light source 824 includes a laser array (such as a VCSEL array), a portion of the spot distribution may be projected by the array. The DOE can be used to replicate the portion when projecting the spot distribution. For example, the DOE can split the projection from the array into multiple instances, and the projected pattern can be a repetition of the projection from the array. In some example implementations, the DOE may be configured to repeat the projection vertically, horizontally, or at an angle between vertical and horizontal relative to the projection. The repeated instances may be overlapping, non-overlapping, or any suitable configuration. Although the examples describe a DOE configured to split the projection from the array and stack the instances up and down, the present disclosure should not be limited to specific types of DOE configurations and repetitions of projections.

[0089] Fig. 9A An example hybrid depth sensing system 900 operating in a first (eg, ToF) mode is shown. For example, the hybrid depth sensing system 900 may correspond to Figure 1 In some aspects, the diffuser 930 can be a switchable diffuser, such as described above with respect to Figure 1 The emitter 910 and the sensor 945 may be respectively Figure 1101 and one or more sensors 102. The hybrid depth sensing system 900 includes an emitter 910 and a sensor 945. During the ToF mode, the emitter 910 may operate in a ToF projection mode and the sensor 945 may operate in a ToF sensing mode, such as with respect to Figure 8 Described. The hybrid depth sensing system 900 includes a DOE 920 and a diffuser 930 coupled to the front of the DOE 920. The diffuser 930 can be turned on during the ToF projection mode (e.g., for flood distribution). Specifically, during the ToF projection mode, the emitter 910 can send a pulse signal 911 toward an object 940 while the diffuser 930 diffuses the emitted light to project a flood distribution (e.g., uniform illumination) onto the scene. The reflected signal 941 can reach the sensor 945, and the sensor 945 can calculate the ToF depth information based on the amount of time that the light is reflected back to the sensor 945 for each pixel.

[0090] Fig. 9B An example hybrid depth sensing system 900 is shown operating in a second (eg, SL) mode. During the SL mode, the transmitter 910 may operate in an SL projection mode and the sensor 945 may operate in an SL sensing mode, such as with respect to Figure 8 As described above. Diffuser 930 may be disconnected during SL projection mode (e.g., used as a transparent glass sheet). Specifically, during SL projection mode, emitter 910 may project a sparse distribution (e.g., a DOE pattern) toward scene 950, which will pass through diffuser 930 relatively unaffected (as projection light 913) and reach scene 950 (e.g., as a dot matrix pattern). Reflected light 943 may reach sensor 945, and sensor 945 may calculate SL depth information Z(SL) based on how projection light 913 is distorted on scene 950. As shown relative to Figure 1 As described above, an active depth controller 110 such as Figure 1 The active depth controller 110 may use Z(SL) during the calculation of the ToF depth information Z(ToF+SL). In this way, the active depth controller 110 may reduce or eliminate multipath artifacts in the ToF depth information.

[0091] Fig. 9CAn example hybrid depth sensing system 900 operating in a third (e.g., point ToF) mode is shown. During the point ToF mode, the transmitter 910 may operate in a point ToF projection mode. The point ToF mode may combine aspects of the SL mode and the ToF mode. For example, the transmitter 910 may send a pulse signal 911 toward the scene 950. In some cases, the pulse signal 911 may match the pulse signal 911 used in the ToF mode. In some implementations, during the point ToF mode, the diffuser (e.g., used as a transparent glass sheet) may be disconnected during the point ToF mode. In some cases, during the point ToF mode, the transmitter 910 may project a sparse distribution 980 toward the scene 950. In some examples, the sparse distribution 980 may be pulsed based on the pulse signal 911. In some examples, the sparse distribution 980 may be the same as the SL mode distribution. As illustrated, the sparse distribution 980 of the point ToF may be different from the distribution of the SL mode. For example, the sparse distribution 980 does not require a structural (e.g., non-uniform) pattern. For example, pulse signal 961 and reflected signal 971 may be used to determine point ToF depth information similar to conventional ToF measurements. However, similar to SL-based depth measurements, point ToF depth information may be limited to portions of the scene illuminated by sparse distribution 980 and include gaps in the regions between points.

[0092] In some cases, the reflected signal 941 may reach the sensor 945, and the sensor 945 may calculate the point ToF depth information for each pixel based on the amount of time that the light is reflected back to the sensor 945. In some examples, the ToF depth may be calculated for the pixel position corresponding to the point of the point ToF illumination pattern. In some cases, light traveling through an indirect path (e.g., MPI) may appear in the point ToF depth information as background illumination. For example, diffuse light reflected from the scene 950 may appear in the background of the point (e.g., between the points) as well as on the point. In some cases, background illumination subtraction may be used to adjust the effect of MPI based on the point ToF depth information. In some cases, thresholding may be applied to obtain depth information only from the point. In some cases, based on the depth obtained from the point, background illumination (e.g., illumination between the points) may be subtracted from the point to reduce or eliminate MPI. In some implementations, thresholding and / or background subtraction may be applied locally (e.g., to a small group of pixels). In some aspects, thresholding and background subtraction may be applied globally. As an illustrative and non-limiting example, background illumination depth measurements between points may be averaged, and the average may be subtracted for all pixel values.

[0093] In some cases, different DOEs 920, different emitters 910, and / or different sensors 945 may be used for point ToF mode and SL mode. In some examples, two or more of the ToF mode, SL mode, and / or point ToF mode may share common components. For example, by switching between two corresponding illumination modes using an emitter 910 and a sensor 945 shared with a switchable diffuser 930, it may be advantageous to use point ToF depth information to correct the MPI in the ToF depth information. In some cases, the same pulse signal pattern may be used for both ToF depth measurement and point ToF depth measurement. Therefore, in some cases, the ToF depth information and the point ToF depth information may be aligned and scaled similarly. For example, in some cases, the amount of error (e.g., the amount of MPI in the ToF depth information) may be determined by directly subtracting the point ToF depth information from the ToF depth information. In contrast, the SL depth information may need to be aligned and / or calibrated before being used to determine the amount of error (e.g., the amount of MPI in the ToF depth information and / or the point ToF information). In some cases, such calibration may be required even when SL depth information is acquired using a shared transmitter 910 and sensor 945 with ToF depth information and / or point ToF depth information.

[0094] Fig. 10A 1 is a simplified illustration of a hybrid depth sensing system 1000 operating in a first (eg, ToF) mode. The hybrid depth sensing system 1000 may be an example implementation of the hybrid depth sensing system 900, such as with respect to 9A to 9C 10. Laser 1010 emits light through DOE 1020. In ToF mode, diffuser 1030 may be turned on (indicated by the hatched pattern), and the DOE distribution may be diffused when passing through diffuser 1030, thereby providing diffuse illumination to scene 1040. Laser 1010 may emit light having a pattern (e.g., a pulse pattern) that can be used to determine depth information.

[0095] Fig. 10B is operating in the second (e.g., SL) mode Fig. 10A 1 is a simplified illustration of a hybrid depth sensing system 1000. The hybrid depth sensing system 1000 may be an example implementation of the hybrid depth sensing system 900, such as with respect to Fig. 9B Laser 1010 emits light through DOE 1020. In SL mode, diffuser 1030 may be disconnected (indicated by the absence of a shading pattern), and the DOE distribution may be projected onto scene 1040 through diffuser 1030 (eg, unchanged).

[0096] Fig. 10C is operating in the second (e.g., point ToF) mode Fig. 10A1 is a simplified illustration of a hybrid depth sensing system 1000. The hybrid depth sensing system 1000 may be an example implementation of the hybrid depth sensing system 900, such as with respect to Fig. 9C 1020. In dot ToF mode, diffuser 1030 may be disconnected (indicated by the absence of a shadow pattern) and the DOE distribution may be projected onto scene 1040 through diffuser 1030 (e.g., unchanged). Laser 1010 may emit light having a pattern (e.g., a pulse pattern) that may be used to determine depth information. In some examples, the dot pattern may be as shown in FIG. Fig. 10C In some cases, the dot pattern may also be Fig. 10B The SL dot pattern illustrated is the same random dot pattern.

[0097] Fig.11 1 is a diagram illustrating an adjusted depth image based on a ToF depth image and a SL depth image in an experimental configuration. In the illustrated example, depth information of a scene 1105 of a corner and a beverage container is captured by a hybrid depth sensing system 1110. In the experiment, the hybrid depth sensing system 1110 captures ToF depth information in a ToF illumination mode and captures SL depth information in an SL illumination mode. The ToF depth information and the SL depth information are acquired using a shared emitter (e.g., 9A to 9C transmitter 910), shared sensors (e.g., 9A to 9C Sensors 945), DOE (e.g., 9A to 9C 920) and switchable diffusers (e.g. 9A to 9C As illustrated, the raw ToF image 1115, the ToF depth image 1130, and the ToF depth profile 1135 exhibit MPI near corners in the scene and around the edges of the beverage container. Also as illustrated, the raw SL image 1120, the SL depth image 1140, and the SL depth profile 1145 exhibit lower resolution than the ToF depth information, but the MPI has minimal impact on the SL depth information compared to the ToF depth information. By fusing the ToF depth information and the SL depth information (e.g., by Figure 1 110) to generate an adjusted depth image 1150 and an adjusted depth profile 1155. As illustrated, the adjusted depth image 1150 and the adjusted depth profile 1155 combine the low MPI of the SL information with the high resolution of the ToF image. As noted above, similar techniques may be used to adjust ToF depth information based on point ToF depth information and / or to adjust point ToF depth information based on SL depth information without departing from the scope of the present disclosure.

[0098] Fig.12 1 is a flow diagram illustrating an example of a process 1200 for processing one or more frames. At block 1202, the process 1200 includes obtaining a first depth image of a scene. In some cases, the first depth image of the scene is associated with a first illumination configuration. In some examples, the first illumination configuration includes illuminating the scene using a first type of illumination.

[0099] At box 1204, process 1200 includes obtaining a second depth image of the scene. In some cases, the second depth image is associated with a second illumination configuration that is different from the first illumination configuration. In some aspects, the second illumination configuration includes illuminating the scene with a second type of illumination. In some cases, the second type of illumination is different from the first type of illumination. In some implementations, the first depth image and the second depth image are based on time of flight (ToF) measurements. In some cases, the first illumination configuration includes a pulsed illumination pattern, and the second illumination configuration includes the pulsed illumination pattern. In some cases, the second illumination configuration includes a uniformly spaced dot pattern. In some cases, the second illumination configuration includes a non-uniform dot pattern (e.g., a SL dot pattern).

[0100] At block 1206 , process 1200 includes determining an MPI associated with the first depth image based on the second depth image.

[0101] At block 1208, process 1200 includes generating an adjusted depth image based on determining the MPI associated with the first depth image. In some examples, the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images that are different from the first depth image. In some cases, the one or more additional depth images include the second depth image.

[0102] In some examples, the one or more additional depth images include a third depth image of the scene. In some cases, the third depth image is associated with the second illumination configuration. In some examples, the second depth image is based on structured light measurements and the third depth image is based on time of flight (ToF) measurements.

[0103] In some implementations, the first depth image is based on ToF measurements and the second depth image is based on detecting a non-uniform illumination pattern associated with the second illumination configuration. In some examples, the second depth image is based on a structured light illumination configuration.

[0104] In some examples, process 1200 includes generating the adjusted depth image based on determining that the MPI associated with the first depth image includes determining that the numerical correspondence between the pixel of the first depth image and the corresponding pixel of the second depth image exceeds the MPI threshold. In some cases, process 1200 includes replacing the pixel of the first depth image with the adjusted depth pixel based on determining that the numerical correspondence between the pixel of the first depth image and the corresponding pixel of the second depth image exceeds the MPI threshold. In some aspects, the adjusted depth pixel includes the corresponding pixel of the second depth image. In some cases, the adjusted depth pixel includes additional corresponding pixels of a third depth image, which is different from the first depth image and the second depth image. In some examples, process 1200 includes determining the numerical correspondence between the pixel of the first depth image and the corresponding pixel of the second depth image based on at least one or more of moving window image fusion or fixed grid image fusion.

[0105] In some examples, process 1200 includes using an illumination source to transmit light along an optical axis; using a diffractive optical element (DOE) positioned relative to the optical axis to generate an illumination pattern for the light generated by the illumination source; and one of: using a switchable diffuser to diffuse the illumination pattern in an active mode; and using the diffuser to allow the illumination pattern to pass through in an inactive mode. In some examples, the first illumination configuration includes operating the switchable diffuser in the active mode to generate the first type of illumination. In some examples, the second illumination configuration includes operating the switchable diffuser in the inactive mode to generate the second type of illumination.

[0106] In some examples, the processes described herein (e.g., process 1200 and / or other processes described herein) can be performed by a computing device or apparatus. In one example, one or more of these processes can be performed by Figure 1 In another example, one or more of these processes may be performed by Fig.13 The computing system 1300 shown executes. For example, Fig.13 The computing device of the computing system 1300 shown may include components of the hybrid depth sensing system 100 and may implement Fig.12 Operations of process 1200 and / or other processes described herein.

[0107] The computing device may include any suitable device, such as a vehicle or a computing device of a vehicle (e.g., a driver monitoring system (DMS) of a vehicle), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smart watch, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein (including process 1200 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0108] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0109] Process 1200 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, and data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0110] Additionally, process 1200 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0111] Fig.13 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.13 An example of a computing system 1300 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using connection 1305. Connection 1305 may be a physical connection using a bus, or a direct connection into processor 1310, such as in a chipset architecture. Connection 1305 may also be a virtual connection, a networked connection, or a logical connection.

[0112] In some embodiments, computing system 1300 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component may be a physical device or a virtual device.

[0113] The example system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that couple various system components including system memory 1315, such as read only memory (ROM) 1320 and random access memory (RAM) 1325, to the processor 1310. The computing system 1300 may include a cache 1312 of high-speed memory directly connected to, close to, or integrated as part of the processor 1310.

[0114] Processor 1310 may include any general purpose processor and hardware or software services, such as services 1332, 1334, and 1336 stored in storage device 1330, that are configured to control processor 1310 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1310 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0115] To enable user interaction, the computing system 1300 includes an input device 1345 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1300 may also include an output device 1335 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1300. The computing system 1300 may include a communication interface 1340, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple ® Lightning ® Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low energy (BLE) wireless signal transmission, IBEACON ® Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1340 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 1300 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0116] The storage device 1330 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick ® card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0117] Storage device 1330 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1310, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to the necessary hardware components (such as processor 1310, connection 1305, output device 1335, etc.).

[0118] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media that can store data and does not include carrier waves and / or transient electronic signals that propagate wirelessly or over a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have stored thereon code and / or machine-executable instructions that may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0119] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0120] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these embodiments in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing each embodiment.

[0121] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0122] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that enable or configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions in other ways. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0123] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptops, mobile phones (e.g., smart phones or other types of mobile phones), tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0124] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0125] In the foregoing description, various aspects of the present application are described with reference to their specific embodiments, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the scope of this specification, the embodiments can be used in any number of environments and applications beyond the environments and applications described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0126] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols without departing from the scope of the present specification.

[0127] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0128] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (eg, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0129] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0130] Various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on specific applications and the design constraints proposed to the entire system. Technicians can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the application.

[0131] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that, when executed, perform one or more of the above methods. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, and a magnetic or optical data storage medium, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0132] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0133] Illustrative aspects of the present disclosure include:

[0134] Aspect 1. A device for depth sensing, the device comprising: a memory; and at least one processor, the at least one processor coupled to the memory, the at least one processor configured to: obtain a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration includes illuminating the scene with a first type of illumination; obtain a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, the second illumination configuration is different from the first illumination configuration, wherein the second illumination configuration includes illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determine multipath interference (MPI) associated with the first depth image based on the second depth image; and generate an adjusted depth image based on determining the MPI associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0135] Aspect 2. The apparatus according to aspect 1, wherein the one or more additional depth images include the second depth image.

[0136] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the one or more additional depth images include a third depth image of the scene, wherein the third depth image is associated with the second illumination configuration, wherein the second depth image is based on structured light measurement, and the third depth image is based on time of flight (ToF) measurement.

[0137] Aspect 4. The apparatus according to any one of aspects 1 to 3, wherein the first depth image and the second depth image are based on time-of-flight (ToF) measurements.

[0138] Aspect 5. The apparatus according to any one of aspects 1 to 4, wherein the first illumination configuration comprises a pulsed illumination pattern, and the second illumination configuration comprises the pulsed illumination pattern.

[0139] Aspect 6. The device according to any one of aspects 1 to 5, wherein the second illumination configuration comprises a uniformly spaced dot pattern.

[0140] Aspect 7. The device according to any one of aspects 1 to 6, wherein the second illumination configuration comprises a non-uniform dot pattern.

[0141] Aspect 8. An apparatus according to any one of aspects 1 to 7, wherein the first depth image is based on ToF measurements and the second depth image is based on detecting a non-uniform illumination pattern associated with the second illumination configuration.

[0142] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein the second depth image is based on a structured light illumination configuration.

[0143] Aspect 10. An apparatus according to any one of Aspects 1 to 9, wherein, in order to generate an adjusted depth image based on determining the MPI associated with the first depth image, the at least one processor is configured to: determine that a numerical correspondence between a pixel of the first depth image and a corresponding pixel of the second depth image exceeds an MPI threshold; and based on determining that the numerical correspondence between the pixel of the first depth image and the corresponding pixel of the second depth image exceeds the MPI threshold, replace the pixel of the first depth image with the adjusted depth pixel.

[0144] Clause 11. The apparatus according to any one of Clauses 1 to 10, wherein the adjusted depth pixel comprises the corresponding pixel of the second depth image.

[0145] Clause 12. The apparatus according to any one of Clauses 1 to 11, wherein the adjusted depth pixels include additional corresponding pixels of a third depth image, the third depth image being different from the first depth image and the second depth image.

[0146] Aspect 13. An apparatus according to any one of Aspects 1 to 12, wherein determining the numerical correspondence between the pixels of the first depth image and the corresponding pixels of the second depth image includes at least one or more of moving window image fusion or fixed grid image fusion.

[0147] Aspect 14. An apparatus according to any one of Aspects 1 to 13, wherein the apparatus further comprises: an illumination source configured to transmit light along an optical axis; a diffractive optical element (DOE) positioned relative to the optical axis to generate an illumination pattern for the light generated by the illumination source; and a switchable diffuser configured to: diffuse the illumination pattern in an active mode; and allow the illumination pattern to pass through in an inactive mode.

[0148] Aspect 15. An apparatus according to Aspect 14, wherein the first illumination configuration includes operating the switchable diffuser in the active mode to generate the first type of illumination, and the second illumination configuration includes operating the switchable diffuser in the inactive mode to generate the second type of illumination.

[0149] Aspect 16. A method for depth sensing, the method comprising: obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration comprises illuminating the scene with a first type of illumination; obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, the second illumination configuration being different from the first illumination configuration, wherein the second illumination configuration comprises illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determining multipath interference (MPI) associated with the first depth image based on the second depth image; and generating an adjusted depth image based on determining the MPI associated with the first depth image, wherein the adjusted depth image comprises one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0150] Aspect 17. The method according to aspect 16, wherein the one or more additional depth images include the second depth image.

[0151] Aspect 18. An apparatus according to any one of Aspects 16 to 17, wherein the one or more additional depth images include a third depth image of the scene, wherein the third depth image is associated with the second illumination configuration, wherein the second depth image is based on structured light measurement, and the third depth image is based on time of flight (ToF) measurement.

[0152] Aspect 19. The apparatus according to any one of aspects 16 to 18, wherein the first depth image and the second depth image are based on time of flight (ToF) measurements.

[0153] Aspect 20. The apparatus according to any one of aspects 16 to 19, wherein the first illumination configuration comprises a pulsed illumination pattern, and the second illumination configuration comprises the pulsed illumination pattern.

[0154] Aspect 21. The device according to any one of aspects 16 to 20, wherein the second illumination configuration comprises a uniformly spaced dot pattern.

[0155] Aspect 22. The device according to any one of aspects 16 to 21, wherein the second illumination configuration comprises a non-uniform dot pattern.

[0156] Aspect 23. An apparatus according to any one of aspects 16 to 22, wherein the first depth image is based on ToF measurements and the second depth image is based on detecting a non-uniform illumination pattern associated with the second illumination configuration.

[0157] Clause 24. An apparatus according to any one of Clauses 16 to 23, wherein the second depth image is based on a structured light illumination configuration.

[0158] Aspect 25. An apparatus according to any one of Aspects 16 to 24, wherein, in order to generate an adjusted depth image based on determining the MPI associated with the first depth image, the at least one processor is configured to: determine that a numerical correspondence between a pixel of the first depth image and a corresponding pixel of the second depth image exceeds an MPI threshold; and based on determining that the numerical correspondence between the pixel of the first depth image and the corresponding pixel of the second depth image exceeds the MPI threshold, replace the pixel of the first depth image with the adjusted depth pixel.

[0159] Clause 26. The apparatus according to any one of Clauses 16 to 25, wherein the adjusted depth pixel comprises the corresponding pixel of the second depth image.

[0160] Clause 27. An apparatus according to any one of Clauses 16 to 26, wherein the adjusted depth pixels include additional corresponding pixels of a third depth image, the third depth image being different from the first depth image and the second depth image.

[0161] Aspect 28. An apparatus according to any one of Aspects 16 to 27, wherein determining the numerical correspondence between the pixels of the first depth image and the corresponding pixels of the second depth image includes at least one or more of moving window image fusion or fixed grid image fusion.

[0162] Aspect 29. An apparatus according to any one of Aspects 16 to 28, wherein the apparatus further comprises: an illumination source configured to transmit light along an optical axis; a diffractive optical element (DOE) positioned relative to the optical axis to generate an illumination pattern for the light generated by the illumination source; and a switchable diffuser configured to: diffuse the illumination pattern in an active mode; and allow the illumination pattern to pass through in an inactive mode.

[0163] Aspect 30. An apparatus according to Aspect 29, wherein the first illumination configuration includes operating the switchable diffuser in the active mode to generate the first type of illumination, and the second illumination configuration includes operating the switchable diffuser in the inactive mode to generate the second type of illumination.

[0164] Aspect 31: A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations described in aspects 1 to 30.

[0165] Aspect 32: An apparatus comprising means for performing any of the operations described in aspects 1 to 30.

[0166] Aspect 33: A device for depth sensing, the device comprising: a memory; and at least one processor, the at least one processor coupled to the memory, the at least one processor configured to: obtain a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration comprises illuminating the scene with a first illumination pattern based on time-of-flight measurement; obtain a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, the second illumination configuration is different from the first illumination configuration, wherein the second illumination configuration comprises illuminating the scene with a second illumination pattern, and the second depth image is based on structured light depth measurement; determine multipath interference (MPI) associated with the first depth image based on the second depth image; and generate an adjusted depth image based on determining the MPI associated with the first depth image, wherein the adjusted depth image comprises one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0167] Aspect 34. A method, the method comprising: obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration comprises illuminating the scene with a first type of illumination; obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration, the second illumination configuration being different from the first illumination configuration, wherein the second illumination configuration comprises illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determining multipath interference (MPI) associated with the first depth image based on the second depth image; and generating an adjusted depth image based on determining the MPI associated with the first depth image, wherein the adjusted depth image comprises one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

[0168] Aspect 35: A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations described in aspects 1 to 30.

[0169] Aspect 36: An apparatus comprising means for performing any of the operations described in aspects 1 to 30.

[0170] Aspect 37: A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations described in aspects 1 to 30 and aspects 33 to 34.

[0171] Aspect 38: An apparatus comprising means for performing any of the operations described in aspects 1 to 15 and aspect 33.

[0172] Aspect 39: A method comprising any of the operations described in aspects 1 to 30 and aspects 33 to 34.

Claims

1. A device for depth sensing, the device comprising: Memory; and at least one processor, the at least one processor coupled to the memory, the at least one processor configured to: obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration comprises illuminating the scene with a first type of illumination; obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration that is different from the first illumination configuration, wherein the second illumination configuration comprises illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determining a multipath interference (MPI) associated with the first depth image based on the second depth image; as well as An adjusted depth image is generated based on determining the MPI associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image. 2 . The apparatus of claim 1 , wherein the one or more additional depth images include the second depth image.

3. The apparatus of claim 1 , wherein the one or more additional depth images include a third depth image of the scene, wherein the third depth image is associated with the second illumination configuration, wherein the second depth image is based on structured light measurements and the third depth image is based on time of flight (ToF) measurements. The apparatus of claim 1 , wherein the first depth image and the second depth image are based on time-of-flight (ToF) measurements.

5. The apparatus of claim 4, wherein the first illumination configuration comprises a pulsed illumination pattern and the second illumination configuration comprises the pulsed illumination pattern. The apparatus of claim 4 , wherein the second illumination configuration comprises a uniformly spaced dot pattern.

7. The apparatus of claim 4, wherein the second illumination configuration comprises a non-uniform dot pattern.

8. The apparatus of claim 1, wherein the first depth image is based on ToF measurements and the second depth image is based on detecting a non-uniform illumination pattern associated with the second illumination configuration.

9. The apparatus of claim 8, wherein the second depth image is based on a structured light illumination configuration.

10. The device according to claim 1, wherein: To generate the adjusted depth image based on determining the MPI associated with the first depth image, the at least one processor is configured to: Determining that a numerical correspondence between a pixel of the first depth image and a corresponding pixel of the second depth image exceeds an MPI threshold; as well as Based on determining that the numerical correspondence between the pixel of the first depth image and the corresponding pixel of the second depth image exceeds the MPI threshold, replacing the pixel of the first depth image with an adjusted depth pixel. The device of claim 10 , wherein the adjusted depth pixel comprises the corresponding pixel of the second depth image. 12 . The device of claim 10 , wherein the adjusted depth pixels include additional corresponding pixels of a third depth image, the third depth image being different from the first depth image and the second depth image.

13. The device of claim 10, wherein the at least one processor is configured to determine the numerical correspondence between the pixels of the first depth image and the corresponding pixels of the second depth image based on at least one or more of moving window image fusion or fixed grid image fusion.

14. The apparatus according to claim 1, further comprising: an illumination source configured to transmit light along an optical axis; a diffractive optical element (DOE) positioned relative to the optical axis to generate an illumination pattern for the light generated by the illumination source; and A switchable diffuser, the switchable diffuser being configured to: diffusing the illumination pattern in an active mode; as well as The illumination pattern is allowed to pass through in the inactive mode.

15. The apparatus of claim 14, wherein the first illumination configuration comprises operating the switchable diffuser in the active mode to generate the first type of illumination, and the second illumination configuration comprises operating the switchable diffuser in the inactive mode to generate the second type of illumination.

16. A method for depth sensing, the method comprising: obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration comprises illuminating the scene with a first type of illumination; obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration that is different from the first illumination configuration, wherein the second illumination configuration comprises illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determining a multipath interference (MPI) associated with the first depth image based on the second depth image; as well as An adjusted depth image is generated based on determining the MPI associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image. The method of claim 16 , wherein the one or more additional depth images include the second depth image.

18. The method of claim 16, wherein the one or more additional depth images include a third depth image of the scene, wherein the third depth image is associated with the second illumination configuration, wherein the second depth image is based on structured light measurements and the third depth image is based on time of flight (ToF) measurements.

19. The method of claim 16, wherein the first depth image and the second depth image are based on time of flight (ToF) measurements.

20. The method of claim 19, wherein the first illumination configuration comprises a pulsed illumination pattern and the second illumination configuration comprises the pulsed illumination pattern.

21. The method of claim 19, wherein the second illumination configuration comprises a uniformly spaced dot pattern.

22. The method of claim 19, wherein the second illumination configuration comprises a non-uniform dot pattern.

23. The method of claim 16, wherein the first depth image is based on ToF measurements and the second depth image is based on detecting a non-uniform illumination pattern associated with the second illumination configuration. The method of claim 23 , wherein the second depth image is based on a structured light illumination configuration.

25. The method of claim 16, wherein generating the adjusted depth image based on determining the MPI associated with the first depth image comprises: Determining that a numerical correspondence between a pixel of the first depth image and a corresponding pixel of the second depth image exceeds an MPI threshold; as well as Based on determining that the numerical correspondence between the pixel of the first depth image and the corresponding pixel of the second depth image exceeds the MPI threshold, replacing the pixel of the first depth image with an adjusted depth pixel.

26. The method of claim 25, wherein the adjusted depth pixels comprise the corresponding pixels of the second depth image. 27 . The method of claim 25 , wherein the adjusted depth pixels include additional corresponding pixels of a third depth image, the third depth image being different from the first depth image and the second depth image.

28. The method of claim 25, wherein determining the numerical correspondence between the pixels of the first depth image and the corresponding pixels of the second depth image is based on at least one or more of moving window image fusion or fixed grid image fusion.

29. The method according to claim 16, further comprising: Using an illumination source to transmit light along an optical axis; generating an illumination pattern for the light generated by the illumination source using a diffractive optical element (DOE) positioned relative to the optical axis; as well as One of the following: using a switchable diffuser to diffuse the illumination pattern in an active mode; as well as The switchable diffuser is used to allow the illumination pattern to pass through in an inactive mode.

30. The method of claim 29, wherein the first illumination configuration comprises operating the switchable diffuser in the active mode to generate the first type of illumination, and the second illumination configuration comprises operating the switchable diffuser in the inactive mode to generate the second type of illumination.

31. An apparatus for depth sensing, the apparatus comprising: Memory; and at least one processor, the at least one processor coupled to the memory, the at least one processor configured to: obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration comprises illuminating the scene with a first illumination pattern based on time-of-flight measurements; obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration that is different from the first illumination configuration, wherein the second illumination configuration includes illuminating the scene with a second illumination pattern, and the second depth image is based on structured light depth measurements; determining a multipath interference (MPI) associated with the first depth image based on the second depth image; as well as An adjusted depth image is generated based on determining the MPI associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

32. A method for depth sensing, the method comprising: obtaining a first depth image of a scene, wherein the first depth image of the scene is associated with a first illumination configuration, wherein the first illumination configuration comprises illuminating the scene with a first type of illumination; obtaining a second depth image of the scene, wherein the second depth image is associated with a second illumination configuration that is different from the first illumination configuration, wherein the second illumination configuration comprises illuminating the scene with a second type of illumination, wherein the second type of illumination is different from the first type of illumination; determining a multipath interference (MPI) associated with the first depth image based on the second depth image; as well as An adjusted depth image is generated based on determining the MPI associated with the first depth image, wherein the adjusted depth image includes one or more pixels from the first depth image and one or more adjusted pixels based on one or more additional depth images, the one or more additional depth images being different from the first depth image.

Images