Alternating light distribution for active depth sensing

Abstract

Aspects of the present disclosure relate to systems and methods for active depth sensing. An exemplary device configured to perform active depth sensing includes a projector. The projector is configured to emit a first light distribution during a first time and to emit a second light distribution different from the first light distribution during a second time. A final set of depth values ​​for one or more objects in a scene is based on one or more reflections of the first light distribution and one or more reflections of the second light distribution. The projector may include a laser array, and the device may be configured to switch between a first plurality of lasers in the laser array emitting light during the first time and a second plurality of lasers emitting light during the second time.

Description

[0001] Related Applications

[0002] This patent application claims the priority of Greek patent application No. 20190100583, entitled “ALTERNATING LIGHT DISTRIBUTIONS FOR ACTIVE DEPTH SENSING”, filed on December 30, 2019, in the Athens Industrial Property Organization. The Greek patent application is assigned to the present assignee. The disclosure of the prior application is considered part of and incorporated by reference into the present patent application. Technical Field

[0003] The present disclosure relates generally to active depth sensing systems, and particularly to a projector that alternately emits different light distributions for active depth sensing. Background Art

[0004] For active depth sensing, the device may include a projector for emitting a light distribution, for which reflections of the light distribution are sensed and measured to determine the distance of an object in the scene. For example, the device may include a light projector that projects a distribution of infrared (IR) light (such as a distribution of IR light spots) onto a scene. An active light receiver receives reflections of the infrared light during frame capture, and the device determines the depth or distance of an object in the scene based on the received reflections. Summary of the invention

[0005] The present summary is provided to introduce a selection of some concepts in a simplified form, which concepts are further described in the following detailed description. The present summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] Some aspects of the present disclosure relate to an exemplary apparatus for active depth sensing. The apparatus includes a projector configured to emit a first light distribution during a first time and to emit a second light distribution different from the first light distribution during a second time. A final set of depth values ​​for one or more objects in a scene, such as for a final depth map, is based on one or more reflections of the first light distribution and one or more reflections of the second light distribution.

[0007] In some embodiments, the projector is further configured to emit a third light distribution that is different from the first light distribution and the second light distribution during a third time. The final set of depth values ​​for the one or more objects in the scene is further based on one or more reflections of the third light distribution. In some embodiments, the projector includes a laser array, and for each laser in the laser array, the laser emits light during one or more of the first time or the second time. In some embodiments, for each laser in the laser array, the laser emits light during one of the first time or the second time. The projector may include a switch for switching between a first plurality of lasers in the laser array emitting light during the first time and a second plurality of lasers in the laser array emitting light during the second time.

[0008] In some embodiments, the apparatus further comprises an image sensor configured to receive one or more reflections of the first light distribution during capture of a first frame associated with the first time, and to receive one or more reflections of the second light distribution during capture of a second frame associated with the second time. The apparatus may further comprise an image signal processor configured to generate a first set of depth values ​​based on the first frame, generate a second set of depth values ​​based on the second frame, and combine the first set of depth values ​​and the second set of depth values ​​to generate the final set of depth values.

[0009] In some embodiments, the projector is configured to periodically alternate between emitting the first light distribution and emitting the second light distribution. Each emission of the first light distribution is associated with the capture of one of the first frame set by the image sensor, and each emission of the second light distribution is associated with the capture of one of the second frame set by the image sensor. In some embodiments, the image signal processor is configured to generate the first depth value set based on the first frame set, and to generate the second depth value set based on the second frame set. In some embodiments, the first depth value set is included in a first depth map, and the second depth value set is included in a second depth map. A first position in the first depth map of a first depth value from the first depth value set corresponds to a second position in the second depth map of a second depth value from the second depth value set. Generating the final depth value set includes: determining whether the first depth value is a valid depth value; determining whether the second depth value is a valid depth value; and setting a final depth value in the final depth value set to the valid depth value based on only one of the first depth value or the second depth value being determined to be a valid depth value. For example, the first depth value set is included in a first depth map of size P x ​​Q, and each position pxq in the first depth map includes a depth value from the first depth value set. The second depth value set may be included in a second depth map of size P x ​​Q, and each position pxq in the second depth map includes a depth value from the second depth value set and corresponds to the position pxq in the first depth map. In some embodiments, generating the final depth value set includes generating a final depth map of size P x ​​Q, and for each position pxq of the final depth map, determining whether the depth value at the position pxq from the first depth map is a valid depth value, determining whether the depth value at the position pxq from the second depth map is a valid depth value, and setting the final depth value at the position pxq in the final depth map to the valid depth value based on only one of the depth values ​​being determined to be a valid depth value.

[0010] In some embodiments, generating the final depth value set further includes setting the final depth value to one of the first depth value, the second depth value, or an average of the first depth value and the second depth value based on both the first depth value and the second depth value being determined as valid depth values. For example, generating the final depth map of size P x ​​Q further includes setting the final depth value at position pxq in the final depth map to one of the valid depth values ​​in the first depth map, the valid depth values ​​in the second depth map, or an average of the valid depth values ​​in the first depth map and the valid depth values ​​in the second depth map based on the two depth values ​​being determined as valid depth values.

[0011] In some embodiments, the apparatus further comprises: a processor configured to provide instructions to the image signal processor for execution; a memory configured to store the final depth map; and a display configured to display the final depth map.

[0012] In some embodiments, the projector is configured to alternate between emitting the first light distribution and emitting the second light distribution. The first light distribution is an original light distribution, the second light distribution is a dual light distribution, and the original light distribution and the dual light distribution are designed with reference to each other to allow the same portion of a frame captured by an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution.

[0013] In some embodiments, the first light distribution and the second light distribution are determined in one or more spatial dimensions, time dimensions, and array dimensions based on one or more dimensional parameters.

[0014] Some other aspects of the present disclosure relate to a method for active depth sensing. An exemplary method includes emitting, by a projector, a first light distribution during a first time. The method also includes emitting, by the projector, a second light distribution different from the first light distribution during a second time. A final set of depth values ​​for one or more objects in a scene (such as for a final depth map) is based on one or more reflections of the first light distribution and one or more reflections of the second light distribution.

[0015] In some embodiments, the method further includes emitting, by the projector, a third light distribution different from the first light distribution and the second light distribution during a third time. The final set of depth values ​​for the one or more objects in the scene is further based on one or more reflections of the third light distribution. In some embodiments, the method further includes emitting, by each laser in the laser array of the projector, light during one or more of the first time or the second time. In some embodiments, the method further includes emitting, by each laser in the laser array, light during one of the first time or the second time. In some embodiments, the method further includes switching between emitting light by a first plurality of lasers in the laser array during the first time and emitting light by a second plurality of lasers in the laser array during the second time.

[0016] In some embodiments, the method also includes: receiving, by the image sensor, one or more reflections of the first light distribution during capture of a first frame associated with the first time; receiving, by the image sensor, one or more reflections of the second light distribution during capture of a second frame associated with the second time; generating, by the image signal processor, a first set of depth values ​​based on the first frame; generating, by the image signal processor, a second set of depth values ​​based on the second frame; and combining the first set of depth values ​​and the second set of depth values ​​to generate the final set of depth values.

[0017] In some embodiments, the method further comprises periodically alternating between emitting the first light distribution and emitting the second light distribution. Each emission of the first light distribution is associated with the capture of one of the first set of frames by the image sensor, and each emission of the second light distribution is associated with the capture of one of the second set of frames by the image sensor. The method further comprises generating the first set of depth values ​​based on the first set of frames, and generating the second set of depth values ​​based on the second set of frames.

[0018] In some embodiments, the first depth value set is included in a first depth map, and the second depth value set is included in a second depth map. A first position in the first depth map of a first depth value from the first depth value set corresponds to a second position in the second depth map of a second depth value from the second depth value set. Generating the final depth value set includes: determining whether the first depth value is a valid depth value; determining whether the second depth value is a valid depth value; and setting a final depth value in the final depth value set to the valid depth value based on only one of the first depth value or the second depth value being determined to be a valid depth value. For example, the first depth value set is included in a first depth map of size P x ​​Q, and each position pxq in the first depth map includes a depth value from the first depth value set. The second depth value set may be included in a second depth map of size P x ​​Q, and each position pxq in the second depth map includes a depth value from the second depth value set and corresponds to the position pxq in the first depth map. In some embodiments, generating the final set of depth values ​​includes generating a final depth map of size P x ​​Q, and for each position pxq of the final depth map, including determining whether the depth value at the position pxq from the first depth map is a valid depth value, determining whether the depth value at the position pxq from the second depth map is a valid depth value, and based on only one of the depth values ​​being determined to be a valid depth value, setting the final depth value at the position pxq in the final depth map to the valid depth value.

[0019] In some embodiments, generating the final depth value set further includes setting the final depth value to one of the first depth value, the second depth value, or an average of the first depth value and the second depth value based on both the first depth value and the second depth value being determined as valid depth values. For example, generating the final depth map of size P x ​​Q further includes setting the final depth value at position pxq in the final depth map to one of the valid depth values ​​in the first depth map, the valid depth values ​​in the second depth map, or an average of the valid depth values ​​in the first depth map and the valid depth values ​​in the second depth map based on the two depth values ​​being determined as valid depth values.

[0020] In some embodiments, the projector is configured to alternate between emitting the first light distribution and emitting the second light distribution. The first light distribution is an original light distribution, the second light distribution is a dual light distribution, and the original light distribution and the dual light distribution are designed with reference to each other to allow the same portion of a frame captured by an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution.

[0021] In some embodiments, the first light distribution and the second light distribution are determined in one or more spatial dimensions, time dimensions, and array dimensions based on one or more dimensional parameters.

[0022] Some other aspects of the present disclosure relate to an exemplary computer-readable medium. The computer-readable medium stores instructions that, when executed by a processor of a device performing active depth sensing, cause the device to emit a first light distribution by a projector during a first time. Execution of the instructions also causes the device to emit a second light distribution by the projector during a second time that is different from the first light distribution. A final set of depth values ​​for one or more objects in a scene, such as for a final depth map, is based on one or more reflections of the first light distribution and one or more reflections of the second light distribution.

[0023] In some embodiments, execution of the instructions further causes the device to emit a third light distribution from the projector during a third time that is different from the first light distribution and the second light distribution. The final set of depth values ​​for the one or more objects in the scene is further based on one or more reflections of the third light distribution. In some embodiments, execution of the instructions further causes the device to switch between emitting light from a first plurality of lasers in the laser array of the projector during the first time and emitting light from a second plurality of lasers in the laser array during the second time.

[0024] In some embodiments, execution of the instructions also causes the device to: receive, by the image sensor, one or more reflections of the first light distribution during capture of a first frame associated with the first time; receive, by the image sensor, one or more reflections of the second light distribution during capture of a second frame associated with the second time; generate, by the image signal processor, a first set of depth values ​​based on the first frame; generate, by the image signal processor, a second set of depth values ​​based on the second frame; and combine the first set of depth values ​​and the second set of depth values ​​to generate the final set of depth values.

[0025] In some embodiments, execution of the instructions further causes the device to periodically alternate between emitting the first light distribution and emitting the second light distribution. Each emission of the first light distribution is associated with the capture of one of the first set of frames by the image sensor. Each emission of the second light distribution is associated with the capture of one of the second set of frames by the image sensor. Execution of the instructions further causes the device to generate the first set of depth values ​​based on the first set of frames, and to generate the second set of depth values ​​based on the second set of frames.

[0026] In some embodiments, the first depth value set is included in a first depth map, and the second depth value set is included in a second depth map. A first position in the first depth map of a first depth value from the first depth value set corresponds to a second position in the second depth map of a second depth value from the second depth value set. Generating the final depth value set includes: determining whether the first depth value is a valid depth value; determining whether the second depth value is a valid depth value; and setting a final depth value in the final depth value set to the valid depth value based on only one of the first depth value or the second depth value being determined to be a valid depth value. For example, the first depth value set is included in a first depth map of size P x ​​Q, and each position pxq in the first depth map includes a depth value from the first depth value set. The second depth value set may be included in a second depth map of size P x ​​Q, and each position pxq in the second depth map includes a depth value from the second depth value set and corresponds to the position pxq in the first depth map. In some embodiments, generating the final set of depth values ​​includes generating a final depth map of size P x ​​Q, and for each position pxq of the final depth map, including determining whether the depth value at the position pxq from the first depth map is a valid depth value, determining whether the depth value at the position pxq from the second depth map is a valid depth value, and based on only one of the depth values ​​being determined to be a valid depth value, setting the final depth value at the position pxq in the final depth map to the valid depth value.

[0027] In some embodiments, generating the final depth value set further includes setting the final depth value to one of the first depth value, the second depth value, or an average of the first depth value and the second depth value based on both the first depth value and the second depth value being determined as valid depth values. For example, generating the final depth map of size P x ​​Q further includes setting the final depth value at position pxq in the final depth map to one of the valid depth values ​​in the first depth map, the valid depth values ​​in the second depth map, or an average of the valid depth values ​​in the first depth map and the valid depth values ​​in the second depth map based on the two depth values ​​being determined as valid depth values.

[0028] In some embodiments, execution of the instructions further causes the device to alternate between emitting the first light distribution by the projector and emitting the second light distribution, the first light distribution being an original light distribution, the second light distribution being a dual light distribution, and the original light distribution and the dual light distribution being designed with reference to each other to allow the same portion of a frame capture of an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution.

[0029] In some embodiments, the first light distribution and the second light distribution are determined in one or more spatial dimensions, time dimensions, and array dimensions based on one or more dimensional parameters.

[0030] Some other aspects of the present disclosure relate to an exemplary device for performing active depth sensing. The device includes components for emitting a first light distribution during a first time. The device includes components for emitting a second light distribution different from the first light distribution during a second time. A final set of depth values ​​for one or more objects in a scene, such as for a final depth map, is based on one or more reflections of the first light distribution and one or more reflections of the second light distribution.

[0031] In some embodiments, the device further comprises means for emitting a third light distribution different from the first light distribution and the second light distribution during a third time. The final set of depth values ​​for the one or more objects in the scene is further based on one or more reflections of the third light distribution. In some embodiments, the device further comprises means for switching between emitting light by a first plurality of lasers in a laser array during the first time and emitting light by a second plurality of lasers in the laser array during the second time. In some embodiments, the device further comprises: means for receiving one or more reflections of the first light distribution during the capture of a first frame associated with the first time; means for receiving one or more reflections of the second light distribution during the capture of a second frame associated with the second time; means for generating a first set of depth values ​​based on the first frame; means for generating a second set of depth values ​​based on the second frame; and means for combining the first set of depth values ​​and the second set of depth values ​​to generate the final set of depth values.

[0032] In some embodiments, the first depth value set is included in a first depth map, and the second depth value set is included in a second depth map. A first position in the first depth map of a first depth value from the first depth value set corresponds to a second position in the second depth map of a second depth value from the second depth value set. Generating the final depth value set includes: determining whether the first depth value is a valid depth value; determining whether the second depth value is a valid depth value; and setting a final depth value in the final depth value set to the valid depth value based on only one of the first depth value or the second depth value being determined to be a valid depth value. For example, the first depth value set is included in a first depth map of size P x ​​Q, and each position pxq in the first depth map includes a depth value from the first depth value set. The second depth value set may be included in a second depth map of size P x ​​Q, and each position pxq in the second depth map includes a depth value from the second depth value set and corresponds to the position pxq in the first depth map. In some embodiments, generating the final set of depth values ​​includes generating a final depth map of size P x ​​Q, and for each position pxq of the final depth map, including determining whether the depth value at the position pxq from the first depth map is a valid depth value, determining whether the depth value at the position pxq from the second depth map is a valid depth value, and based on only one of the depth values ​​being determined to be a valid depth value, setting the final depth value at the position pxq in the final depth map to the valid depth value.

[0033] In some embodiments, generating the final depth value set further includes setting the final depth value to one of the first depth value, the second depth value, or an average of the first depth value and the second depth value based on both the first depth value and the second depth value being determined as valid depth values. For example, generating the final depth map of size P x ​​Q further includes setting the final depth value at position pxq in the final depth map to one of the valid depth values ​​in the first depth map, the valid depth values ​​in the second depth map, or an average of the valid depth values ​​in the first depth map and the valid depth values ​​in the second depth map based on the two depth values ​​being determined as valid depth values.

[0034] In some embodiments, the device further comprises means for periodically alternating between emitting the first light distribution and emitting the second light distribution. Each emission of the first light distribution is associated with the capture of one frame in the first set of frames, and each emission of the second light distribution is associated with the capture of one frame in the second set of frames. The device further comprises means for generating the first set of depth values ​​based on the first set of frames and means for generating the second set of depth values ​​based on the second set of frames.

[0035] The device may be configured to alternate between emitting the first light distribution and emitting the second light distribution. The first light distribution is an original light distribution. The second light distribution is a dual light distribution. The original light distribution and the dual light distribution are designed with reference to each other to allow the same portion of a frame capture of an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution.

[0036] In some embodiments, the first light distribution and the second light distribution are determined in one or more spatial dimensions, time dimensions, and array dimensions based on one or more dimensional parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Various aspects of the disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings and in which like references refer to similar elements.

[0038] Figure 1 is an illustration of an exemplary active depth sensing system including a projector for emitting a light distribution.

[0039] Figure 2A is an illustration of an exemplary depth map generated based on the distribution of light projected by a projector of an active depth sensing system.

[0040] Figure 2Bis an illustration of an exemplary depth map generated based on different light distributions projected by a projector of an active depth sensing system.

[0041] Figure 3 is a block diagram of an exemplary device including a projector configured to emit different light distributions for active depth sensing.

[0042] Figure 4 is an illustration of an exemplary laser array including lasers configured to emit a first light distribution and one or more other light distributions.

[0043] Figure 5 is an illustrative flow chart depicting exemplary operation of a projector for active depth sensing.

[0044] Figure 6 is an illustrative flow chart depicting example operations of active depth sensing based on a projector alternating between emitting a first light distribution and emitting a second light distribution.

[0045] Figure 7 is a diagram of an exemplary alternating emission between a first light distribution and a second light distribution and a resulting depth map.

[0046] Figure 8 is an illustrative flow chart depicting exemplary operations of combining a first set of depth values ​​(such as for a first depth map) and a second set of depth values ​​(such as for a second depth map) to generate a final set of depth values ​​(such as for a final depth map).

[0047] Fig. 9 is an illustration of an exemplary first depth map and an exemplary second depth map showing that depth values ​​will be combined to generate a final set of depth values.

[0048] Fig.10 is an illustrative flow chart depicting example operations for optimizing projected light distribution for active depth sensing.

[0049] Fig.11 is an illustration of an exemplary feedback system for determining one or more candidates for a final light distribution for active depth sensing. DETAILED DESCRIPTION

[0050] Various aspects of the present disclosure relate to active depth sensing, which includes a projector configured to switch between emitting different light distributions for active depth sensing.

[0051] Some embodiments of active depth sensing systems include structured light depth sensing. As used herein, structured light depth sensing refers to active depth sensing, for which a projector emits multiple instances of focused light in a predefined distribution. As used herein, a light distribution may include a distribution of light spots, arcs of light, or another suitable shape for each focused light instance. The defined distribution means that the device knows the position of each light instance within the distribution before emission. This disclosure refers to the light distribution as a light spot distribution to illustrate various aspects of the present disclosure, but any suitable shape of light instances and any suitable distribution of light instances may be used.

[0052] A light spot can be projected onto a scene, and a receiver of the active depth sensing system can receive reflections of the light spot (as well as other light from the scene). The depth of an object in the scene can be determined by comparing the pattern of the received light to the pattern of the emitted light. When comparing the patterns, a portion of a predefined distribution of the emitted light can be identified in the received light. The positions of the portions of the distribution identified in the received light, as well as any skew or stretch of the portions, are then used to determine the depth of one or more objects in the scene.

[0053] Figure 1 is a diagram of an exemplary active depth sensing system 100 configured to determine the depth of objects 106A and 106B in a scene 106 using light distribution. The active depth sensing system 100 can be used to generate a set of depth values ​​for the scene 106 (such as for a depth map (not shown)). For example, the scene 106 may include a face, and the active depth sensing system 100 can be used to generate a set of depth values ​​(such as a depth map including multiple depth values) indicating the depth of various parts of the face for identifying or verifying the face. As used herein, generating a depth map may refer to generating a set of depth values. The active depth sensing system 100 may include a projector 102 and a receiver 108. The projector 102 may be referred to as a "transmitter", "projector", "transmitter", "light source", etc., and is not limited to a specific transmitting component. Throughout the following disclosure, the terms projector, transmitter, and light source may be used interchangeably. The receiver 108 may be referred to as a "detector", "sensor", "sensing element", "photodetector", etc., and is not limited to a specific receiving component.

[0054] The projector 102 can be configured to project or emit a distribution of light spots 104 onto a scene 106. White circles in the distribution 104 can indicate where light is not projected for a possible point location, while black circles in the distribution 104 can indicate where light is projected for a possible point location. The present disclosure may refer to the distribution 104 as a codeword distribution, where the defined portion of the distribution 104 is a codeword. As used herein, a codeword is a rectangular (such as a square) portion of a light distribution. For example, a 5×5 codeword 140 is shown in the distribution 104. As shown, the 5×5 codeword 140 includes five rows of possible light spots and five columns of possible light spots. The distribution 104 can be configured to include an array of codewords. For active depth sensing, in the distribution 104, the codewords can be unique to each other. For example, the codeword 140 is different from all other codewords in the distribution 104. In addition, the positions of the unique codewords relative to each other are known. In this way, one or more codewords in a distribution can be identified in the reflection, and the positions of the identified codewords relative to each other, the shape or distortion of the identified codewords relative to the transmitted codeword, and the position of the identified codewords on the receiver sensor are used to determine the depth of the object in the scene that reflected the codeword.

[0055] The projector 102 may include one or more light sources 124 (such as one or more lasers). In some embodiments, the one or more light sources 124 include a laser array. Each laser may be a vertical cavity surface emitting laser (VCSEL) or other suitable laser. In some embodiments, the projector may also include a lens 126 and a light modulator 128. The projector 102 may also include an aperture 122 from which the emitted light escapes the projector 102. In some embodiments, the projector 102 may also include a diffractive optical element (DOE) to diffract the emission from the one or more light sources 124 into additional emission. In some aspects, the light modulator 128 (for adjusting the emission intensity) may include a DOE. When projecting the light spot distribution 104 onto the scene 106, the projector 102 may transmit one or more laser emissions from the light source 124 through the lens 126 (and / or through the DOE or light modulator 128) and onto objects 106A and 106B in the scene 106. Projector 102 may be located on the same reference plane as receiver 108, and projector 102 and receiver 108 may be separated by a distance referred to as a baseline (112).

[0056] In some exemplary embodiments, the light projected by the projector 102 may be infrared (IR) light. IR light may include portions of the visible light spectrum and / or portions of the light spectrum that are not visible to the naked eye. In one example, the IR light may include: near infrared (NIR) light, which may or may not include light within the visible light spectrum; and / or IR light (such as far infrared (FIR) light), which is outside the visible light spectrum. The term IR light should not be limited to light having a specific wavelength within or near the wavelength range of IR light. In addition, IR light is provided as an exemplary emission from the projector. In the description below, other suitable wavelengths of light may be used. For example, light in portions of the visible light spectrum outside the wavelength range of IR light or ultraviolet light may be used.

[0057] Scene 106 may include objects at different depths from the structured light system (such as from projector 102 and receiver 108). For example, objects 106A and 106B in scene 106 may be at different depths. Receiver 108 may be configured to receive reflections 110 of emitted light spot distribution 104 from scene 106. To receive reflections 110, receiver 108 may capture frames. When capturing frames, receiver 108 may receive reflections 110, as well as (i) other reflections of light spot distribution 104 from other parts of scene 106 at different depths, and (ii) ambient light. Noise may also be present in the capture.

[0058] In some exemplary embodiments, the receiver 108 may include a lens 130 to focus or direct the received light (including the reflections 110 from the objects 106A and 106B) onto a sensor 132 of the receiver 108. The receiver 108 may also include an aperture 120. Assuming an example of receiving only the reflections 110, the depths of the objects 106A and 106B may be determined based on the baseline 112, the displacement and distortion of the light distribution 104 (such as in a codeword) in the reflections 110, and the intensity of the reflections 110. For example, a distance 134 from the position 116 to the center 114 along the sensor 132 may be used to determine the depth of the object 106B in the scene 106. Similarly, a distance 136 from the position 118 to the center 114 along the sensor 132 may be used to determine the depth of the object 106A in the scene 106. The distance along the sensor 132 may be measured according to the number of pixels of the sensor 132 or a distance unit (such as millimeters).

[0059] In some exemplary embodiments, the sensor 132 may include an array of photodiodes (such as avalanche photodiodes) for capturing frames. To capture a frame, each photodiode in the array may capture light that hits the photodiode and may provide a value (capture value) indicating the intensity of the light. Thus, a frame may be an array of capture values ​​provided by the array of photodiodes.

[0060] As a supplement or alternative to the sensor 132 including a photodiode array, the sensor 132 may include a complementary metal oxide semiconductor (CMOS) sensor. To capture an image through the photosensitive CMOS sensor, each pixel of the sensor may capture the light hitting the pixel and may provide a value indicative of the light intensity. In some exemplary embodiments, the photodiode array may be coupled to the CMOS sensor. In this way, the electrical pulses generated by the photodiode array may trigger the corresponding pixels of the CMOS sensor to provide capture values.

[0061] The sensor 132 may include at least a plurality of pixels equal to the number of possible light points in the distribution 104. For example, the array of photodiodes or the CMOS sensor may include at least a plurality of photodiodes or a plurality of pixels respectively corresponding to the number of possible light points in the distribution 104. The sensor 132 may be logically divided into groups of pixels or photodiodes corresponding to the size of the bits of the codeword (such as a 4×4 group for a 4×4 codeword). The groups of pixels or photodiodes may also be referred to as bits, and the portions of data captured from the bits of the sensor 132 may also be referred to as bits. In some exemplary embodiments, the sensor 132 may include at least the same number of bits as the distribution 104. If the light source 124 emits IR light (such as NIR light with a wavelength of, for example, 940 nm), the sensor 132 may be an IR sensor to receive the reflection of the NIR light.

[0062] As shown, the distance 134 (corresponding to the reflection 110 from the object 106B) is less than the distance 136 (corresponding to the reflection 110 from the object 106A). Using triangulation based on the baseline 112 and the distances 134 and 136, different depths of the objects 106A and 106B in the scene 106 may be determined when generating a depth map of the scene 106. Determining the depth may be further based on the displacement or distortion of the distribution 104 in the reflection 110.

[0063] Although in Figure 1, but one or more of the components may be implemented together or include additional functionality. The active depth sensing system 100 may not require all of the described components, or the functionality of the components may be separated into separate components. There may also be additional components not shown. For example, the receiver 108 may include a bandpass filter to allow signals having a certain range of wavelengths to pass to the sensor 132 (thereby filtering out signals with wavelengths outside of the range). In this way, some incidental signals (such as ambient light) can be prevented from being received as interference during capture by the sensor 132. The range of the bandpass filter can be centered on the emission wavelength of the projector 102. For example, if the projector 102 is configured to emit NIR light having a wavelength of 940nm, the receiver 108 may include a bandpass filter that is configured to allow NIR light with wavelengths in the range of, for example, 920nm to 960nm. Thus, with respect to Figure 1 The examples described are for illustrative purposes.

[0064] For conventional active depth sensing systems using a light distribution, the light distribution is fixed. Therefore, the same light distribution is used for active depth sensing in each instance. However, some codewords in the distribution may be difficult to identify in the reflections received at the receiver. As a result, depth values ​​may not be determined for portions of the scene that reflect an unidentified codeword of the light distribution.

[0065] For example, when an array of light points (such as distribution 104) reflects off the surface of an object in a scene, the angle of reflection may cause multiple light points to be received at the same location of an image sensor (such as sensor 132). If the waves of the light points received at the same location are out of phase, the intensities of the light points may cancel each other out, so that the image sensor's capture may not perceive the received light points. In contrast, if the waves are in phase, the intensities may be amplified, and due to their amplified intensities, the capture at one or more image sensor pixels may be saturated. This interference may be referred to as speckle interference, which is based on the scattering of light by surfaces in the scene. Due to the speckle interference, the light points of a codeword may not be identified, and the active depth sensing system may not be able to identify the codeword that lacks one or more light points in the reflection. As a result, a depth map that includes speckle interference in the image sensor's capture may not include valid depth values ​​for the locations of the unidentified codewords (which may be shown as black shading in the depth map shown).

[0066] The active depth sensing system 100 may use larger codewords to overcome situations where one or more codewords may not be recognized. For example, instead of using a 4×4 codeword (and not recognizing a portion of the codeword in a capture made by the receiver), the active depth sensing system may use a 5×5 codeword or larger. Increasing the size of the codeword increases the number of light points per codeword, and the additional light points may help the active depth sensing system 100 recognize the codewords in the capture. However, as the codeword size increases, the resolution of the depth map decreases. For example, one depth value may be determined for one codeword recognized in the capture. If the codeword size increases from 4x4 to 5x5, the number of codewords in the distribution decreases. As a result, the resolution of the depth map when using 5x5 codewords may be almost half the resolution of the depth map when using 4x4 codewords.

[0067] Instead of using codewords of different sizes, the active depth sensing system 100 can be configured to emit a sparse light distribution. As used herein, a sparse light distribution means that the number of light points in a defined area of ​​a first light distribution is less than the number of light points of a second light distribution. For example, the spacing between the light points of the first light distribution can be greater than the spacing between the light points of the second light distribution. Increasing the spacing between the light points can improve the ability of the active depth sensing system to identify each specific light point in one or more reflections of the light distribution being captured. However, similar to the problems with increasing the codeword size of the light distribution, using a sparse light distribution can result in a lower resolution depth map than using a denser light distribution.

[0068] Additionally, when the light distribution is fixed, some portions of the distribution do not include light points. For example, light distribution 104 includes locations where no light instances exist. Because there are gaps between light points, the scene may be unevenly illuminated by light distribution 104. Uneven illumination of the scene may also cause difficulties in generating depth values ​​for generating a depth map.

[0069] Figure 2A 200 is an illustration of a depth map 204 generated based on a light distribution 202 projected by a projector of an active depth sensing system. Black portions of the depth map 204 indicate invalid depth values ​​(i.e., depth values ​​that the active depth sensing system cannot determine). Portions of the scene outside of the face and bust shown in the depth map 204 may be outside the sensing range of the active depth sensing system. However, portions of the face that are within the sensing range are also associated with invalid depth values. For example, portions 206 and 208 of the depth map 204 include black portions that indicate invalid depth values. The invalid depth values ​​in portions 206 and 208 may be due to the active depth sensing system being unable to recognize the codeword of the distribution 202 in one or more reflections used to generate the depth map.

[0070] Active depth sensing using a light distribution different from distribution 202 may still result in portions of the depth map including invalid depth values. For example, a complementary light distribution to distribution 202 may still be associated with invalid depth values ​​in a depth map obtained based on the complementary distribution. As used herein, a complementary light distribution is the inverse of a basic light distribution. For example, for light distribution 202, each square indicates a sector of the distribution that may include light spots. Dark squares indicate sectors that do not include light spots, while light squares indicate sectors that include light spots. In some embodiments, light distribution 202 is emitted by a laser array, and each square in distribution 202 may be associated with a laser of the laser array. Dark squares may indicate lasers that do not emit light spots, while light squares may indicate lasers that emit light spots when generating light distribution 202. In some embodiments, a complementary light distribution may refer to a light distribution in which lasers in a laser array that are not emitted for a basic distribution are emitted for a complementary distribution, while lasers in a laser array that are emitted for a basic distribution are not emitted for a complementary distribution.

[0071] Figure 2B is an illustration 250 of a depth map 254 generated based on a light distribution 252 projected by a projector of an active depth sensing system. The distribution 252 is Figure 2A 202 in FIG. 204. A depth map 254 generated using distribution 252 still includes black portions indicating invalid depth values. As shown, portions 256 and 258 of depth map 254, which correspond to portions 206 and 208 of depth map 204, include black portions indicating invalid depth values ​​(the black portions are in different locations within corresponding portions 206 and 256 and corresponding portions 208 and 258).

[0072] In some aspects, a final depth map is generated based on multiple light distributions. In some embodiments, the projector 102 is configured to project different light distributions at different times. For example, the projector 102 can be configured to project a first light distribution at a first time and a second light distribution at a second time. The resulting depth map of one or more objects in the scene is therefore based on one or more reflections of the first light distribution and one or more reflections of the second light distribution. The codewords between the light distributions may be different, and the active depth sensing system 100 may be able to recognize codewords in the second light distribution corresponding to locations where codewords cannot be recognized in the first light distribution. In some embodiments, different codewords for different light distributions may refer to one or more of codewords of different sizes for each distribution, complementary light distributions, or different emission times. In this way, more valid depth values ​​can be generated when generating a depth map without reducing the resolution of the depth map (such as by increasing the size of the codewords).

[0073] Return to reference Figure 1, projector 102 can include any suitable light source 124. In some embodiments, light source 124 can include one or more vertical cavity surface emitting lasers (VCSELs). In some other embodiments, light source 124 can include one or more distributed feedback (DFB) lasers or other suitable lasers. The lasers can be arranged in a laser array that is configured to emit light distribution 104 as a first light distribution and to emit one or more other light distributions having the same resolution as light distribution 104.

[0074] In some embodiments, one or more lasers can be coupled to a DOE, and the light emitted from the lasers is received and directed by the associated DOE. The coupled DOE is a material located in the projection path of the light from the light source. The DOE can be configured to split a single spot of light into multiple spots of light. For example, the material of the DOE can be a translucent or transparent polymer with a known refractive index. The surface of the DOE can include peaks and valleys (changing the depth of the DOE) so that when the light passes through the DOE, a single spot of light splits into multiple spots of light. For example, the DOE can be configured to receive one or more spots of light from one or more lasers and project a desired distribution with a greater number of spots of light than the number of spots of light emitted by the one or more lasers.

[0075] If the light source 124 includes a laser array (such as a VCSEL array), at least a portion of the light spot distribution can be projected by the array without a DOE. The present disclosure describes a laser array that is configured to emit a light distribution without using a DOE. However, any suitable configuration of the light source 124 or other components for projecting a light distribution by the projector 102 can be used to perform various aspects of the present disclosure.

[0076] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" refers to being directly connected to or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be needed to practice the teachings disclosed herein. In other examples, known circuits and devices are shown in block diagram form to avoid confusing the teachings of the present disclosure. Some parts of the following detailed description are presented in the form of programs, logic boxes, processes, and other symbolic representations of the operation of data bits in a computer memory. In the present disclosure, programs, logic boxes, processes, etc. are considered to be self-consistent sequences of steps or instructions that lead to desired results. The steps are those steps that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0077] It should be kept in mind, however, that all of these or similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise clearly indicated from the discussion below, it should be understood that throughout this application, discussions using terms such as "access," "receive," "issue," "use," "select," "determine," "standardize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "solve," etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the computer system's registers and memories and transforms it into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0078] In the accompanying drawings, a single box may be described as performing one or more functions; however, in actual practice, one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, boxes, modules, circuits, and steps are generally described below according to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the present disclosure. Moreover, the exemplary device may include components other than the components shown, including well-known components such as processors, memories, etc.

[0079] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or being coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0080] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. In addition, the term "system" is not limited to multiple components or specific embodiments. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0081] Figure 3 is a block diagram of an exemplary device 300 including a projector 301 configured to emit different light distributions for active depth sensing. In some other examples, the transmitter can be separate from and coupled to the device 300. The exemplary device 300 can include or be coupled to the projector 301 and a receiver 302 separated from the projector 301 by a baseline 303. The receiver 302 can be an IR sensor configured to capture frames, and the projector 301 can be a projector configured to project two or more different light distributions.

[0082] The exemplary device 300 may also include a processor 304, a memory 306 storing instructions 308, and a light controller 310 (which may include one or more image signal processors 312). The device 300 may optionally include (or be coupled to) a display 314 and multiple input / output (I / O) components 316. The device 300 may also include a power supply 318, which may be coupled to the device 300 or integrated into the device. The device 300 may include additional features or components not shown. For example, a wireless interface that may include multiple transceivers and a baseband processor may be included to enable the wireless communication device to perform wireless communications. In another example, the device 300 may include one or more cameras (such as a contact image sensor (CIS) camera or other suitable camera for capturing images using visible light). The projector 301 and the receiver 302 may be an active depth sensing system (such as a 3D image sensor) controlled by the light controller 310 and / or the processor 304. Figure 1 Device 300 may include or be coupled to an additional light projector (or flood illuminator), or may include a different configuration for a light projector. Device 300 may also include or be coupled to an additional receiver (not shown) for capturing multiple frames of a scene. The present disclosure is not limited to any example or illustration, including exemplary device 300.

[0083] The memory 306 may be a non-transitory or non-temporary computer-readable medium storing computer-executable instructions 308 to perform all or part of one or more operations described in the present disclosure. If each light distribution projected by the projector 301 is divided into codewords, the memory 306 may optionally store a codeword library 309 for each light distribution, wherein the library 309 includes a plurality of codewords. The codeword library 309 may indicate which codewords are present in the distribution and the relative positions between the codewords in the distribution. The device 300 may use the codeword library 309 to identify the codeword in one or more reflections within the capture from the receiver 302 associated with the light distribution corresponding to the codeword library 309. The memory 306 (such as one or more codeword libraries 309) may include additional information about one or more light distributions. In some embodiments, the memory 306 includes information about the emission time of the light distribution (such as the length of time to project the light distribution before the projector 301 projects a different light distribution). In some embodiments, the codeword library 309 may include codewords of different sizes between different light distributions projected by the projector 301.

[0084] The processor 304 may be one or more suitable processors capable of executing scripts or instructions of one or more software programs, such as instructions 308 stored in the memory 306. In some aspects, the processor 304 may be one or more general-purpose processors that execute the instructions 308 to cause the device 300 to perform any number of functions or operations. In additional or alternative aspects, the processor 304 may include an integrated circuit or other hardware to perform functions or operations without the use of software. In some embodiments, the processor 304 includes one or more application processors to execute applications stored in the executable instructions 308. For example, if the device 300 is a smart phone or other computing device, the processor 304 may execute instructions for the operating system of the device 300, and the processor 304 may provide instructions to the light controller 310 to control the active depth sensing system.

[0085] Despite Figure 3 304, but in various arrangements, the processor 304, memory 306, light controller 310, optional display 314, and optional I / O components 316 may be coupled to each other. For example, the processor 304, memory 306, light controller 310, optional display 314, and / or optional I / O components 316 may be coupled to each other via one or more local buses (not shown for simplicity).

[0086] The display 314 may be any suitable display or screen that allows a user to interact and / or presents items for the user to view (such as a depth map, a preview image of a scene, a lock screen, etc.). In some aspects, the display 314 may be a touch-sensitive display. The I / O component 316 may be or include any suitable mechanism, interface, or device for receiving input (such as commands) from a user and providing output to the user. For example, the I / O component 316 may include (but is not limited to) a graphical user interface, a keyboard, a mouse, a microphone and a speaker, a squeezable border or border of the device 300, a physical button located on the device 300, and the like. The display 314 and / or the I / O component 316 may provide a preview image or depth map of a scene to the user and / or receive user input for adjusting one or more settings of the device 300.

[0087] The light controller 310 may include a signal processor 312, which may be one or more processors to configure the projector 301 and process frames captured by the receiver 302. In some aspects, the image signal processor 312 may execute instructions from a memory (such as instructions 308 from the memory 306 or instructions stored in a separate memory coupled to the image signal processor 312). In other aspects, the image signal processor 312 may include specific hardware for operation. The image signal processor 312 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

[0088] In some embodiments, processor 304 may be configured to provide instructions to image signal processor 312. The instructions may be executed by image signal processor 312 to configure filters or other components of an image processing pipeline for processing frames from receiver 302. The instructions may also be executed by image signal processor 312 to configure projector 301 to project one or more light distributions for active depth sensing. Although the following aspects of the disclosure may be described with respect to device 300, any suitable device or configuration of device components may be used to perform various aspects of the disclosure, and the disclosure is not limited to a particular device configuration.

[0089] Figure 4 4 is an illustration 400 of a laser array 402 including lasers 404 configured to emit a first light distribution and one or more other light distributions (including a second light distribution). The laser array 402 can have any suitable size. In some embodiments, the laser array 402 includes an array of VCSELs (each laser 404 is a VCSEL). In this manner, the illustrated portion of the laser array 402 can be configured to emit a first 4x4 codeword 406 and a second 4x4 codeword 408.

[0090] The light distribution (including the codeword) can be optimized to improve depth sensing using the light distribution. For example, the light spot is located in the light distribution to increase the probability of identifying the associated codeword in the reflection of the light. If the laser array 402 is configured to emit two or more light distributions, each distribution can be optimized (including the optimization of the codeword). In this way, the first light distribution can be referred to as the original light distribution (including the original codeword 406), and the second light distribution can be referred to as the dual light distribution (including the dual codeword 408). In the present disclosure, the adjectives "original" and "dual" refer to that the light distribution (and the associated codeword) is optimized.

[0091] For optimization, the operation of the laser array 402 satisfies one or more criteria to optimize the light distribution. One criterion may be that the distribution of light spots in each codeword is unique. In this way, multiple instances of the same codeword do not appear in the light distribution. Other criteria may be that the operation of the VCSEL during active depth sensing remains below a threshold duty cycle (e.g., a threshold ratio of the time the VCSEL is powered or pulsed to the time the VCSEL is not powered or pulsed). In this way, overheating of the laser array 402 is prevented. Another criterion may be that the location of the light spots causes the corresponding depth map to meet a threshold quality metric (such as below the number or ratio of invalid depth values ​​during calibration).

[0092] The above criteria correspond to optimizing a single light distribution. The light distributions can also be optimized relative to each other. For example, another criterion can be that each VCSEL emits a light spot during at least one of the multiple light distributions emitted by the laser array 402. In this way, the illumination of the scene is more uniform than when a single fixed light distribution is used for active depth sensing. In addition, the position of the original codeword 406 in the original light distribution corresponds to the position of the dual codeword 408 in the dual light distribution. Other criteria can be that the codewords at corresponding positions between the original light distribution and one or more dual light distributions are unique to each other. In this way, in the light distribution, the same codeword is not used for the same position. Another criterion can be that the average brightness or light intensity emitted by each VCSEL is approximately the same as that of other VCSELs in the laser array. In this way, the average intensity of one VCSEL is not higher than the average intensity of other VCSELs to allow the total power of the VCSEL to be adjusted (therefore adjusting the total intensity of the light distribution), while the emission of each VCSEL remains below the required intensity threshold (such as based on eye safety standards, government regulations, etc.).

[0093] In this manner, as used herein, optimization can refer to configuring different light distributions to improve the accuracy and resolution of a depth map (while complying with equipment or regulatory requirements). For example, different light distributions can be configured to reduce areas in a depth map where depth values ​​cannot be generated (such as by one or more codewords that do not identify the light distribution). In another example, different light distributions can be configured to balance the resolution of the final depth map relative to areas of the final depth map where there are no depth values ​​(such as areas where the resolution satisfies a first threshold and the depth values ​​remain below a second threshold). Configuring different light distributions can include configuring the size of the codewords for each light distribution, the emission time of each light distribution, the layout of the codewords for each light distribution, or the relationship between areas of different light distributions. Exemplary relationships between different light distributions can include codewords that complement each other in the same area (such as the original codeword 406 and the dual codeword 408). Reference Fig.10 Optimization of various light distributions is described in more detail.

[0094] In some embodiments, the laser array 402 can be configured to emit a dual light distribution that is complementary to the original light distribution. For example, the dual codeword 408 is complementary to the original codeword 406, where the VCSEL is pulsed for the original light distribution, or is pulsed for the dual light distribution. In addition, referring again to Figure 2A and 2B , Figure 2B The distribution of 252 and Figure 2A 202 in the first light distribution. In this way, it is ensured that each VCSEL (or other suitable laser or light source) is pulsed for at least one of the first light distribution or the second light distribution. In addition, when the other light distribution is pulsed, the VCSEL or laser is in a static state (not pulsating), which helps prevent overheating of the laser array. In some embodiments, each laser (such as each VCSEL) is pulsed for only one of the first light distribution (which can be at a first time) or the second light distribution (which can be at a second time). In this way, the laser can emit for only one of the first time or the second time.

[0095] Return to reference Figure 4 , for the first column of the original codeword 406, three lasers 404 emit light, and one laser 404 does not emit light. For the dual codeword 408 that is complementary to the original codeword 406, one laser 404 that does not emit light for the first column of the original codeword 406 emits light for the dual codeword 408. Moreover, the other three lasers 404 do not emit light for the dual codeword 408. In some embodiments, a first plurality of lasers 404 of the laser array 402 that emits light for a first light distribution can be connected together, and a second plurality of lasers 404 of the laser array 402 that emits light for a second light distribution can be connected together. For example, lasers 404A, 404C, and 404D configured to emit light for the first column of the original codeword 406 can be connected to be coupled to the power supply 414 during a first time, and laser 404B configured to emit light for the first column of the dual codeword 408 can be separately connected to other lasers in the laser 402 to be coupled to the power supply 414 during a second time. In some implementations, lasers 404A, 404C, and 404D can be coupled via substrate 418, and laser 404B can be coupled to other lasers via substrate 410. The projector can include a switch 412 configured to switch between coupling substrate 418 to power source 414 and coupling substrate 410 to power source 414.

[0096] If the light distributions are complementary to each other, each laser 404 is coupled only to substrate 418 or substrate 410. If the light distributions are not complementary, one or more lasers may be coupled to both substrates 418 and 410. Although not shown, the projector may be configured to emit three or more light distributions. For example, the laser array 402 may be configured to emit an original distribution, a first dual distribution, and a second dual distribution. Some of the first plurality of lasers 404 and the second plurality of lasers 404 of the laser array 402 may therefore be coupled to a third substrate (not shown), and the switch 412 may be configured to switch between the three substrates (and thus cause the projector to switch between three different light distributions). If there is a second dual distribution, at least one of the first dual distribution or the second dual distribution is not complementary to the original distribution because the codewords at corresponding positions in different distributions will be unique. Therefore, the dual light distribution is not necessarily complementary to the original light distribution.

[0097] As described herein, the projector 301 ( Figure 3 ) is configured to switch between emitting a first light distribution and emitting a second light distribution (and optionally, emitting additional light distributions). Figure 5 is an illustrative flow chart depicting an exemplary operation 500 of a projector 301 for active depth sensing. At 502, the projector 301 emits a first light distribution at a first time. At 504, the projector 301 emits a second light distribution at a second time. The generated depth map of one or more objects in the scene is based on one or more reflections of the first light distribution and one or more reflections of the second light distribution. The first time and the second time may include any suitable time instance (such as a suitable start time or end time). The first time and the second time may also include any suitable duration (such as a first duration for projecting the first light distribution and a second duration for projecting the second light distribution). In some embodiments, the projector 301 may also emit a third light distribution at a third time (506). The projector 301 may further emit other light distributions (not shown) at other times. The generated depth map of one or more objects in the scene may also be based on one or more reflections of the third light distribution (as well as one or more reflections of any other light distribution).

[0098] The receiver 302 is configured to capture frames at a frame capture rate (such as 30 frames per second). In some embodiments, the projector 301 is configured to emit a first light distribution for a first frame capture and emit a second light distribution for a second frame capture. In this way, a single frame capture does not include reflections of both the first light distribution and the second light distribution. For example, a first plurality of lasers of the laser array can be configured to pulse the light spot during a first frame capture (such as at the beginning of a frame or at an exposure window for capturing a frame by the receiver 302), and a second plurality of lasers of the laser array can be configured to pulse the light spot during a second frame capture. In some embodiments, a switch of the projector 301 is configured to configure which lasers pulse during which frame capture. In some other embodiments, the pulsation and charging intervals of the lasers can be coordinated and timed so that the first plurality of lasers and the second plurality of lasers flash at appropriate times (such as during a first frame laser capture for the first plurality of lasers and during a second frame laser capture for the second plurality of lasers) for active depth sensing.

[0099] In some embodiments, projector 301 is configured to periodically alternate between emitting a first light distribution and emitting a second light distribution. Each emission of the first light distribution can be associated with one frame in a first frame capture set of an image sensor of receiver 302, and each emission of the second light distribution can be associated with one frame in a second frame capture set of an image sensor of receiver 302. Image signal processor 312 can be configured to generate a first set of depth values ​​based on the first set of frames received from receiver 302, and to generate a second set of depth values ​​based on the second set of frames received from receiver 302.

[0100] Figure 6 is an illustrative flow chart depicting an exemplary operation 600 of active depth sensing based on projector 301 alternating between emitting a first light distribution and emitting a second light distribution. With n set to 1, during frame capture n by an image sensor of receiver 302, projector 301 emits a first light distribution (602). In some embodiments, a laser array of projector 301 pulses the original light distribution during frame capture n (604). At 606, the image sensor of receiver 302 captures frame n, and the frame capture includes one or more reflections of the first light distribution. The frame capture is provided to light controller 310, and image signal processor 312 generates a set of depth values ​​n based on the frame capture (608). For example, image signal processor 312 can generate a first depth map based on the frame capture by identifying a codeword of the first light distribution in the frame capture and determining the depth of one or more objects in the scene based on the identified codeword.

[0101] To illustrate, Figure 7is an illustration 700 of an exemplary alternating emission between a first light distribution and a second light distribution (such as by Figure 3 301 in FIG. 700 ). Diagram 700 also shows an exemplary resulting set of depth values ​​(shown as a depth map) for alternating light distributions. In an exemplary embodiment, the first light distribution is Figure 2A The distribution 202 in the second light distribution is Figure 2B 252 in the image capture image. In this example, the first light distribution is an original distribution and the second light distribution is a dual distribution that is a complementary distribution of the first light distribution. The projector 301 projects a first distribution 702 associated with frame capture 1. For example, the projector 301 may project the first distribution 702 during at least a portion of an exposure window for frame capture 1 (which may be referred to as during frame capture 1). The receiver 302 captures frame 1, and the image signal processor 312 generates a first set of depth values ​​(which may be represented as a first depth map 704) based on the captured frame 1.

[0102] Return to reference Figure 6 , n may be incremented (610). The projector 301 may then emit a second light distribution associated with frame capture n of the image sensor of the receiver 302 (612). For example, the projector 301 may project the second distribution 706 during at least a portion of the exposure window for frame capture n (which may be referred to as during frame capture n). In some embodiments, the laser array of the projector 301 may pulse the dual light distribution associated with frame capture n (614). At 616, the image sensor of the receiver 302 captures frame n, and the frame capture includes one or more reflections of the second light distribution. The frame capture is provided to the light controller 310, and the image signal processor 312 generates a depth value set n based on the frame capture (618). For example, the image signal processor 312 may generate a second depth value set (such as a second depth value set of a second depth map) based on the frame capture by identifying a codeword of the second light distribution in the frame capture and determining the depth of one or more objects in the scene based on the identified codeword. For illustration, return to reference Figure 7 , the projector 301 projects a second distribution 706 associated with frame capture 2. The receiver 302 captures frame 2, and the image signal processor 312 generates a second set of depth values ​​(represented by a second depth map 708) based on the captured frame 2.

[0103] Return to reference Figure 6 , n can be incremented (620), and the process can return to step 602. For illustration, refer again to Figure 7, projector 301 emits a first distribution 702 of light associated with frame capture 3, and image signal processor 312 can generate a depth value represented by depth map 710 based on captured frame 3. Projector 301 can then emit a second distribution 706 of light associated with frame capture 4, and image signal processor 312 can generate a depth value (which can be represented by depth map 710) based on captured frame 4. Figure 7 The projector 301 may continue to alternate between emitting the first light distribution 702 and emitting the second light distribution 706 for additional frames captured by the image sensor during active depth sensing, and the image signal processor 312 may continue to generate a set of depth values ​​based on the frame captures of the receiver 302. Figure 6 and Figure 7 The exemplary embodiment in FIG. 3 depicts projector 301 alternating between two light distributions, but projector 301 may be configured to switch between three or more light distributions. Figure 6 and Figure 7 The exemplary embodiment in depicts that projector 301 is alternated for each frame capture, but projector 301 may be alternated at any suitable time. For example, projector 301 may be alternated for every other frame capture or for every x frame captures.

[0104] The image signal processor 312 may capture the first frame based on the Figure 7 The image signal processor 312 may also generate a first set of depth values ​​based on a second frame capture (such as that captured by Figure 7 ) to generate a second depth value set. In some embodiments, the image signal processor 312 can generate a depth value set for each frame capture. As described above, portions of each depth value set may include invalid depth values. However, the locations in the scene corresponding to the invalid depth values ​​captured in the first frame may be different from the locations in the scene corresponding to the invalid depth values ​​captured in the second frame. The image signal processor 312 can be configured to combine the depth value sets to generate a final depth value set. In this way, if a certain location is associated with at least one valid depth value from multiple depth value sets, the number of locations in the scene associated with invalid depth values ​​can be reduced. For example, referring again to Figure 7, the device 300 may generate a first set of depth values, which may be represented by a first depth map 704 of size P×Q, the first depth map being based on a first frame capture associated with the emission of the first light distribution 702. The device 300 may also generate a second set of depth values, which may be represented by a second depth map 708 of size P×Q, the second depth map being based on a second frame capture associated with the emission of the second light distribution 706. The device 300 may be configured to combine at least portions of the first set of depth values ​​(represented by the first depth map 704) and the second set of depth values ​​(represented by the second depth map 708) to generate a final set of depth values ​​(which may be represented by a final depth map). For example, the device 300 may compare one or more depth values ​​at corresponding locations in the first depth map and the second depth map to generate a final depth value at the location in the final depth map.

[0105] Figure 8 is an exemplary flow chart depicting exemplary operations 800 for combining depth values ​​of a first depth map of size P x ​​Q and depth values ​​of a second depth map of size P x ​​Q to generate depth values ​​of a final depth map. At 802, a variable p is set to 1 (p is less than or equal to P). The variable p is used to track corresponding rows of depth values ​​in the depth map during the combination of depth values ​​of the two depth maps. At 804, a variable q is set to 1 (q is less than or equal to Q). The variable q is used to track corresponding columns of depth values ​​in the depth map during the combination of depth values ​​of the two depth maps.

[0106] The device 300 (such as the image signal processor 312 or other suitable device component) determines whether a first depth value at a location pxq in a first depth map is valid. Fig. 9900 is an illustration of an exemplary first depth map 902 and an exemplary second depth map 904 of depth values ​​that are combined to generate depth values ​​of a final depth map 906. The first depth map 902 may be a visual illustration of a first set of depth values ​​determined by the image signal processor 312, and the second depth map 904 may be a visual illustration of a second set of depth values ​​determined by the image signal processor 312. As shown, the first depth map 902 is of size P×Q, and the second depth map 904 is of size P×Q. The depth values ​​of the depth maps 902 and 904 are combined to generate depth values ​​of a final depth map 906 of size P×Q. The device 300 may determine whether the depth value at 1×1 in the first depth map 902 is valid. When generating an invalid depth value, if an insufficient number of light points or the location of the light point is not identified for the portion of the reflection associated with the location 1×1 in the depth map 902, the device 300 may not identify the codeword associated with the location. Therefore, the device 300 may not be able to determine a valid depth value for the location. The device 300 indicates that the position is associated with an invalid depth value by setting the depth value to a predefined value, indicating an invalid depth value by using a non-number (NaN) indication, leaving the depth value blank for the position, or indicating an invalid depth value by any other suitable means. In some embodiments, the device 300 may identify multiple possible codewords that may be present in the reflected portion associated with the position 1x1 in the depth map 902. In this way, the device 300 can associate a confidence value with each possible codeword based on the likelihood that the codeword exists in the reflected portion. For example, the device 300 can identify 75% of the light points of the possible codeword in the correct position relative to each other, but the device 300 cannot identify 25% of the light points of the possible codeword. Similar identification can be performed for other possible codewords. In this way, the device 300 can determine that two or more codewords are equally likely to be associated with the position (such as a 50% confidence value for two codewords). Thus, the device 300 cannot determine the actual codeword associated with the position. In some embodiments, the device 300 determines that the depth value is valid based on the identified codeword with the maximum confidence value and whose confidence value is above a threshold. Although the above example provides the identification of a single codeword based only on the light points that identify the codeword in the reflection, the identification of the codeword (such as generating a confidence value for the possible codeword) can also be based on identifying adjacent codewords in the reflection, identifying codewords associated with other light distributions and corresponding to the current codeword to be identified, or identifying other suitable parameters of the codeword. If a codeword is identified for the location, the device 300 can determine the effective depth value of the location in the depth map 902 based on the position and distortion of the light points of the codeword relative to each other in the reflection.

[0107] When it is determined that the depth value is invalid, device 300 may determine that the depth value is set to a predetermined value to indicate an invalid depth value, the depth value is not set to a number or value, the depth value is associated with a NaN indication, or the depth value is any other suitable indication of an invalid depth value. Depth map 902 has a black shadow at position 1x1, indicating an invalid depth value for the position in depth map 902.

[0108] Return reference Figure 8 , in some embodiments, device 300 may also determine whether a second depth value at position p x q in the second depth map is a valid depth value (808). For example, device 300 may determine whether the depth value at position 1×1 in depth map 904 ( Fig. 9 ) is valid (such as described above). Depth map 904 has a black shadow at position 1x1, indicating an invalid depth value for the position in depth map 904.

[0109] In this way, the final depth value at position p x q in the final depth map is based on the validity of the first depth value and / or the second depth value. In some other embodiments, if the first depth value is valid, device 300 may use the first depth value as the final depth value (regardless of the second depth value). For example, for a depth map associated with the original light distribution, if the depth value is valid, the valid depth value will automatically be used as the final depth value of the final depth map without considering the validity of the depth values from the depth map associated with the dual light distribution. In another example, for a depth map that is the latest depth map from the combined depth maps, if the depth value is valid, the valid depth value is automatically used as the final depth value. In this example, if more than two sets of depth values are to be combined (which may correspond to two or more depth maps), device 300 may use the depth values from the latest set of depth values (corresponding to the latest depth map) where the depth values are determined to be valid. In such examples, if the first depth value is invalid, device 300 may continue to determine the validity of the second depth value in order to determine whether there is any valid depth value corresponding to position p x q in the final depth map.

[0110] Return reference Figure 8 , if neither the first depth value nor the second depth value is a valid depth value (810), device 300 prevents determination of a final depth value that is valid at position p x q in the final depth map. For example, device 300 may set the final depth value to an invalid value (such as a NaN indication for indicating infinity or an error, a predefined value, etc.). Then, the process proceeds to decision 820. As Fig. 9 shown, the final depth value at position 1x1 in final depth map 906 is an invalid depth value (indicated by the black shadow in the upper left corner of depth map 906).

[0111] Return to reference Figure 8 In decision 810, if at least one of the first depth value or the second depth value is valid, the device 300 may use at least one of the first depth value or the second depth value for generating a final depth value in the final depth map (812). In some embodiments, if both the first depth value and the second depth value are valid, the device 300 combines the first depth value and the second depth value (814). For example, the device 300 may average the depth values. In some other examples, the device 300 may find a median depth value, perform a weighted average based on the age of the depth map, or any other suitable means for combining depth values.

[0112] In some other embodiments of generating the final depth value, if only one of the depth values ​​is valid, the device 300 may use the valid depth value from the first depth value or the second depth value (816). For example, if the first depth value is valid and the second depth value is invalid, the device 300 may set the final depth value to the first depth value. In some other embodiments of generating the final depth value, the device 300 may use the second depth value (818) only when the first depth value is invalid. For example, as long as the first depth value is valid (even if the second depth value may be valid), the device 300 may use the first depth value as the final depth value. However, if the first depth value is invalid and the second depth value is valid, the device 300 may use the second depth value as the final depth value.

[0113] Proceeding to decision 820, if the variable q is not equal to Q (the device 300 has not generated a final depth value for each column q along row p), q is incremented (822) and the process returns to step 806. At 820, if the variable q is equal to Q, the process proceeds to decision 824. At 824, if the variable p is not equal to P (the device 300 has not yet generated a final depth value for each row p of the final depth map), p is incremented (826) and the process returns to step 804. At 824, if the variable p is equal to P, all final depth values ​​(for the final depth map) have been generated and the process ends. Fig. 9 As shown, in some embodiments, the device 300 may compare the depth values ​​in the depth map row by row from top to bottom. However, any suitable method (such as from bottom to top, pseudo-random or other means) may be used to gradually traverse the depth values ​​of the depth map. Fig. 9, the final set of depth values ​​shown by the final depth map 906 (as a combination of the first set of depth values ​​shown by the first depth map 902 and the second set of depth values ​​shown by the second depth map 904) includes fewer invalid depth values ​​than the first set of depth values ​​or the second set of depth values ​​(as shown by the fewer locations of invalid depth values ​​in the final depth map 906 than in the depth maps 902 and 904). For example, the mouth and eyes of the face in the final depth map 906 have fewer black areas than the mouth and eyes of the face in the depth maps 902 and 904.

[0114] although Figure 8 and Fig. 9 Depicting two depth maps (representing two sets of depth values) being combined to generate a final depth value set for a final depth map, but any number of sets of depth values ​​may be combined to generate a final set of depth values. In some embodiments, if three or more sets of depth values ​​are combined, the exemplary operation 800 may be extended to combine multiple sets of depth values. For example, the valid depth values ​​may be averaged, or the valid latest depth value may be used, to generate a final depth value.

[0115] In some other embodiments, multiple sets of depth values ​​may be associated with the same light distribution. Figure 7 , depth map 704 and depth map 710 show two sets of depth values ​​associated with first light distribution 702, and depth map 708 (and another depth map for frame capture 4, not shown) shows two sets of depth values ​​associated with second light distribution 706. As a result, there may be a first number of sets of depth values ​​associated with different frame captures but having the same first light distribution, and there may be a second number of sets of depth values ​​associated with different frame captures but having the same second light distribution. If projector 301 projects three different light distributions, there may be a third number of sets of depth values ​​associated with different frame captures but having the same third light distribution.

[0116] In some embodiments, the depth value set may be as described herein with reference to Figure 8 In some other embodiments, the depth values ​​associated with the first light distribution can be combined to generate a first final depth value set, the depth values ​​associated with the second light distribution can be combined to generate a second final depth value set, and the first final depth value set and the second final depth value set can be combined to generate an overall final depth value set. Each operation of combining depth value sets (such as to generate a first final depth value set, to generate a second final depth value set, or to generate an overall final depth value set) can be as described herein with reference to Figure 8However, although some embodiments for combining depth sets of depth values ​​have been described, any suitable means for generating a final set of depth values ​​(such as for a final depth map) based on two or more sets of depth values ​​may be performed. For example, for clarity, Figure 8 Combining similarly sized sets of depth values ​​(represented by similarly sized depth maps) is depicted. In some other embodiments, the depth maps or sets of depth values ​​may be of different sizes. For example, a first light distribution may be associated with a codeword of a first size, while a second light distribution may be associated with a codeword of a second size. In this way, a first depth map generated based on the first light distribution and a second depth map generated based on the second light distribution have different sizes. When combining the first depth map and the second depth map, each position in the first depth map may be associated with a position in the second depth map. If codewords of different sizes are used, a position pxq in a first depth map of size P x ​​Q may correspond to a position rxs in a second depth map of size R x S. Corresponding positions of depth values ​​between depth maps may be based on similar positions in their respective depth maps. For example, if Then the position pxq in the first depth map may correspond to the position rxs in the second depth map. The depth values ​​associated with the corresponding positions may be combined to generate a final depth value. The final depth map may have any suitable resolution. For example, the final depth map may have a first size (of the first depth map), a second size (of the second depth map), or a size between the first size and the second size. To generate the final set of depth values, any suitable means for combining depth values ​​from different size sets may be performed.

[0117] Reference again Figure 6 and Figure 7, although the above method has been described when a set of depth values ​​or depth maps are generated separately for different distributions, reflections from one distribution can be used to help identify codewords in reflections from another distribution. For example, for the original light distribution and the dual light distribution, the device 300 can map or otherwise determine the corresponding positions of the codewords between the light distributions. This mapping can be stored in the memory 306 (such as included in one or more codeword libraries 309). In a specific example, if the dual light distribution is complementary to the original light distribution, when a codeword for the dual light distribution is identified in a frame capture, the device 300 determines that the complement of the codeword exists at a similar position in another frame capture for the original light distribution. In this way, determining the depth value of the same area of ​​the scene corresponding to the original light distribution and the dual light distribution can be based on identifying the codeword from any one of the light distributions. In addition, identifying the codeword in multiple light distributions corresponding to the same area of ​​the scene allows the device 300 to verify that one or more codewords have not been misidentified. Identifying a codeword comprising a reflection of a light distribution in an image sensor capture may be referred to as decoding, and using multiple light distributions for different frame captures to decode similar locations in the light distribution may be referred to as complementary decoding. Complementary decoding reduces the effects of speckle interference that prevents identification of one or more codewords (and therefore determination of a valid depth value for a location in a scene) because such complementary decoding provides spatial diversity in light transmission and reception.

[0118] As described herein, projector 301 can be configured to emit different light distributions at different times, and the different light distributions can be used to generate a final set of depth values ​​for a scene (such as for a final depth map). The first light distribution can be different from the second light distribution in one or more ways. In some embodiments, the first light distribution can be different from the second light distribution in space, and the first light distribution can be different from the second light distribution in time. Spatially different, the first light distribution and the second light distribution can include different positions of light spots. Temporally different, the first light distribution and the second light distribution can be projected for different durations. In addition, the parameters of the depth value sets to be combined can be different. For example, a first depth value set or depth map can have a first size or resolution, and a second depth value set or depth map can have a second size or resolution. The first size and the second size can be based on codewords of different sizes for the first light distribution and for the second light distribution.

[0119] Determining multiple light distributions (including the spatial distribution of light spots, the time and duration of projecting light spots, and the size of the codewords for each light distribution) can be conceptualized as a multidimensional problem. One or more dimensions may include one or more spatial dimensions, time dimensions, and array dimensions (for which the array is a sequence of multiple light distributions relative to each other). For example, if the first light distribution is (X1, Y1) (where (x1, y1) in (X1, Y1) indicates a VCSEL located at a position of row x1 and column y1 in a VCSEL array), the second distributed light is (X2, Y2), one or more time aspects of projecting (X1, Y1) and (X2, Y2) are T, and the array of (X1, Y1) and (X2, Y2) in the sequence is Z, the multidimensional problem may include configuring (X1, Y1), (X2, Y2), and T (where Z is based on (X1, Y1) and (X2, Y2)) to optimize the projected distribution for active depth sensing. Optimizing multiple light distributions may include configuring (X1, Y1), (X2, Y2), and T to improve the resolution and decoding success rate of the final depth map. Although two light distributions (X1, Y1) and (X2, Y2) are described in the example, any number of distributions (such as (X3, Y3), (X4, Y4), etc.) may be configured.

[0120] The sizes of light distributions (X1,Y1) and (X2,Y2) are the same (X1=X2 and Y1=Y2). Each position (x,y) in (X1,Y1) can be zero to indicate that no light is projected for that position, or can be one to indicate that light is projected for that position. Each position (x,y) of (X2,Y2) can be zero to indicate that no light is projected for that position, or can be one to indicate that light is projected for that position. If (X1,Y1) and (X2,Y2) are complementary, then the value ((x2,y2)) at position (x,y) of (X2,Y2) may be the inverse of the value ((x1,y1)) at position (x,y) of (X1,Y1). For example, if (X1,Y1) and (X2,Y2) are complementary, and (x1,y1) is equal to 1, then (x2,y2) is equal to 0. Z can be conceptualized as an array (Z1, Z2), for which each (z1, z2) in (Z1, Z2) is equal to ((x1, y1), (x2, y2)).

[0121] T may include a component t1 associated with the projection (X1, Y1) and a component t2 associated with the projection (X2, Y2). Component t1 may indicate the duration that (X1, Y1) is projected, and component t2 may indicate the duration that (X2, Y2) is projected. The duration may be a duration associated with a frame capture or a total duration associated with multiple frame captures. For example, t1 may be the duration of projecting (X1, Y1) for frame capture 1, and t2 may be the duration of projecting (X2, Y2) for frame capture 2. In another example, t1 may be the average duration, median duration, or total duration of two or more frame captures 1, 3, 5, etc. t1 may be the average duration, median duration, or total duration of two or more frame captures 2, 4, 6, etc. In this way, a first light distribution (including t1) may be defined as (X1, Y1, t1), and a second light distribution (including t2) may be defined as (X2, Y2, t2). Therefore, determining (X1, Y1), (X2, Y2), and T includes determining (X1, Y1, t1) and (X2, Y2, t2) when optimizing light distribution.

[0122] As described above, optimizing the light distribution may include improving the resolution of the final depth map and / or the decoding success rate of the depth values ​​(while meeting equipment or regulatory requirements). In some embodiments, multiple candidate light distributions (such as multiple first light distributions (X1, Y1, t1) and their associated second light distributions (X2, Y2, t2)) are determined. The test scene is used to generate a final depth value set of the final depth map for each candidate, and the final depth value set (such as the final depth map) can be compared to determine the final first light distribution and the final second light distribution.

[0123] Fig.10 is an illustrative flow chart depicting exemplary operations 1000 for optimizing a projected light distribution for active depth sensing. For example, operations 1000 may be performed when optimizing (X1, Y1, t1) and (X2, Y2, t2). At 1002, device 300 ( Figure 3 ) can obtain one or more distribution parameters. The distribution parameters may include one or more parameters to configure a first light distribution (such as (X1, Y1, t1)) and a second light distribution (such as (X2, Y2, t2)). The one or more distribution parameters may be based on equipment limitations, standard requirements, safety regulations, or best practices.

[0124] With respect to device limitations, one or more components of device 300 may have physical limitations or requirements. For example, a VCSEL may have a maximum duty cycle (for when the VCSEL emits light) to prevent overheating. In this way, one or more distribution parameters (which may be used to determine t1 and t2 of the VCSEL) may be based on the maximum duty cycle. In another example, each image sensor pixel that receives a reflection of light emitted by the VCSEL is associated with the amount of light when the image sensor pixel is saturated. In this way, one or more distribution parameters (which may be used to determine t1 and t2 of the VCSEL) may be based on the amount of light associated with image sensor pixel saturation. Additionally, if reflections from multiple locations of (X1, Y1) or (X2, Y2) are received at an image sensor pixel, one or more distribution parameters (which may be used to determine (X1, Y1) or (X2, Y2)) may be based on the amount of light associated with image sensor pixel saturation. For example, if each position (x1, y1) in (X1, Y1) corresponds to a VCSEL in the VCSEL array, whether one or more VCSELs emit light during emission of a first light distribution or during emission of a second light distribution can be based on an amount of light associated with saturation of one or more image sensor pixels.

[0125] The device 300 may also be required to comply with one or more standards (such as a standard from the Institute of Electrical or Electronic Engineering (IEEE) or another standards body) or one or more regulations (such as regulations from a federal, state, provincial, international or other government agency). For example, for some wavelengths of emitted light, it may be required that the amount of light emitted from the VCSEL be less than the maximum energy within a certain time frame. In this way, the combination of the emission intensity and the emission duration of the VCSEL can be configured to be less than the maximum energy. A combination of multiple VCSELs (such as an entire VCSEL array or a portion of a VCSEL array) can also be configured so that the emission intensity and emission duration result in a total energy emitted that is less than the maximum energy over a period of time. In this way, how many VCSELs in a region emit light simultaneously and / or the duty cycle of each VCSEL emitting light in the region can be determined and adjusted to comply with any suitable standards and regulations, and the light distribution can be configured based on such determination and adjustment.

[0126] In addition to physical limitations, standard requirements, or regulatory requirements, the light distribution can also be based on one or more best practices. As used herein, best practices may include one or more concepts for designing light distribution to improve the performance of an active depth sensing system. In some embodiments, best practices may include configuring the light distribution so that the entire scene is evenly illuminated. Compared with uneven scene illumination, uniform illumination of the scene can allow codewords to be identified in more areas of reflection from the scene. For example, if the codeword is associated with only one light point (only one VCSEL emits light for the codeword), the position of the light point may be difficult to identify in the reflection, and the codeword may not be identified. On the contrary, if the codeword is associated with all possible light points except one (only one VCSEL will not emit light for the codeword), the position of the light point in the reflection may be difficult to identify, and the codeword may not be identified. For example, if the first light distribution includes a codeword of size 4x4, the first codeword is associated with only one VCSEL to be illuminated, and the second codeword is associated with 15 VCSELs to be illuminated, and it may be difficult to identify the first codeword and the second codeword in the reflection. The first light distribution can be configured so that each codeword of the first light distribution is associated with the minimum number of VCSELs that emit light. In some embodiments, the first light distribution can also be configured so that each codeword is associated with the maximum number of VCSELs that emit light. In this way, each codeword of the first light distribution is associated with a bounded range of VCSELs to emit light (such as 16 VCSELs per codeword, a range of 6 to 11 VCSELs, or another suitable range). If the second light distribution is complementary to the first light distribution (and has codewords of the same size as the first light distribution), each codeword of the second light distribution can also be associated with a bounded range of VCSELs to emit light. In this way, the scene can be evenly illuminated by both the first light distribution and the second light distribution.

[0127] In another example of best practice, a codeword associated with fewer light spots can be an adjacent codeword associated with more light spots in the light distribution. In this way, the position of the codeword relative to other codewords in the reflection can be based not only on the position of the identified light spots, but also on the number of identified light spots in the area of ​​the reflection. In another example of best practice, adjacent VCSELs for a codeword or for an adjacent codeword can be configured to not all emit for the light distribution. For example, for the same light distribution, VCSELs from a 3x3 portion of the VCSEL array may not all emit light. In this way, the light spots may not be concentrated to one or more areas of the light distribution.

[0128] Although some exemplary device restrictions, standard requirements, regulatory requirements, and best practices are provided as exemplary distribution parameters, other suitable device restrictions, standard requirements, regulatory requirements, and best practices may be used. Examples are provided to describe various aspects of the present disclosure, but distribution parameters are not limited to the examples provided. Thus, any suitable distribution parameters may be obtained and used.

[0129] In some embodiments, the distribution parameters may be classified as spatial parameters. As used herein, spatial parameters may refer to distribution parameters used to determine the positions of light points in a light distribution. Thus, spatial parameters may be used to configure X1 and Y1 of a first light distribution and to configure X2 and Y2 of a second light distribution. The number of light points per codeword and the position of the light points per codeword may be exemplary spatial parameters.

[0130] In some embodiments, the distribution parameters may be classified as parameters of the array. As used herein, the parameters of the array may refer to the distribution parameters used to determine the limits of the multiple light distributions based on the relationship between the multiple light distributions. For example, the parameter of the array may be that each VCSEL will emit light during at least one light distribution. If only two light distributions are emitted, it may be determined that the second light distribution is complementary to the first light distribution to ensure that each VCSEL emits light during one of the emitted light distributions.

[0131] In some embodiments, the distribution parameter may be classified as a time parameter. As used herein, a time parameter may refer to a distribution parameter used to determine the time for emitting a light distribution. Thus, the time parameter may be used to configure t1 for a first light distribution or to configure t2 for a second light distribution. As an example, if the emission intensity of the VCSEL does not change when emitting, the maximum energy limit for the VCSEL may be met based on the duration of the VCSEL emitting light. Thus, the time parameter may include the maximum amount of time that the VCSEL may emit light for a light distribution.

[0132] Distribution parameters may exist in multiple categories. For example, the maximum amount of time that a VCSEL can emit light for a light distribution (according to the above example of a time parameter) applies to each light distribution. Thus, the maximum amount of time may also be a parameter of the array. In another example, the maximum energy limit may also apply to an area covering the light distribution of multiple adjacent VCSELs. Thus, meeting the maximum energy limit may be based on the emission time as well as the emission position. Thus, the distribution parameters may be spatial parameters and temporal parameters.

[0133] Return to reference Fig.10, the device 300 may determine a light distribution to be projected based on one or more distribution parameters (1004). For example, the device 300 may configure a first light distribution (such as (X1, Y1, t1)) and a second light distribution (such as (X2, Y2, t2)) based on one or more distribution parameters. In some embodiments, the device 300 determines the light distribution based on one or more spatial parameters for the light distribution (1006). In some embodiments, the device 300 determines the light distribution based on one or more parameters of an array of multiple light distributions (1008). In some embodiments, the device 300 determines the light distribution based on one or more temporal parameters for the light distribution (1010).

[0134] In one example where the device 300 configures a first light distribution (e.g., (X1, Y1, t1)) and a second light distribution (e.g., (X2, Y2, t2)) based on one or more distribution parameters, the device 300 may use preset configurations of the first light distribution and the second light distribution. For example, a final (X1, Y1, t1) and (X2, Y2, t2) previously determined for another device, theoretically designed for the device 300, or determined in other ways may be provided to the device 300 or stored in the device 300 (such as in the memory 306). The provided or stored (X1, Y1, t1) and (X2, Y2, t2) may then be used to set an initial first light distribution and an initial second light distribution. At decision block 1012, if the light distribution satisfies the distribution parameters, the operation 1000 may proceed to step 1014. At 1014, the device 300 may store the light distribution as a candidate for a final light distribution.

[0135] In some embodiments of determining whether a light distribution satisfies distribution parameters, the device 300 may perform active depth sensing on the scene using the light distribution. For example, during calibration of the device 300, a test scene may be placed in the emission field of the light distribution. Active depth sensing may be performed, and the resulting depth values ​​may be viewed to determine whether each distribution parameter is satisfied. If a light distribution determined for another device is used, the light distribution may not be sufficient for the device 300 to satisfy the distribution parameters. For example, differences in lenses, image sensors, or other device components may cause the light distribution to not satisfy one or more distribution parameters. In another example, a previously determined light distribution may not be suitable for the current case. For example, if the light distribution is designed for an object that is farther away from the device 300 than an object in a scene (such as a test scene), it may be determined based on the depth difference that the light distribution does not satisfy one or more distribution parameters. Returning to reference decision box 1012, if the light distribution does not satisfy the distribution parameters, the device 300 may not store the light distribution as a candidate for the final light distribution.

[0136] In some embodiments, if the light distribution is a candidate for a final light distribution, the device 300 may use the light distribution for further active depth sensing without adjusting the light distribution. In this way, the first candidate light distribution may be the final light distribution for active depth sensing.

[0137] In some other embodiments, the light distribution may be adjusted to attempt to improve the performance of active depth sensing based on the scene, or the light distribution may be adjusted to attempt to make the light distribution satisfy the distribution parameters. For example, at 1016, the device 300 may adjust the light distribution based on one or more distribution parameters. The process may return to decision block 1012, and the adjusted light distribution may be used for active depth sensing to determine whether the adjusted light distribution is a candidate for the final light distribution.

[0138] Although not shown, the results of using the candidate light distributions can be compared to determine the final light distribution. For example, a depth map or depth values ​​determined using a first candidate can be compared to a depth map or depth values ​​determined using a second candidate. The candidate selected as the final light distribution can be based on any suitable criteria. For example, the final light distribution can be based on the candidate light distribution resulting in the least number of invalid depth values ​​(such as Fig. 9 In another example, the final light distribution can be based on the candidate light distribution resulting in invalid depth values ​​in the depth map that are less than a maximum threshold while providing a depth map with a resolution greater than a minimum threshold. The candidate used to generate the highest resolution depth map with the least number of invalid depth values ​​(below the maximum threshold) can be the final light distribution.

[0139] Referring back to step 1016, any suitable adjustment to the light distribution may be performed. In some embodiments, the adjustment to the light distribution may be a feedback system for generating one or more candidates. For example, the adjustment may be based on the current light distribution and one or more distribution parameters.

[0140] Fig.11 1 is a diagram of an exemplary feedback system 1100 for determining one or more candidates for a final light distribution. The process of the exemplary feedback system 1100 is described as being performed by the device 300 ( Figure 3 ). For example, the process may be performed by the light controller 310 or the processor 304. However, the process may be performed by any suitable device component. Alternatively, the process may be performed by another device. For example, if the device 300 is calibrated during manufacturing, a calibration device or a test device may perform the process to determine the final light distribution. Thus, the process of the exemplary feedback system 1100 is not limited to being performed by a specific device component or the device 300 itself.

[0141] At 1104, the device 300 may generate a light distribution based on one or more distribution parameters 1102. As used in the example, generating the light distribution includes determining which VCSELs of the VCSEL array will emit light during each light distribution and determining the time when each light distribution will be emitted. For example, for a first light distribution and a second light distribution to be generated, the device 300 may determine (X1, Y1, Z, t1) and (X2, Y2, Z, t2) (or (X1, Y1, t1) and (X2, Y2, t2), where Z is inferred from the determined variables). If this is the first time a light distribution is generated for the feedback system 1100, there may not be any feedback for a previous light distribution. In some embodiments, the initial light distribution may be a preconfigured set of distributions (such as a light distribution determined for another device or determined based on a simulation or theoretical operation of the device 300).

[0142] If the light distribution is after a previous light distribution, generating the light distribution at 1104 may also be based on feedback about the previous light distribution. In some embodiments, generating a light distribution based on feedback about a previous light distribution may include adjusting the previous light distribution. For example, one or more variables (X1, Y1, t1) or (X2, Y2, t2) of a previous distribution may be adjusted based on feedback and one or more distribution parameters 1102 to generate a light distribution at 1104. In some other embodiments, generating a light distribution at 1104 includes using a previous light distribution. The light distribution may then be adjusted at 1106 to 1110. In some other embodiments, the device 300 may generate a new light distribution based on feedback and one or more distribution parameters. For example, a second set of predefined distributions may be stored or received, and generating a light distribution at 1104 may include the device 300 using the second set of distributions. In some embodiments, the second set of distributions may be modified based on feedback and one or more distribution parameters 1102. In some other embodiments, additional sets of distributions may be stored or obtained. The device 300 may select one of the set as the light distribution based on the feedback and one or more distribution parameters 1102 when generating the light distribution at 1104. For example, if the time parameter (which may be indicated by the feedback) is not satisfied using the previous light distribution, the device 300 may select a distribution set having a different t1 or t2 than the previous light distribution.

[0143] At 1106, the device 300 may configure the generated light distribution based on one or more spatial parameters. For example, the device 300 may ensure that each codeword is associated with a bounded range of VCSEL emissions for each light distribution. If the device 300 identifies that any codeword is associated with a plurality of VCSELs that are outside the bounded range of emissions for the distribution, the device 300 may adjust the light distribution based on the identified codeword. In some embodiments, feedback on the previous distribution may also indicate whether the spatial parameters are not satisfied. If the spatial parameters are not satisfied for the previous light distribution, the device 300 may adjust the generated light distribution based on the previous non-satisfaction of the spatial parameters (such as adjusting X1 or Y1 for the first distribution or adjusting X2 or Y2 for the second distribution). If no adjustment is made based on one or more spatial parameters (such as the device 300 determines that the current light distribution should satisfy one or more spatial parameters, and determines that the previous light distribution satisfies one or more spatial parameters), configuring the distribution at 1106 may include not adjusting the light distribution generated at 1104 (such as retaining the previously generated distribution). For example, when configuring the distribution based on the spatial parameters at 1106, the device 300 may not adjust X1 and Y1 for the first distribution and may not adjust X2 and Y2 for the second distribution.

[0144] At 1108, the device 300 may configure the generated light distribution based on one or more parameters of the array. Similar to the description above for 1106, configuring at 1108 may include adjusting the distribution based on whether one or more parameters of the array were satisfied for a previous distribution (such as indicated by the feedback). Configuring at 1108 may also include adjusting the distribution based on whether one or more parameters of the array were not satisfied for the current distribution. If the device 300 determines based on the feedback and the one or more parameters 1102 that one or more parameters of the array are satisfied for the distribution as currently generated, configuring at 1108 may also include not adjusting the current light distribution (such as retaining the distribution generated at 1104 or the distribution adjusted at 1106).

[0145] At 1110, the device 300 may also configure the generated light distribution based on one or more time parameters. Similar to the description above for 1106 and 1108, configuring at 1110 may include adjusting the distribution based on whether one or more time parameters are satisfied for a previous distribution (such as indicated by feedback). Configuring at 1110 may also include adjusting the distribution based on whether one or more time parameters are not satisfied for the current distribution. If the device 300 determines based on the feedback and one or more parameters 1102 that one or more time parameters are satisfied for the distribution as currently generated, configuring at 1110 may also include not adjusting the current light distribution (such as retaining the distribution generated at 1104 or the distribution adjusted at 1106 or 1108). At 1112, the device 300 determines whether the current distribution satisfies one or more parameters 1102. As used herein, satisfying one or more parameters 1102 may be referred to as a qualified distribution, while not satisfying one or more parameters 1102 may be referred to as a failed distribution. 1112 may be similar to Fig.10 Decision box 1012 in . If the light distribution qualifies, the light distribution can be considered a candidate for a final light distribution 1116 for active depth sensing.

[0146] In some other embodiments of 1112, determining whether a light distribution is qualified or failed includes determining whether the active depth sensing system performs better using the current light distribution or a previous light distribution. For example, a depth map of a scene and corresponding to the current light distribution can be compared to a depth map of a scene and corresponding to a previous light distribution. If the corresponding depth map is identified as a better depth map, the current light distribution may be qualified. For example, if the depth map includes a smaller number of invalid depth values, the depth map may be better. In another example, if the resolution of the depth map is higher than less than a threshold number of invalid depth values, the depth map may be better. If the current light distribution is qualified, the depth map of the next light distribution can be compared to the depth map of the current light distribution at 1112. If the current light distribution fails, the depth map of the next light distribution can be compared to the depth map of the previous light distribution.

[0147] At 1114, the device 300 generates feedback based on the generated light distribution after the configuration at 1110. For example, at 1104, the device 300 may determine (X1, Y1, t1) and (X2, Y2, t2) as the light distribution. After the configuration at 1110, the adjusted light distribution may be (X1', Y1', t1') and (X2', Y2', t2'). The generated feedback may include variables (X1', Y1', t1') and (X2', Y2', t2').

[0148] In some embodiments, the device 300 may encode a sequence of variables as feedback. For example, the device 300 may encode a sequence indicating [X1', Y1', t1', X2', Y2', t2']. The encoded sequence may also indicate whether the light distribution passed or failed, whether any adjustments were made after generation (such as whether there is a difference between [X1, Y1, t1, X2, Y2, t2] and [X1', Y1', t1', X2', Y2', t2']), which distribution parameters caused the distribution to fail, or any other suitable information.

[0149] As described above, generating and adjusting the light distribution can be conceptualized as a multi-dimensional problem. For example, adjusting X1 or Y1 may affect t1, t2, X2, Y2, or Z. Similarly, adjusting different variables may affect other variables. Therefore, variables including spatial dimensions, time dimensions, and array dimensions are configured in a space, each dimension is related to other dimensions (such as a 4d space). Thus, configuring the light distribution at 1106 to 1110 can be performed simultaneously, sequentially, or in any suitable order based on the relationship between the dimensions to determine the variables (X1, Y1, t1) and (X2, Y2, t2) for the final light distribution. The final light distribution can then be used for active depth sensing, such as the above reference Figures 4 to 9 described.

[0150] Unless specifically described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be at least partially implemented by a non-transitory processor-readable storage medium (such as a processor) including instructions 308. Figure 3 The instructions are implemented in the memory 306 of the exemplary device 300, and when executed by the processor 304 (or the controller 310 or the image signal processor 312), the device 300 performs one or more of the above methods. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0151] Non-volatile processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0152] The various illustrative logical blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors such as Figure 3 The processor 304 or image signal processor 312 in the exemplary device 300 of the present invention is executed. This (such) processor may include, but is not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. The term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the technology described herein. In addition, in some aspects, the functionality described herein may be provided within a dedicated software module or hardware module configured as described herein. Similarly, the technology may be fully implemented in one or more circuits or logic elements. A general-purpose processor may be a microprocessor, but optionally, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration.

[0153] Although the present disclosure shows illustrative aspects, it should be noted that various changes and modifications may be made herein without departing from the scope of the appended claims. In addition, unless otherwise explicitly stated, the functions, steps or actions of the method claims according to the aspects described herein do not need to be performed in any particular order. For example, the steps of the described exemplary operations can be performed in any order and at any frequency when performed by the device 300, the controller 310, the processor 304 and / or the image signal processor 312. In addition, although an element may be described or claimed in the singular, plural forms are conceivable unless it is explicitly stated that it is limited to the singular. Therefore, the present disclosure is not limited to the examples shown, and any means for performing the functionality described herein are included in the various aspects of the present disclosure.

Claims

1. A device for active depth sensing, include: A projector, the projector being configured to: emitting a first light distribution during a first time; as well as emitting a second light distribution different from the first light distribution during a second time, wherein the projector is configured to alternate between emitting the first light distribution and emitting the second light distribution, wherein: The first light distribution is an original light distribution; The second light distribution is a dual light distribution; and The original light distribution and the dual light distribution are designed with reference to each other to allow the same portion of a frame captured by an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution; Image sensor, configured as: receiving one or more reflections of the first light distribution during capture of a first frame associated with the first time; and receiving one or more reflections of the second light distribution during capture of a second frame associated with the second time; and An image signal processor configured to: generating a first set of depth values ​​based on the first frame; generating a second set of depth values ​​based on the second frame; and The first set of depth values ​​and the second set of depth values ​​are combined to generate a final set of depth values ​​associated with one or more objects in the scene.

2. The device according to claim 1, in, The projector is also configured to emit a third light distribution different from the first light distribution and the second light distribution during a third time, wherein the final set of depth values ​​for the one or more objects in the scene is further based on one or more reflections of the third light distribution.

3. The device according to claim 1, in: The projector comprises a laser array; and For each laser in the laser array, the laser emits light during one or more of the first time or the second time.

4. The device according to claim 3, in, For each laser in the laser array, the laser emits light during one of the first time or the second time.

5. The device according to claim 1, in: The projector is configured to periodically alternate between emitting the first light distribution and emitting the second light distribution, wherein: Each emission of the first light distribution is associated with capture of one frame of a first set of frames by the image sensor; and each emission of the second light distribution is associated with capture of one frame of a second set of frames by the image sensor; The image signal processor is configured to: generating the first set of depth values ​​based on the first set of frames; and The second set of depth values ​​is generated based on the second set of frames.

6. The device according to claim 1, in: The first set of depth values ​​is included in a first depth map; The second set of depth values ​​is included in a second depth map, wherein a first position in the first depth map from a first depth value in the first set of depth values ​​corresponds to a second position in the second depth map from a second depth value in the second set of depth values; and Generating the final depth value set includes: determining whether the first depth value is a valid depth value; determining whether the second depth value is a valid depth value; and Based on only one of the first depth value or the second depth value being determined to be a valid depth value, a final depth value in the final depth value set is set as the valid depth value.

7. The device according to claim 6, in, Generating the final depth value set further includes setting the final depth value to one of the following based on both the first depth value and the second depth value being determined to be valid depth values: the first depth value; the second depth value; or The average value of the first depth value and the second depth value.

8. The device according to claim 1, further comprising: include: a processor configured to provide instructions to the image signal processor for execution; A memory configured to store the final depth map; as well as A display is configured to display the final depth map.

9. The device according to claim 1, in, The first light distribution and the second light distribution are determined in one or more spatial dimensions, time dimensions and array dimensions based on one or more dimensional parameters.

10. A method for active depth sensing, wherein include: emitting, by the projector, a first light distribution during a first time period; as well as emitting, by the projector, a second light distribution different from the first light distribution during a second time, wherein the projector is configured to alternate between emitting the first light distribution and emitting the second light distribution, wherein: The first light distribution is an original light distribution; The second light distribution is a dual light distribution; and The original light distribution and the dual light distribution are designed with reference to each other to allow the same portion of a frame captured by an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution; receiving, by an image sensor, one or more reflections of the first light distribution during capture of a first frame associated with the first time; receiving, by the image sensor, one or more reflections of the second light distribution during capture of a second frame associated with the second time; generating, by the image signal processor, a first set of depth values ​​based on the first frame; generating, by the image signal processor, a second set of depth values ​​based on the second frame; and The first set of depth values ​​and the second set of depth values ​​are combined to generate a final set of depth values ​​associated with one or more objects in the scene.

11. The method of claim 10, further comprising emitting, by the projector, a third light distribution different from the first light distribution and the second light distribution during a third time period, in, The final set of depth values ​​for the one or more objects in the scene is further based on one or more reflections of the third light distribution.

12. The method according to claim 10, further comprising: include: Each laser in the laser array of the projector emits light during one or more of the first time or the second time.

13. The method according to claim 12, further comprising: include: Light is emitted by each laser in the laser array during one of the first time or the second time.

14. The method according to claim 10, further comprising: include: and periodically alternating between emitting the first light distribution and emitting the second light distribution, wherein: Each emission of the first light distribution is associated with capture of one frame of a first set of frames by the image sensor; and each emission of the second light distribution is associated with capture of one frame of a second set of frames by the image sensor; generating the first set of depth values ​​based on the first set of frames; and The second set of depth values ​​is generated based on the second set of frames.

15. The method according to claim 10, in: The first set of depth values ​​is included in a first depth map; The second set of depth values ​​is included in a second depth map, wherein a first position in the first depth map from a first depth value in the first set of depth values ​​corresponds to a second position in the second depth map from a second depth value in the second set of depth values; and Generating the final depth value set includes: determining whether the first depth value is a valid depth value; determining whether the second depth value is a valid depth value; and Based on only one of the first depth value or the second depth value being determined to be a valid depth value, a final depth value in the final depth value set is set as the valid depth value.

16. The method according to claim 15, in, Generating the final depth value set further includes setting the final depth value to one of the following based on both the first depth value and the second depth value being determined to be valid depth values: the first depth value; the second depth value; or The average value of the first depth value and the second depth value.

17. The method according to claim 10, in, The first light distribution and the second light distribution are determined in one or more spatial dimensions, time dimensions and array dimensions based on one or more dimensional parameters.

18. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device performing active depth sensing, cause the device to: emitting, by the projector, a first light distribution during a first time period; A second light distribution different from the first light distribution is emitted by the projector during a second time period, wherein: Execution of the instructions further causes the device to alternate between emitting the first light distribution and emitting the second light distribution by the projector, wherein: The first light distribution is an original light distribution; The second light distribution is a dual light distribution; and The original light distribution and the dual light distribution are designed with reference to each other to allow the same portion of a frame captured by an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution; receiving, by an image sensor, one or more reflections of the first light distribution during capture of a first frame associated with the first time; receiving, by the image sensor, one or more reflections of the second light distribution during capture of a second frame associated with the second time; generating, by the image signal processor, a first set of depth values ​​based on the first frame; generating, by the image signal processor, a second set of depth values ​​based on the second frame; and The first set of depth values ​​and the second set of depth values ​​are combined to generate a final set of depth values ​​associated with one or more objects in the scene.

19. The computer readable medium of claim 18, in, Execution of the instructions further causes the device to: A third light distribution different from the first light distribution and the second light distribution is emitted by the projector during a third time, wherein the final set of depth values ​​for the one or more objects in the scene is further based on one or more reflections of the third light distribution.

20. The computer readable medium of claim 18, in, Execution of the instructions further causes the device to: and periodically alternating between emitting the first light distribution and emitting the second light distribution, wherein: Each emission of the first light distribution is associated with capture of one frame of a first set of frames by the image sensor; and each emission of the second light distribution is associated with capture of one frame of a second set of frames by the image sensor; generating the first set of depth values ​​based on the first set of frames; and The second set of depth values ​​is generated based on the second set of frames.

21. The computer readable medium of claim 18, in: The first set of depth values ​​is included in a first depth map; The second set of depth values ​​is included in a second depth map, wherein a first position in the first depth map from a first depth value in the first set of depth values ​​corresponds to a second position in the second depth map from a second depth value in the second set of depth values; and Generating the final depth value set includes: determining whether the first depth value is a valid depth value; determining whether the second depth value is a valid depth value; and Based on only one of the first depth value or the second depth value being determined as a valid depth value, a final depth value in the final depth value set is set as the valid depth value.

22. The computer readable medium of claim 18, in, The first light distribution and the second light distribution are determined in one or more spatial dimensions, time dimensions and array dimensions based on one or more dimensional parameters.

23. A device for active depth sensing, include: means for emitting a first light distribution during a first time; means for emitting a second light distribution different from the first light distribution during a second time, wherein the device is further configured to alternate between emitting the first light distribution and emitting the second light distribution, wherein: The first light distribution is an original light distribution; The second light distribution is a dual light distribution; and The original light distribution and the dual light distribution are designed with reference to each other to allow the same portion of a frame captured by an image sensor to be complementary decoded into codewords from the original light distribution and from the dual light distribution; means for receiving one or more reflections of said first light distribution during capture of a first frame associated with said first time; means for receiving one or more reflections of said second light distribution during capture of a second frame associated with said second time; means for generating a first set of depth values ​​based on the first frame; means for generating a second set of depth values ​​based on the second frame; and Means for combining the first set of depth values ​​and the second set of depth values ​​to generate a final set of depth values ​​associated with one or more objects in a scene.

Images