Active depth sensing based on spots

Abstract

Systems and techniques are described herein for determining depths. For instance, a method for determining depths is provided. The method may include obtaining an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determining a pixel of the image that corresponds to a spot of the plurality of spots in the image; determining a dot of the plurality of dots of the pattern that corresponds to the pixel; and determining a depth for the spot based on the dot and the spot.

Description

ACTIVE DEPTH SENSING BASED ON SPOTSTECHNICAL FIELD

[0001] The present disclosure generally relates to active depth sensing. For example, aspects of the present disclosure include systems and techniques for active depth sensing based on spots.BACKGROUND

[0002] An active depth-sensing device may project patterned light (e.g., a pattern) into a scene and capture an image of the pattern as projected onto the scene. The active depth-sensing system may determine depths to various points in the scene (e.g., distances between the active depth-sensing system and the various) based on the appearance of the pattern in the image. For example, the pattern may include a grid of dots. The dots may appear as spots in the image. How close the spots in the image are to one another at a given point in the image may relate to the distance between the active depth-sensing system and a point in the scene corresponding to the point in the image.SUMMARY

[0003] The following presents a simplified summary relating to one or more aspects disclosed herein. Thus, the following summary should not be considered an extensive overview relating to all contemplated aspects, nor should the following summary be considered to identify key or critical elements relating to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts relating to one or more aspects relating to the mechanisms disclosed herein in a simplified form to precede the detailed description presented below.

[0004] Systems and techniques are described for determining depths. According to at least one example, a method is provided for determining depths. The method includes: obtaining an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determining a pixel of the image that corresponds to a spot of the plurality of spots in the image; determining a dot of the plurality of dots of the pattern that corresponds to the pixel; and determining a depth for the spot based on the dot and the spot.

[0005] In another example, an apparatus for determining depths is provided that includes at least one memory and at least one processor (e.g., configured in circuitry) coupled to the at least one memory. The at least one processor configured to: obtain an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determine a pixel of the image that corresponds to a spot of the plurality of spots in the image; determine a dot of the plurality of dots of the pattern that corresponds to the pixel; and determine a depth for the spot based on the dot and the spot.

[0006] In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: obtain an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determine a pixel of the image that corresponds to a spot of the plurality of spots in the image; determine a dot of the plurality of dots of the pattern that corresponds to the pixel; and determine a depth for the spot based on the dot and the spot.

[0007] In another example, an apparatus for determining depths is provided. The apparatus includes: means for obtaining an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; means for determining a pixel of the image that corresponds to a spot of the plurality of spots in the image; means for determining a dot of the plurality of dots of the pattern that corresponds to the pixel; and means for determining a depth for the spot based on the dot and the spot.

[0008] In some aspects, one or more of the apparatuses described herein is, can be part of, or can include an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle (or a computing device, system, or component of a vehicle), a mobile device (e.g., a mobile telephone or so-called “smart phone”, a tablet computer, or other type of mobile device), a smart or connected device (e.g., an Intemet-of-Things (loT) device), a wearable device, a personal computer, a laptop computer, a video server, a television (e.g., a network- connected television), a robotics device or system, or other device. In some aspects, each apparatuscan include an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, each apparatus can include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, each apparatus can include one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, each apparatus can include one or more sensors. In some cases, the one or more sensors can be used for determining a location of the apparatuses, a state of the apparatuses (e.g., a tracking state, an operating state, a temperature, a humidity level, and / or other state), and / or for other purposes.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative examples of the present application are described in detail below with reference to the following figures:

[0012] FIG. l is a depiction of an example active depth-sensing system configured to use a pattern of dots for determining depths of objects in a scene, according to various aspects of the present disclosure;

[0013] FIG. 2 is a block diagram illustrating an example system including a device and a projector configured to emit patterns of light for active depth sensing, according to various aspects of the present disclosure;

[0014] FIG. 3 is a diagram illustrating an example of a simplified structured-light system, according to various aspects of the present disclosure;

[0015] FIG. 4 is a diagram illustrating an example of a 24x36 coded primitive pattern of light generated by an array of light-emitting sources and collimated using a lens;

[0016] FIG. 5 A is a diagram illustrating a projected pattern of light;

[0017] FIG. 5B is a diagram illustrating a tessellated coded pattern of light;

[0018] FIG. 6 is a diagram of an example system that may determine depths of a scene, according to various aspects of the present disclosure;

[0019] FIG. 7 is a includes an example representation of spots, according to various aspects of the present disclosure;

[0020] FIG. 8 is a includes an example representation of several images of spots, according to various aspects of the present disclosure;

[0021] FIG. 9 is a includes an example representation of several images of spots, according to various aspects of the present disclosure;

[0022] FIG. 10A includes an example representation of 21 images of spots corresponding to a single dot, according to various aspects of the present disclosure;

[0023] FIG. 10B includes another example representation of the 21 images of the spots of FIG. 10A, according to various aspects of the present disclosure;

[0024] FIG. 11 includes an example histogram that may represent a number of pixels that capture various numbers of spots through a relevant range of depths, according to various aspects of the present disclosure;

[0025] FIG. 12A includes an example representation of 21 images of spots corresponding to three dots, according to various aspects of the present disclosure;

[0026] FIG. 12B includes another example representation of the 21 images of the three spots of FIG. 12A, according to various aspects of the present disclosure;

[0027] FIG. 13 is a diagram of a system 1300 for determining a relationship between pixels and depths, according to various aspects of the present disclosure;

[0028] FIG. 14 is a flow diagram illustrating another example process for active depth sensing based on spots, in accordance with aspects of the present disclosure;

[0029] FIG. 15 is a block diagram illustrating an example computing-device architecture of an example computing device which can implement the various techniques described herein.DETAILED DESCRIPTION

[0030] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0031] The ensuing description provides example aspects only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the exemplary aspects will provide those skilled in the art with an enabling description for implementing an exemplary aspect. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0032] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term “aspects of the disclosure” does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0033] Various systems and / or applications make use of three-dimensional (3D) information representing a scene, such as systems and / or applications that perform face recognition, authentication systems that uses a subject’s face identification (ID), object scanning, autonomous driving, robotics navigation and / or object detection, aviation navigation (e.g., for unmanned aerial vehicles, airplanes, among others), indoor navigation, augmented reality (AR), 3D scene understanding, object grasping, object tracking, among other tasks. Recent needs to capture 3D information from a scene (e.g., forface ID, object scanning, autonomous driving, AR applications, among others) have created a high demand for active depth sensing technologies.

[0034] Active depth-sensing technology (e.g., structured-light technology) is one example that offers a reliable and high-quality depth capture system. For example, a structured-light system can include a structured-light sensor or other device for scanning and / or determining the dimensions and / or movement of a scene and / or one or more objects (e.g., a person, a device, an animal, a vehicle, etc.) in the scene. The structured-light sensor or device can project a known shape or pattern of light onto the scene including the one or more objects and can determine the dimensions and / or movement of the scene (e.g., the dimensions and / or movement of the one or more objects within the scene) based on measured or detected deformations of the shape or pattern.

[0035] In some cases, an active depth-sensing system (e.g., a structured-light system) can project a configurable pattern of light. For example, a structured-light system can include at least one transmitter and at least one receiver. A transmitter of the structured-light system can project or transmit a pattern of dots onto a target object. The projected light can include a plurality of dots (e.g. light points or other shapes), and in some cases can be focused into any suitable size and dimensions. For example, the light may be projected in lines, squares, or any other suitable shape and / or dimension. As noted above, a structured-light system can act as a depth sensing system that can be used to generate a depth map of a scene.

[0036] In some example implementations, the light projected by the transmitter of an active depthsensing system (e.g., a structured-light system) can be infrared (IR) light. IR light can include portions of the visible light spectrum (e.g., near-infrared (NIR) light) and / or portions of the light spectrum that are not visible to the human eye (e.g., IR light outside of the NIR spectrum). For instance, IR light may include NIR light, which may or may not include light within the visible light spectrum. In some cases, other suitable wavelengths of light may be transmitted by the active depth-sensing system (e.g., the structured-light system). For example, light can be transmitted by the active depth-sensing system in the ultraviolet light spectrum, the microwave spectrum, radio frequency spectrum, visible light spectrum, and / or other suitable light signals. In some cases, the light can be transmitted through an optical element (e.g., a lens and / or other element).

[0037] Many structured-light system are based on a single light (e.g., a single laser) that forms the desired coded light pattern. Other structured-light systems include an array-based structured-light system that uses an array of lights (e.g., an array of lasers). For instance, an array based structured- light system can include an array of a certain number of light emitting lasers, with each light emitting laser emitting IR light (e.g., in response to an electrical drive signal applied to an electrical input of each of the light emitting lasers). One example of a multi-light structured-light system is a Verticalcavity surface-emitting laser (VCSEL) array. For instance, a VCSEL array can include an array of VCSELs, with each VCSEL emitting IR light. Additionally or alternatively, some structured-light systems may include a diffractive optical element (DOE) that may split light from a source (such as a laser) into a pattern.

[0038] Many structured-light systems apply pattern-matching techniques to determine a disparity between the pattern as it is projected into a scene and the pattern as it appears in an image of the scene. For example, a structured-light system may project dots of a pattern into a scene and capture an image of the scene. The dots of the pattern may appear as spots in the image. In the present disclosure, the term “dot” may refer to a point of light transmitted by a light source (e.g., a laser). The light source may include an array of light sources of a structured-light system configured to transmit a pattern of dots. A dot can be of any shape (including circles, squares, triangles, lines, etc.) and a pattern of dots can include any number of dots arranged an any suitable arrangement. For descriptive purposes, in the present disclosure, the term “spot” may refer to a number of pixels with a respective number of intensity values (e.g., a 7x7 array of pixels, each with a respective intensity value) in an image of the scene captured by an image sensor that receives a reflected pattern of dots (e.g., reflected from one or more objects in the scene). For example, intensities of pixels included in a spot can correspond to a particular dot projected into the scene. In order to determine how spots (in the image of the scene) correspond to dots (of the pattern as projected), many structured-light systems may use a patternmatching technique to correlate dots with spots. After corelating the spots with the dots, the structured- light systems may use geometric techniques to determine a depth to points in the scene that reflected the spots. The geometric techniques may use the position of the dots in the pattern and the position of the spots in the image to determine the depths. Thus, it may be important that the correlation between the dots and the spots is correct.

[0039] Pattern matching may be relatively computationally intense, for example, using many computing operations requiring time and / or power consumption. For example, pattern matching may involve comparing a window (e.g., a portion of a certain size) of the pattern with a number of windows of the image to determine a correspondence between dots and spots. Pattern matching may involve many such comparisons, and each may involve several calculations. Further, pattern matching may work best when objects reflecting the dots reflect enough of the dots for a pattern to be matched. For example, a relatively small object may reflect few dots as spots which may make matching the few spots to dots of the pattern difficult or impossible.

[0040] Systems, apparatuses, methods (also referred to as processes), and computer-readable media (collectively referred to herein as “systems and techniques”) are described herein for active depth sensing based on spots in images captured using an active depth-sensing system (e.g., a structured- light system). For example, the systems and techniques described herein may obtain an image of a scene that is illuminated with a pattern of dots. The image may include a number of spots. Each of the spots may correspond to a dot of the pattern of dots. The systems and techniques may, for each spot captured in the image, determine a pixel of the image that corresponds to the spot, determine a dot of the pattern that corresponds to the pixel, and determine a depth corresponding to the pixel based on the dot and the pixel.

[0041] For example, to determine the pixel of the image that corresponds to the spot, the systems and techniques may determine a position of a centroid of the spot based on values of pixels of the spot and determine the pixel based on the centroid. For example, the systems and techniques may determine a window or array of pixels representative of the spot (e.g., a window of pixels around the spot, such as a 3x3 window of pixels, a 5x5 window of pixels, etc.). In some aspects, the systems and techniques may determine a centroid of the window based on moments of values of the pixels of the window. The centroid may have a sub-pixel resolution. The systems and techniques may determine that the pixel that includes the centroid is the pixel of the image that corresponds to the spot and / or the pixel that represents the spot.

[0042] Having determined the pixel of the image that corresponds to the spot (based on the centroid of the spot being in the pixel), the systems and techniques may determine a dot of the pattern that corresponds to the pixel. For example, the systems and techniques may determine the dot based on an association between pixels and dots. The association may relate pixels of an image (or pixels of adetector) with dots of a pattern. For example, for each pixel of an image (or detector), the association may include a relation between the pixel and one or more dots that may include centroids of spots in the pixel. For instance, a given pixel may capture a centroid of a spot of a single dot as reflected by an object at a particular depth. The given pixel may be related to the single dot and the relationship may be stored in the association. As another example, a given pixel may capture a centroid of a spot of a first dot as reflected by an object at a first depth and may capture a centroid of a spot of a second dot as reflected by an object at a second depth. The given pixel may be related to the first dot and the second dot, and the relationships may be stored in the association. In some cases, the association may be stored in a look-up table (LUT).

[0043] In some cases, the association may be determined through a calibration process. For example, an active depth-sensing system (that is substantially similar to the active depth-sensing system of the systems and techniques) may project a pattern (that is substantially similar to the pattern projected by the systems and techniques) and capture a number of images using an image-capture system (that is substantially similar to the image-capture system of the systems and techniques) as a reflective surface is positioned a number of distances from the active depth-sensing system. For each pixel, across each of the number of images, all dots that result in a centroid of a spot captured by the pixel may be stored as a relationship between the pixel and the dots.

[0044] For example, in a first image a first number of pixels may represent spots (that correspond to respective dots). Centroids may be determined for each of the spots. Pixels including each of the centroids may be related to the corresponding dots and the relationships may be stored in the association. Further, in a second image captured at a distance from the surface that is different from the distance from which the first image was captured, a second number of pixels may represent spots (that correspond to respective dots). Centroids of the spots may be determined. Pixels including the centroids may be identified. The pixels of the second image that include centroids may be different than the pixels of the first image that include pixels based on the position of the spots being different in the second image than the positions of the spots in the first image based on the distance from the surface being different between the second image and the first image. The pixels of the second image that include centroids may be related to the corresponding dots and the relationships may be stored in the association.

[0045] Further, in some aspects, after capturing a number of images at a number of respective distances, a smooth fitting line may be drawn through and close to the centroids of the spots. This line describes expected positions of all possible centroids of spots, corresponding to all possible distances, for a given dot. The line may be mathematically represented by a polynomial. The polynomial may be stored as the association.

[0046] In some instances, a pixel may be related to a number of dots. For example, a number of dots, as captured at different depths, may result in spots that may be captured by the pixel. In such instances, the association may be between one pixel and multiple dots. In such cases, the systems and techniques may distinguish between the number of dots based on a neighborhood of dots and a neighborhood of spots. For example, the systems and techniques may use a pattern-matching technique to determine which one of the multiple dots the pixel corresponds to in a particular captured image.

[0047] Having determined a dot of the pattern that corresponds to the pixel, the systems and techniques may determine a depth corresponding to the centroid of the spot based on the dot and the centroid of the spot. For example, the systems and techniques may determine a depth corresponding to the centroid of the spot based on a relationship between centroids of spots and depths. The relationship may be specific to the dot. The relationship may be determined through a calibration process, for example, as part of the same calibration process described above. For example, for each pixel that includes a centroid of a spot corresponding to a dot, a depth of the point in the scene corresponding to the spot may be stored. During the calibration process, the relationship between the depths and the centroids for a given dot may be determined based on the depths and centroids measured during the calibration process. The relationship may be, or may include, a polynomial that describes the depth of a point corresponding to a dot as a function of a position of centroid of a spot corresponding to the dot. For example, for each dot, a polynomial equation describing depth as a function of centroid positions may be determined. Although the calibration process provides many pairs of centroid-depth relationships, the calibration process may not cover all possible such relationships (for example, the calibration may include capturing images of spots at a discrete number of relative depths). The spot of each dot may travel through many pixels not measured during the calibration process. Nevertheless, the polynomial may provide depths between and beyond the depths measured during calibration. For example, centroid positions of spots from a dot may be measured every 20 centimeters of relative depth from 1 meter to 3 meters of depth. A polynomial determinedbased on the measured spots and depths may be continuous and may be capable of providing a depth based on any depth (including any depth between 1 and 3 meters, e.g., 3 meters and 7 centimeters and including depths less than 1 meter away and greater than 3 meters away).

[0048] In this way, the systems and techniques may match individual spots with corresponding dots and determine depths based on the spots. In many cases, such matching may not require matching of a neighborhood of spots with a neighborhood of dots. Thus, the systems and techniques may involve fewer instances of using computationally-intensive pattern matching. Further, in cases in which the systems and techniques use pattern-matching techniques, the systems and techniques may limit the candidates based on the association between pixels and dots such that the pattern-matching techniques are faster and less computationally intensive. Accordingly, the systems and techniques may match spots with dots in a way that is faster, involves fewer calculations, and uses less power than other active depth-sensing techniques. Further, matching individual spots with dots, according to various aspects of the present disclosure, may allow smaller objects to be resolved by the systems and techniques than are resolvable by other active depth-sensing systems because the systems and techniques may rely less on a neighborhood of spots to correlate with corresponding neighborhoods of dots.

[0049] Additionally or alternatively, the systems and techniques may be able to determine depths based on a single image whereas other depth-determination techniques may use multiple images (e.g., to average between multiple images). Further, the systems and techniques may have a sub-pixel disparity resolution which may result in more accurate depth measurements and / or depth measurements with a higher depth resolution. For example, the systems and techniques may determine a position of a centroid of a spot to determine a depth corresponding to the spot. The centroid can be determined to a sub-pixel level (e.g., xC = 2.2 and yC = 2.7), in consequence, the depth obtained also provide higher resolution than using integer-specified spot position (e.g., xC = 2 and yC = 3). Additionally or alternatively, the systems and techniques may result in little or no depth blurring at edges of objects. Further, the systems and techniques may allow accurate depth sensing even for slanted surfaces. For example, many other depth-sensing techniques use patches consisting multiple “spots” with pattern recognition and / or template matching to perform depth determination. When the dot patterns are projected onto slanted or curved surfaces, the corresponding received spot patterns are distorted. The distortion can also happen when such patterns of spots are partially disappeared orchanged due to the depth edges. Such distortion can reduce the depth sensing sensitivity and / or introduce depth errors based on a difficulty in matching the patterns of dots with the distorted patterns of spots on the slanted or curved surfaces. However, the systems and techniques can determine a depth of a point based on one spot, which cuts such distortions to the minimum. Additionally or alternatively, the systems and techniques may determine depths based on the position of a centroid of the spot; unlike other techniques, such as iToF depth sensing, which uses spot intensity or ratio of intensities of different channels to determine depth. The systems and techniques use the centroid of spot to determine the depth, is much more robust to the temporal intensity fluctuation, the environment lighting changing and other noises. Thus, the systems and techniques be robust to intensity variations, ambient light, and multi-path interference. Further the systems and techniques may be temporally and / or spatially stable. Additionally or alternatively, the systems and techniques may be easily combined with time-of-flight depth measurement techniques.

[0050] Various aspects of the application will be described with respect to the figures below.

[0051] FIG. 1 is a depiction of an example active depth-sensing system 100 (which may be referred to alternatively as a structured-light system) configured to use a pattern 104 of dots for determining depths of objects 106A and 106B in a scene 106, according to various aspects of the present disclosure. Active depth-sensing system 100 may be used to generate a depth map (not pictured) of a scene 106. For example, the scene 106 may include an object (e.g., a face), and the active depth sensing active depth-sensing system 100 may be used to generate a depth map including a plurality of depth values indicating depths of portions of the object for identifying or authenticating the object (e.g., for face authentication). Active depth-sensing system 100 includes a projector 102 and a receiver 108. Projector 102 may be referred to as a “structured light source”, “transmitter,” “emitter,” “light source,” or other similar term, and should not be limited to a specific transmission component. Throughout the following disclosure, the terms projector, transmitter, and light source may be used interchangeably. Receiver 108 may be referred to as a “detector,” “sensor,” “sensing element,” “photodetector,” and so on, and should not be limited to a specific receiving component.

[0052] Projector 102 may be configured to project or transmit a pattern 104 of dots (e.g., light points or shapes) onto scene 106. The white circles in pattern 104 indicate where no light is projected, and the black circles in pattern 104 indicate where light is projected. The disclosure may alternatively referto the pattern 104 as a codeword distribution or a distribution, where defined portions of the pattern 104 are codewords (also referred to as codes).

[0053] Projector 102 includes one or more light sources 124 (such as one or more lasers). In some implementations, the one or more light sources 124 includes a laser array. In one illustrative example, each laser may be a vertical cavity surface emitting laser (VCSEL). In another illustrative example, each laser may include a distributed feedback (DFB) laser. In another illustrative example, the one or more light sources 124 may include a resonant cavity light emitting diodes (RC-LED) array. In some implementations, the projector may also include a lens 126 and a light modulator 128. Projector 102 may also include an aperture 122 from which the transmitted light escapes projector 102. In some implementations, projector 102 may further include a diffractive optical element (DOE) to diffract the emissions from one or more light sources 124 into additional emissions. In some aspects, the light modulator 128 (which may adjust the intensity of the emission) may include a DOE.

[0054] In projecting pattern 104 of dots onto scene 106, projector 102 may transmit one or more lasers from light source 124 through lens 126 (and / or through a DOE and / or light modulator 128) and onto objects 106A and 106B in scene 106. Projector 102 may be positioned on the same reference plane as receiver 108, and projector 102 and receiver 108 may be separated by a known distance, which may be referred to as baseline 112.

[0055] In some implementations, the light projected by projector 102 may be infrared (IR) light. IR light may include portions of the visible light spectrum and / or portions of the light spectrum that is not visible to the naked eye. In one example, IR light may include near infrared (NIR) light, which may or may not include light within the visible light spectrum, and / or IR light (such as far infrared (FIR) light) which is outside the visible light spectrum. The term IR light should not be limited to light having a specific wavelength in or near the wavelength range of IR light. Further, IR light is provided as an example emission from the projector. In the following description, other suitable wavelengths of light may be used. For example, light in portions of the visible light spectrum outside the IR light wavelength range or ultraviolet (UV) light may be used.

[0056] Scene 106 may include objects at different depths from active depth-sensing system 100 (such as from projector 102 and receiver 108). For example, objects 106A and 106B in scene 106 may be at different depths. Receiver 108 may be configured to receive, from scene 106, reflections 110 ofthe transmitted pattern 104 of dots. To receive reflections 110, receiver 108 may capture a frame. When capturing the frame, receiver 108 may receive reflections 110, as well as (i) other reflections of pattern 104 of dots from other portions of scene 106 at different depths, (ii) ambient light, and (iii) noise. In the present disclosure, the terms “frame” and “image” may be used interchangeably to refer to what is captured by receiver 108. The frame, or image, may or may not be, or include, a visible image but may rather include intensity values including intensities of reflections 110. The intensity values may be based on reflections 110 of visible light, IR light, or UV light.

[0057] In some implementations, receiver 108 may include a lens 130 to focus or direct the received light (including reflections 110 from the objects 106A and 106B) on to a sensor 132 of receiver 108. Receiver 108 also may include an aperture 120. Assuming for the example that only reflections 110 are received, depths of the objects 106A and 106B (e.g., distances between projector 102 or receiver 108 and objects 106A and 106B respectively) may be determined based on baseline 112 and displacement and distortion of dots of pattern 104 in reflections 110. In some cases, an intensity of reflections 110 may also be used to determine depths of objects 106A and 106B. For example, a distance 134 along sensor 132 from position 116 to a center 114 of sensor 132 may be used in determining a depth of object 106B in scene 106. Similarly, a distance 136 along sensor 132 from a position 118 to center 114 may be used in determining a depth of object 106A in scene 106. The distance along sensor 132 may be measured in terms of number of pixels of sensor 132 or a unit of distance (such as millimeters).

[0058] In some implementations, sensor 132 may include an array of photodiodes (such as avalanche photodiodes) for capturing a frame. To capture the frame, each photodiode in the array may capture the light that hits the photodiode and may provide a value indicating the intensity of the light (a capture value). The frame therefore may be an array of capture values provided by the array of photodiodes. In addition or alternative to sensor 132 including an array of photodiodes, sensor 132 may include a complementary metal-oxide semiconductor (CMOS) sensor. To capture the image by a photosensitive CMOS sensor, each pixel of the sensor may capture the light that hits the pixel and may provide a value indicating the intensity of the light. In some example implementations, an array of photodiodes may be coupled to the CMOS sensor. In this manner, the electrical impulses generated by the array of photodiodes may trigger the corresponding pixels of the CMOS sensor to provide capture values.

[0059] Sensor 132 may include at least a number of pixels equal to the number of possible dots in pattern 104. For example, the array of photodiodes or the CMOS sensor may include at least a number of photodiodes or a number of pixels, respectively, corresponding to the number of possible dots in pattern 104. In some implementations, sensor 132 may include more pixels than the number of possible dots of pattern 104. For example, in some cases, sensor 132 may include five or ten times as many pixels as pattern 104 includes dots. If light source 124 transmits IR light (such as NIR light at a wavelength of, e.g., 940 nanometers (nm)), sensor 132 may be an IR sensor to receive the reflections of the NIR light.

[0060] As illustrated, distance 134 (corresponding to a reflection 110 from object 106B) is less than distance 136 (corresponding to a reflection 110 from object 106A). Using triangulation based on baseline 112 and distance 134 and distance 136, the differing depths of objects 106A and 106B in scene 106 may be determined and a depth map of scene 106 may be generated. Determining the depths may further be based on a displacement or a distortion of pattern 104 in reflections 110.

[0061] In some implementations, projector 102 may be configured to project a fixed light distribution, in which case the same distribution of light is used in every instance for active depth sensing. In some implementations, projector 102 may be configured to project a different pattern of light at different times. For example, projector 102 may be configured to project a first pattern of light at a first time and project a second pattern of light at a second time. A resulting depth map of one or more objects in a scene may thus be based on one or more reflections of the first pattern of light and one or more reflections of the second pattern of light.

[0062] Although a number of separate components are illustrated in FIG. 1, one or more of the components may be implemented together or include additional functionality. All described components may not be required for active depth-sensing system 100, or the functionality of components may be separated into separate components. Additional components not illustrated also may exist. For example, receiver 108 may include a bandpass filter to allow signals having a determined range of wavelengths to pass onto sensor 132 (thus filtering out signals with a wavelength outside of the range). In this manner, some incidental signals (such as ambient light) may be prevented from being received as interference during the captures by sensor 132. The range of the bandpass filter may be centered at the transmission wavelength for projector 102. For example, if projector 102 is configured to transmit NIR light with a wavelength of 940 nm, receiver 108 may include a bandpassfilter configured to allow NIR light having wavelengths within a range of, for example, 920 nm to 960 nm. Therefore, the examples described regarding FIG. 1 is for illustrative purposes.

[0063] FIG. 2 is a block diagram illustrating an example system 200 including a device 202 and a projector 204 configured to emit patterns of light for active depth sensing, according to various aspects of the present disclosure. In some examples, the projector 204 can be part of the device 202. In other examples, projector 204 may be separate from device 202. Similarly, device 202 may include, or be coupled to, a receiver 206. Receiver 206 may be separated from projector 204 by a baseline 208. Receiver 206 may be an IR sensor configured to capture frames, and projector 204 may be a projector configured to project patterns of light.

[0064] Device 202 may include a processor 210, a memory 212 storing instructions 214, and a light controller 218 (which may include one or more image signal processors 220). Device 202 may optionally include (or be coupled to) a display 222 and a number of input / output (I / O) components 224. Device 202 may include additional features or components not shown. For example, a wireless interface, which may include a number of transceivers and a baseband processor, may be included for a wireless communication device to perform wireless communications. In another example, device 202 may include one or more cameras (such as a contact image sensor (CIS) camera or other suitable camera for capturing images using visible light). Projector 204 and receiver 206 may be part of an active depth sensing system (such as the system 100 in FIG. 1). Projector 204 and receiver 206 may be controlled by light controller 218 and / or processor 210. Device 202 may include or be coupled to additional light projectors (or a flood illuminator) or may include a different configuration for the light projectors. Device 202 may also include or be coupled to additional receivers (not shown) for capturing multiple frames of a scene. The disclosure is not to be limited to any specific examples or illustrations, including the example device 202.

[0065] Memory 212 may be a non-transient or non-transitory computer readable medium storing computer-executable instructions 214 to perform all or a portion of one or more operations described in this disclosure. According to various aspects of the present disclosure, memory 212 may store associations (e.g., between pixels and dots) and / or relationships (e.g., between pixels and depths). Further, memory 212 may store a representations of neighborhoods of dots or representations of patterns. If the pattern projected by projector 204 is divided into codewords, memory 212 may optionally store a library of codewords 216 for the distribution of light including the plurality ofcodewords in library of codewords 216. Library of codewords 216 may indicate what codewords exist in patterns and the relative positions between the codewords in respective patterns. Device 202 may use library of codewords 216 to identify codewords in one or more reflections within captures from receiver 206. Device 202 may also include, or be coupled to, a power supply 226.

[0066] Processor 210 may be, or may include, one or more suitable processors capable of executing instructions of one or more software programs (such as instructions 214 stored within the memory 212). In some aspects, processor 210 may be one or more general purpose processors that execute instructions 214 to cause device 202 to perform any number of functions or operations. In additional or alternative aspects, processor 210 may include integrated circuits or other hardware to perform functions or operations without the use of software. In some implementations, processor 210 may include one or more application processors to execute applications stored in instructions 214. For example, if device 202 is a smartphone or other computing device, processor 210 may execute instructions for an operating system of device 202, and processor 210 may provide instructions to light controller 218 for controlling the active depth sensing system.

[0067] While shown to be coupled to each other via processor 210 in the example of FIG. 2, processor 210, memory 212, light controller 218, optional display 222, and optional I / O components 224 may be coupled to one another in various arrangements. For example, processor 210, memory 212, light controller 218, optional display 222, and / or optional I / O components 224 may be coupled to each other via one or more local buses (not shown for simplicity).

[0068] Display 222 may be any suitable display or screen allowing for user interaction and / or to present items (such as a depth map, a preview image of a scene, a lock screen, etc.) for viewing by a user. In some aspects, display 222 may be a touch-sensitive display. I / O components 224 may be, or may include, any suitable mechanism, interface, or device to receive input (such as commands) from the user and to provide output to the user. For example, VO components 224 may include (but are not limited to) a graphical user interface, keyboard, mouse, microphone and speakers, squeezable bezel, or border of device 202, physical buttons located on device 202, and so on. In some cases, display 222 and / or VO components 224 may provide a preview image or depth map of the scene to a user and / or receive a user input for adjusting one or more settings of device 202.

[0069] Light controller 218 may include an image signal processor 220, which may be one or more processors to configure projector 204 and process frames captured by receiver 206. In some aspects, image signal processor 220 may execute instructions from a memory (such as instructions 214 from memory 212 or instructions stored in a separate memory coupled to the image signal processor 220). In other aspects, image signal processor 220 may include specific hardware for operation. Image signal processor 220 may, alternatively or additionally, include a combination of specific hardware and the ability to execute software instructions.

[0070] In some implementations, processor 210 may be configured to provide instructions to image signal processor 220. The instructions may be executed by image signal processor 220 to configure filters or other components of an image processing pipeline for processing frames from receiver 206. The instructions may also be executed by image signal processor 220 to configure projector 204 for projecting one or more distributions of light for active depth sensing. While the following aspects of the disclosure may be described in relation to device 202, any suitable device or configuration of device components may be used for performing aspects of the disclosure, and the present disclosure is not limited by a specific device configuration.

[0071] FIG. 3 is a diagram illustrating an example of a simplified structured-light system 300 (which may be referred to alternatively as an active depth-sensing system), according to various aspects of the present disclosure. In some cases, structured-light system 300 can include a transmitter that is part of a larger structured-light system or device. In some cases, structured-light system 300 can be implemented as part of the active depth-sensing system 100 or the system 200. In some implementations, structured-light system 300 can be used for depth sensing to generate depth maps for various applications, such as in 3D imaging for object recognition (e.g., face recognition etc.), autonomous driving systems (e.g., for navigation or other task), gesture recognition, robotics systems, aviation systems, among others. Structured-light system 300 includes an array 302 of light emitting sources (also referred to as light emitters). Array 302 can also be referred to as a structured-light source. In some cases, array 302 can include multiple arrays. In some examples, array 302 of light emitting sources can include a surface-emitting array of light emitters (e.g., a VCSEL array or other suitable array) that generates beams of light 304 from a surface of a device or component (e.g., a substrate or wafer) of a device. While array 302 illustrated in FIG. 3 is shown having a length of 1mmas an illustrative example, one of ordinary skill will appreciate that array 302 can include any other suitable length (e.g., 0.5mm, 1.5mm, 2mm, or other length).

[0072] In some cases, array 302 (e.g., as a surface-emitting array) can be an addressable array, in which individual elements of the array (or in some cases groups of elements in the array) are independently electrically controlled. For example, a light emitting source of the surface-emitting array can be connected to a photodiode (e.g., through a fiber connection or other connection). A pulse voltage can be applied to a base of a transistor (e.g., a bipolar junction transistor), causing a laser current to pulsate and the light emitting source to have a corresponding output power. An example of an addressable surface-emitting array is a VCSEL array, a resonant cavity light emitting diodes (RC- LED) array, among others.

[0073] Beams of light 304 generated and transmitted by array 302 of light emitting sources can be projected in one or more spatial patterns onto a surface of an object or other surface. The spatial pattern can include any type of pattern, such as a pattern that includes spots or dots, stripes, fringe, squares, and / or another other shape. The spatial patterns may be regular (e.g., with a fixed pattern of shapes) or irregular (e g., with a non-fixed pattern of shapes). The spatial pattern can be provided in one or two dimensions.

[0074] In some examples, array 302 of light emitting sources can provide the desired coded primitive pattern (e.g., with enough points to disambiguate depth). For instance, the configuration of light emitting sources in array 302 can be used to define the primitive pattern. In some examples, the primitive pattern can be defined by a pattern diffractive optical element (DOE), not shown in FIG. 3. The beams of emitted light from array 302 (or from the pattern DOE in examples where a pattern DOE is used to define the coded pattern) can be collimated (or aligned) with a lens 306. The longer the focal length of the lens, the higher the resolution of the projected pattern

[0075] A coded primitive pattern can include a plurality of spatially-coded dots within a certain symbol structure (an nl-by-n2 symbol structure). A primitive pattern can be defined from a subset of all possible codes (e.g., combinations of symbols that are possible within the symbol structure). Primitive pattern 402 of FIG. 4 is an example of a 24x34 primitive pattern. Projected pattern 502 of FIG. 5A is an example of a pattern as projected into a scene. In some cases, array 302 may have a regular grid (e.g., a square grid, a hexagonal grid, or a grid having another shape) of light emittingsources that are used to generate the codes. In some cases, a certain number of potential points on the grid can be activated (or turned on). In one illustrative example, a mask pattern can be generated using a regular grid to generate the codes of a coded pattern by turning on approximately 52% of the potential points of the grid. In some examples, the mask pattern (e.g., the activated light emitting sources) does not change temporally and is temporally fixed. Multiple points from the grid form a code. In one illustrative example, a 4x4 group of points (16 points total) can form a code. As noted above, in some cases, not all points (e.g., not all 16 points from the example above) will be turned on or activated, in which case multiple codes (or primitives) can be formed within the mask with strict properties.

[0076] In some cases, as shown in FIG. 3, a single lens 306 can be used to collimate or align the light beams from array 302 of light emitting sources. In some cases, a separate microlens can be provided for each light emitting source. In such cases, each microlens can be used to collimate a light beam from a respective light emitting source (e.g., a first microlens for collimating a first light beam from a first light emitting source of array 302, a second microlens for collimating a second light beam from a second light emitting source of array 302, and so on). Collimation of the light beams from array 302 can increase the effective brightness of the light, can reduce divergence of the light beam, can eliminate, or reduce blur (or astigmatism), can improve wavefront quality, can make the beam less elliptical, and can focus the light beams (e.g., to focus the light beams onto the tessellation DOE 308 described below).

[0077] In some examples, a primitive pattern can be periodic. For example, the code primitive may be replicated or repeated one or more times in one or more directions (e.g., horizontally and / or vertically) to fill the transmitter field of view (FOV) (e.g., the FOV of structured-light system 300). In some examples, tessellation DOE 308 can be used to tessellate (or replicate) the primitive pattern projected by array 302 of light emitting sources. Tessellation DOE 308 can have a template designed to generate and emit multiple replicas of the coded pattern by generating multiple copies of the primitive pattern in a side-by-side manner (e.g., with the primitive pattern being repeated horizontally and / or vertically).

[0078] FIG. 4 is a diagram illustrating an example of a 24x36 coded primitive pattern 402 of light generated by an array (e.g., array 302 of FIG. 3) of light-emitting sources and collimated using a lens(e.g., lens 306). Primitive pattern 402 (the primitive) may be output by a transmitter. The black dots represent points of light.

[0079] FIG. 5A is a diagram illustrating a projected pattern 502 of light (e.g., projected using the transmitter shown in FIG. 3). The projected pattern 502 has the same pattern as the primitive pattern 402. The points of the projected pattern 502 are more spread out as compared to the points shown in the primitive pattern 402 due to the focal length of the transmitter. For example, a longer focal length of the lens results in a higher resolution of the projected pattern (with the points being closer together).

[0080] FIG. 5B is a diagram illustrating a tessellated coded pattern 504 of light. The tessellated coded pattern 504 of light may be generated by a tessellation DOE (e.g., tessellation DOE 308) replicating the proj ected pattern 502 of light (e.g., in order to fill in the transmitter field ofview (FOV), such as the FOV of the structured-light system 300). For example, the center of the tessellated coded pattern 504 of light is the same as the projected pattern 502 of light, with the projected pattern 502 being replicated to the left, right, above, and below the center.

[0081] As described previously, systems and techniques are described herein for performing active depth sensing based on spots in images captured using an active depth-sensing system (e.g., a structured-light system). FIG. 6 is a diagram of an example system 600 that may determine depths (e.g., a depth map 626) of a scene 608 based on spots in images, according to various aspects of the present disclosure. System 600 includes a projector 602. Projector 602 may be an example of projector 102 of FIG. 1 and / or projector 204 of FIG. 2. Projector 602 may project a pattern 604 of dots 606 into scene 608.

[0082] System 600 includes a sensor 610 that may capture image 612 of scene 608. Sensor 610 may be an example of sensor 132 of receiver 108 of FIG. 1 and / or array 302 of FIG. 3. Image 612 may include spots 614. Each of spots 614 may be representative of a respective one of dots 606. Spot 616 is an example one of spots 614. Spot 616 in image 612 is represented by a number of respective pixels (e.g., a number of intensity values) of image 612 (as captured by pixels of sensor 610).

[0083] System 600 (e.g., using a processor 630 of system 600) may determine a pixel of image 612 corresponding to each of spots 614. For example, system 600 may determine pixel 618 corresponds to spot 616 (e.g., based on pixel 618 including a centroid of spot 616).

[0084] Further, system 600 may determine a dot that corresponds (or relates) to the pixel that corresponds to each of spots 614. For example, for each of spots 614, system 600 may determine a dot of dots 606 that corresponds to the pixel corresponding to the respective spot. For instance, system 600 may determine that dot 620 corresponds to spot 616. System 600 may determine the dots using an association 622 between dots 606 and pixels of image 612. For example, system 600 may store (e.g., at a memory 632) association 622 which may include relationships between pixels of image 612 and dots 606. Association 622 between dots 606 and pixels of image 612 may have been determined through a calibration process that is described in further detail below.

[0085] System 600 may determine a depth of a point in the scene from which each of spots 614 was reflected. For example, for each of spots 614, system 600 may determine a depth from which the spot was reflected based on the position of the pixel of the spot in image 612 (and more specifically based on the position of a centroid 619 of spot 616) and the dot of dots 606 related to the pixel. For instance, system 600 may determine a depth 628 for pixel 618 based on the position of pixel 618 (and / or the position of centroid 619 of spot 616) in image 612 and the determined correlation between pixel 618 and dot 620. System 600 may determine the depths based on a relationship 624 between pixel positions (and / or centroid positions) and depths. For example, for each of dots 606, system 600 may store (e g., at memory 632) a relationship 624 between pixel positions (and / or centroid positions) and depths. Relationship 624 between depths and pixels (and / or centroids) may have been determined through a calibration process that is described in further detail below. In some aspects, system 600 determine a depth map 626 which may include the depths of the pixels corresponding to each spots 614.

[0086] The following figures and accompanying description provide additional detail regarding the systems and techniques as outlined with regard to system 600 of FIG. 6. For example, FIG. 7 provides an example of an image 700 of spots 702. Further detail regarding determining a pixel corresponding to each of spots 702 is described with regard to FIG. 7. Additional examples of images and spots are provided in FIG. 8, FIG. 9, FIG. 10A, FIG. 10B, FIG. 12A, and FIG. 12B; and FIG. 11 provides an example histogram 1100 to give additional information pixels and spots. A description of an association (e.g., association 622) between pixels and dots is provided with regard to FIG. 8, FIG. 9, FIG. 10 A, and FIG. 10B, FIG. 11, FIG. 12 A, and FIG. 12B.

[0087] FIG. 7 is a includes an example of an image 700 (or a portion of an image) including spots 702, according to various aspects of the present disclosure. For example, image 700 may be arepresentation of intensities of light (including, for example, visible light, IR light, or UV light) captured at pixels of a sensor (e.g., sensor 132 of receiver 108 of FIG. 1 or array 302 of structured- light system 300 of FIG. 3). The light may include spots 702 which may correspond to dots of a pattern (e.g., pattern 104 of FIG. 1 or primitive pattern 402 of FIG. 4) projected by a projector (e.g., projector 102 of FIG. 1, projector 204 of FIG. 2). For example, the projector may project dots that may reflect from the scene and be captured as spots 702 in image 700.

[0088] As described above with respect to FIG. 6, the system 600 may determine one pixel that corresponds to each spot of an image. To determine the pixel, the system 600 may determine centroids for spots (for example, a centroid 704 for each spot). For example, the system 600 may determine a window (which may be a number of pixels arranged in a square) that represents each of spots 702 illustrated in the image 700 of FIG. 7. For example, the system 600 may determine a 5x5 or 7x7 window of pixels including intensities of each of spots 702. In some cases, the size of the windows may be predetermined. In other cases, the system 600 may determine the size of the windows based on image 700 and / or spots 702 of image 700 (e.g., based on a number of pixels of a group of pixels including intensities above a threshold). As an example, the system 600 may determine a window 706 including 25 pixels (e.g., a 5x5 grid of pixels) including intensities of spot 710.

[0089] System 600 may determine a centroid of each of the windows. For example, for each spot, the system 600 may determine a first order moment of the pixels of the window 706 of the spot 710 in an X and Y direction. For example, the system 600 may multiply a position (e.g., an X and Y coordinate within the window) of each pixel in the window 706 with a normalized intensity value of the pixel and sum the results. The system 600 may divide the first order moments by a number of pixels in the window 706 to determine the position of a centroid 708 within the window 706. In some aspects, the position of the of the centroid 708 of the window 706 may have a sub-pixel resolution. For example, centroid 708 may have a position of (2.7, 2.5) with regard to window 706, where a bottom left corner of window 706 has a position of (0, 0).

[0090] The system 600 may determine that a pixel including the centroid 708 is the pixel that corresponds to the spot. For example, the system 600 may determine that pixel 712 is the pixel that corresponds to spot 710 based on the centroid 708 being within or otherwise associated with the pixel 712. The pixel including the centroid (e.g., centroid 708) may, or may not, be a center pixel of the window (e.g., window 706) upon which the centroid was determined.

[0091] Having determined a pixel of the image that corresponds to each spot in the image, the systems and techniques may determine a dot of the pattern of dots projected into the scene that corresponds to each of the pixels. For example, the system 600 may determine a dot of pattern 104 of FIG. 1 that corresponds to pixel 712 of FIG. 7. To determine which dots of the pattern correspond to which pixels, the system 600 may use an association. The logical underpinnings for such an association, and an example of a calibration process that may be used to generate such an association, are described with regard to FIG. 8, FIG. 9, FIG. 10A, FIG. 10B, FIG. 11, FIG. 12A, and FIG. 12B.

[0092] FIG. 8 is an example of a composite image 800 of several images of spots, according to various aspects of the present disclosure. For example, composite image 800 may include intensities from a five separate images combined. For example, a projector (e.g., projector 102 of FIG. 1, projector 204 of FIG. 2) may project a pattern of dots into a scene (e.g., toward a flat surface). A first image of the scene may be captured by a sensor (e.g., sensor 132 of receiver 108 of FIG. 1 or array 302 of structured-light system 300 of FIG. 3). A relative distance between the scene and the projector and sensor (e.g., a depth) may be changed, for example, the projector and sensor may be move back or the flat surface may be moved back (e.g., increasing the depth). A second image of the scene may be captured. The relative distance may be increased three more times and three more images may be captured.

[0093] Composite image 800 includes intensities from all five of the images. Composite image 800 may include portions of image (e.g., leftmost portions of images). Further, composite image 800 may include only a narrow field of view, for example, composite image 800 may represent a zoomed-in view of images.

[0094] In the first image, captured based on first depth (e.g., with the flat surface a first distance from the sensor and the projector), a spot may be captured as spot 802a. In a second image, the same spot, captured based on a second depth, may be captured as spot 802b. Further, the same spot may be captured as spot 802c in a third image, as spot 802d in a fourth image and spot 802e in a fifth image. Composite image 800, being a composite of the five images, includes spot 802a, spot 802b, spot 802c, spot 802d, and spot 802e.

[0095] FIG. 9 is a includes an example representation (composite image 900) of several images of spots, according to various aspects of the present disclosure. For example, composite image 900 mayinclude intensities from a number of images combined. Composite image 900 may be generated in much the same way as composite image 800. However, composite image 900 may represent a wider field of view than composite image 800. Thus, spots (or streaks based on spots) may appear smaller in composite image 900 than spots appear in composite image 800 (e.g., based on composite image 900 having a higher resolution than composite image 800). Further, composite image 900 may be based on more than five images captured at five respective depths. For example, composite image 900 may be based on 21 images captured at 21 respective depths.

[0096] Streaks of intensity in composite image 900 may represent spots captured at a number of distances. Some pixels of composite image 900 may capture zero spots despite changing the relative distance between and the projector and sensor through a range of relevant depths. Such pixels may be black in composite image 900 as a result of not capturing intensity of any spot in any of the images. Other pixels may capture one spot at one depth. Still other pixels may capture a first spot at a first depth and a second spot at a second depth.

[0097] FIG. 10A includes an example representation (composite image 1000) of 21 images of a single spot, according to various aspects of the present disclosure. For example, composite image 1000 may include intensities from 21 images combined. Composite image 1000 may be generated in much the same way as composite image 800 and composite image 900. However, composite image 1000 may represent a narrower field of view than composite image 900. Composite image 1000 may be a zoomed-in view of a single streak of intensity of composite image 900. The single streak of intensity of composite image 1000 may include 21 spots based on captures at 21 respective depths. For example, the streak of intensity may include a spot 1002a captured at a first depth, a spot 1002b captured at a second depth (less than the first depth), and a spot 1002c captured at a third depth (less than the second depth).

[0098] FIG. 10A additionally illustrates centroids of the spots. For example, a centroid may be determined for each of the spots as described above with regard to FIG. 7. For example, centroid 1004a may be the centroid of spot 1002a, centroid 1004b may be the centroid of spot 1002b, and centroid 1004c may be the centroid of spot 1002c.

[0099] FIG. 10B includes another example representation of the 21 images of the spots of FIG. 10A, according to various aspects of the present disclosure. FIG. 10B further includes line 1010. Line 1010may be a line determined based on the centroids of the spots. For example, line 1010 may be fit to centroids 1004a, 1004b, 1004c, etc.

[0100] FIG. 11 includes an example histogram 1100 that may represent a number of pixels that capture various numbers of spots through a relevant range of depths, according to various aspects of the present disclosure. For example, histogram 1100 may represent a number of pixels of composite image 900 that include intensities from various numbers of spots. According to example histogram 1100, over 140,000 pixels capture zero spots, over 100,000 pixels capture only one spot, over 40,000 pixels capture only two spots, and fewer than 20,000 pixels capture three or more spots.

[0101] FIG. 12A includes an example representation (composite image 1200) of 21 images of spots corresponding to three dots, according to various aspects of the present disclosure. For example, composite image 1200 may include intensities from 21 images combined. Composite image 1200 may be generated in much the same way as composite image 800 and composite image 900. Further, composite image 1200 may have a field of view similar to composite image 900. Composite image 1000 may be a zoomed-in view of a three streaks of intensity of composite image 900.

[0102] The three streaks of intensity of composite image 1200 may include 21 spots (not all of which are included in composite image 1200) based on captures at 21 respective depths. For example, a first streak of intensity may include a spot 1202a captured at a first depth, a spot 1202b captured at a second depth (less than the first depth), a spot 1202c captured at a third depth (less than the second depth), and a spot 1202d captured at a third depth (less than the second depth). Further, a second streak of intensity may include a spot 1206c captured at a third depth, a spot 1206d captured at a fourth depth (less than the third depth), a spot 1206e captured at a fifth depth (less than the fourth depth), a spot 1206f captured at a sixth depth (less than the fifth depth) and a spot 1206g captured at a seventh depth (less than the sixth depth). The second streak may include two additional spots that may be outside the frame of composite image 1200 (e.g., to the right of composite image 1200). Each of the streaks of intensity may correspond to a respective dot, as projected and captured at various depths. For example, spots 1202a, 1202b, 1202c, 1202d, etc. may correspond to a single dot as projected and captured at the various depths and spots 1206c, 1206d, 1206e, 1206f, 1206g, etc. may correspond to another single dot as projected and captured at the various depths.

[0103] FIG. 12B includes another example representation of the 21 images of the spots of FIG. 12A, according to various aspects of the present disclosure. FIG. 12B further includes line 1212. Line 1212 may be a line determined based on the centroids of the spots corresponding to a single dot. For example, line 1212 may be fit to the centroids of spots 1202a, 1202b, 1202c, 1202d, etc.

[0104] According to a first description, the systems and techniques (e.g., system 600) may obtain an image of a scene into which a pattern of dots has been projected. The dots may appear as spots in the image. System 600 may determine a respective depth in the scene from which each of the dots was reflected. System 600 may use an association between pixels and dots and a relationship between pixels (and / or centroids) and depths to determine the depths from which the dots were reflected. The association and the relationship may have been determined during a calibration process prior to using the association and the relationship to determine the depths.

[0105] For example, during a calibration process, a number of images may be captured of a scene as a relative depth between the scene and the camera which captured the images is changed. The scene may be illuminated by a pattern of dots. A number of spots may be identified in each of the number of images. A centroid may be determined for each spot (e.g., as described with regard to FIG. 7). Each spot can be described by its centroid (x, y).

[0106] Further, during the calibration process, each detected spot may be associated with a corresponding dot (e.g., using a template-matching technique). For example, a neighborhood of dots surrounding each dot may be compared to the spots to correlate spots with dots. Further, combining the result of all images, spots associated with each dot can be shown in sensor’s x-y coordinates, as in FIG. lOA and 12 A.

[0107] Due to noise, sampling, and other reasons, the spots, or centroids (x, y), corresponding to a particular dot may have a discrete distribution and may not be along a straight line or a smooth curve. To predict a complete trace of spots of each dot, a fitting line is drawn through the discrete x-y points. For example, line 1212 is drawn based on centroids of spots 1202a, 1202b, 1202c, 1202d, etc. A linear regression with a polynomial fitting may be used to draw such a trace. Such a polynomial fitting can be described as y = a + b*x + c*xA2 + d*xA3 + ... In some aspects, a second or third order polynomial may suffice. Using the x-y polynomial, the spot of each particular dot may travel the entire range corresponding to the minimal and maximal depth. Each pixel through which line 1212 passes may beassociated with the dot, for example, regardless of whether a centroid is in the pixel in the images captured during calibration. For example, pixel 1214 and pixel 1216 may be associated with the dot that resulted in spots 1202a, 1202b, 1202c, 1202d, etc., even though none of the centroids 1204 are within either of pixel 1214 and pixel 1216. All pixels related to a particular dot can be stored as the association between pixels and dots (e.g., association 622 of FIG. 6). After tracing all dots, the association between pixels and dots may be stored (e.g., as a lookup table (LUT)).

[0108] Similar to the x-y distribution, the calibration stage may also provide a discrete distribution of the centroid (x) and depth Z relation for each dot. Using a linear regression with a polynomial fitting, Z = a + b*x + c*xA2 + d*xA3 + . .. the depths for all possible positions along the spot trace of each dot can be predicted. The polynomial coefficients may be stored for each dot. The polynomial may be the relationship between pixels and depths. The polynomial may relate depths to centroids whether the pixels include centroids or not in the images captured during calibration. For example, the polynomial may provide a depth based on an x coordinate within pixel 1214 and / or pixel 1216. For example, if pixel 1214 has an x coordinate of 28, if an x-coordinate of 28.5 (e.g., of a centroid of a spot) is provided to the polynomial, the polynomial may provide a depth.

[0109] In situations in which one pixel relates to multiple dots, (e.g., as described with regard to pixel 1210, spot 1202c and spot 1206g of FIG. 12A), the neighborhood dot pattern may also recorded during the calibration process. Additionally or alternatively, the neighborhood spot pattern (e.g., the dot pattern as reflected as spots) may be recorded. The spot pattern may vary with the change of depths. One approach may include taking an average of the neighbor spot patterns for all calibration depths and saving it as the neighborhood dot pattern for each particular dot.

[0110] The association between pixels and dots and the relationship between pixels (and / or centroid positions) and depths may be used after the calibration process, for example, by system 600. For example, system 600 may include association 622 between pixels and dots and relationship 624 between pixels (and / or centroid positions) and depths. For example, system 600 may identify centroids (x, y) for each spot (e.g., of spots 614) of image 612.

[0111] System 600 may use association 622 between pixels and dots (e.g., the pixel-dot LUT) to identify spots associated with one dot or multiple dots. If a spot is associated with one dot, the x value of the centroid may be applied to the corresponding x- Z polynomial (e.g., relationship 624 betweenpixels and depths) to derive the output depth Z. Alternatively, if a spot is associated with multiple dots, for example, N dots, template-matching is applied N times to the stored N neighborhood dot pattern of the N dots with the neighborhood spot pattern of the identified spot. The best matching provides the winning dot and its stored x-Z polynomial is used to determine the depth measurement.

[0112] According to a second description, system 600 may obtain an image of a scene into which a pattern of dots has been projected. The dots may appear as spots in the image. System 600 may determine a respective depth in the scene from which each of the dots was reflected. To determine the depths, the systems and techniques may determine a pixel of the image that corresponds to each spot in the image (e.g., as described with regard to FIG. 7). Having determined pixels corresponding to each of the spots, the systems and techniques may determine dots (of a projected pattern) that relate to the pixels. To determine the dots, the systems and techniques may use an association between dots and pixels. The association may have been developed during through a calibration process.

[0113] The calibration process may include projecting a pattern into a scene (e.g., at a flat surface) and capturing a number of images at a number of respective relative depths (e.g., as described with regard to FIG. 8). The calibration process may be performed using a pattern, a projector, and / or a sensor that is substantially similar to the pattern, the projector, and the sensor of the deployed systems and techniques. The calibration process may further include analyzing pixels of the images to determine which dots, if any, are captured as spots by each of the pixels. For example, for each of the pixels of the images (or of the sensor) it may be determined which, if any, dots resulted in spots that have intensity values captured by the respective pixel.

[0114] For example, returning to FIG. 10A, in some aspects, each of group of pixels 1008 may be associated with spot 1002a based on each of group of pixels 1008 including intensity values (e.g., above a threshold) of spot 1002a. Further, as part of the calibration process, spot 1002a may be related to a dot of pattern (e.g., using a pattern-matching technique). Further still, each of group of pixels 1008 may be related to the dot.

[0115] In some aspects, the calibration process may include identifying one pixel to relate to each spot before or as part of relating dots to pixels such that each dot corresponds to only one pixel of each image. For example, in some cases, rather than relating multiple pixels that include intensities of a spot (e.g., group of pixels 1008), the calibration process may include identifying one pixel to relateto a spot for each image. For example, the calibration process may include identifying the pixel to relate to a spot based on a centroid of the spot. The centroid of the spot may be determined as described above with regard to FIG. 7. For example, as part of the calibration process, a pixel 1006 may be identified as being related to spot 1002a based on centroid 1004a being in pixel 1006. Further, as part of the calibration process, spot 1002a may be related to a dot of pattern (e.g., using a pattern-matching technique). Further still, pixel 1006 may be related to the dot.

[0116] Some of the pixels may not be related to any spots (e.g., as described with regard to the 0 values in histogram 1100) and consequently may not be related to any dots. Some pixels may relate to a single spot (e.g., as described with regard to the 1 values in histogram 1100 and as illustrated by pixels of composite image 1000). Such pixels may be related to a single dot. Some pixels may relate to two or more spots (e.g., as described with regard to the 2, 3, and 4 values in histogram 1100 and as illustrated by some of the pixels of composite image 1200). Such pixels may be related to two or more dots.

[0117] For example, returning to FIG. 12A, some of the pixels of composite image 1200 (or the sensor that captured composite image 1200) may be related to the dot that resulted in the first streak and to the dot that resulted in the second streak. For example, according to the calibration process, pixel 1210 may be associated with the dot that resulted in the first streak and to the dot that resulted in the second streak. For example, based on the third image of the 21 images (captured at the third distance), pixel 1210 may be associated with the dot that resulted in the first streak based on spot 1202c having intensity values (or a centroid 1204c) in pixel 1210. Further based on the sixth image of the 21 images (captured at the sixth distance), pixel 1210 may be associated with the dot that resulted in the second streak based on spot 1206g having intensity values (or a centroid 1208g) in pixel 1210.

[0118] Additionally or alternatively, in some aspects, the calibration process may include determining a trace for each dot of a pattern of dots. For instance, for a particular dot, the position of the corresponding spot in an image of the scene depends on the depth of the point (in three- dimensional space) from which the dot is reflected. Considering the depth of points in the scene can be any value between zero and infinity, the corresponding spot positions (or spot trajectory) on the sensor (or in the image), may be a continuous function of the depth. For example, there may be onecentroid position of a spot for each possible depth. Line 1010 of FIG. 10B and line 1212 of FIG. 12B are examples of traces.

[0119] The calibration process may include determining a spot centroid trace of each dot of a pattern. For example, once optics of a system is fixed, a trace of each dot of a pattern of dots of the system may be determined. For instance, the pattern of dots may be projected into a scene, images of the dots in the scene may be captured at various depths, and centroids of spots corresponding to the dots may be recorded. The recorded centroids may be fit by a line (e.g., a spot centroid trace). For example, line 1212 of FIG. 12B may be fit to centroids of spots 1202a, 1202b, 1202c, 1202c, etc. For each spot in all of the images, the centroid of the spot may be calculated in x-y coordinates (which may have a sub-pixel resolution). A polynomial fitting may be used to find the x-y trajectory (e.g., line 1212) which may be the spot centroid trace. A trace of a given dot may be used to determine which pixels (of an image or sensor) may include centroids of given dot. Thus system 600 may use a trace to determine which dots relate to which image pixels (e.g., image pixels including centroids of spots). In any case, the calibration process may generate the association between dots and pixels. For example, by storing indications of to which dot (or dots) each pixel relates additionally or alternatively, by determining and storing a trace for each dot.

[0120] When deployed, systems and techniques may use the association between dots and pixels (previously developed through the calibration process) to determine a dot for each pixel (i.e., the pixels determined to correspond to spots in the obtained image). For example, the association may be stored in a look-up table (LUT) and the systems and techniques may provide a position of the pixel in the obtained image to the LUT and receive an identifier of a dot in return.

[0121] In some cases, for example, cases in which the pixel corresponds to a single dot (e.g., as described with relation to FIG. 10A), the systems and techniques may determine the dot based on the association. In other cases, for example, cases in which two or more dots relate to a single pixel (e.g., as described with regard to the 2, 3, and 4 values in histogram 1100 and illustrated by some of the pixels of composite image 1200), the systems and techniques may take one or more additional steps to determine the dot that corresponds to the pixel.

[0122] For example, in such cases, the systems and techniques may disambiguate between the two or more dots. In some aspects, the systems and techniques may disambiguate between the two or moredots based on a neighborhood of dots and a corresponding neighborhood of spots (e.g., using a patternmatching technique). For example, the systems and techniques may store an indication of a neighborhood of each dot of a pattern. The indication may include a bitmap. In cases in which a pixel corresponds to multiple dots, the systems and techniques may compare a bitmap (or an expanded or compressed representation thereof) of each of the dots to a neighborhood of spots as captured in an image to determine to which dot the pixel corresponds.

[0123] One advantage of such a pattern matching over other structured-light techniques is that the pattern matching according to various aspects of the present disclosure may compare neighborhoods of the dots that are related to the pixel (through the association) to the neighborhoods of spots in the image. Accordingly, the systems and techniques may reduce the number of pattern-matching comparisons compared to other structured-light techniques that may involve pattern matching.

[0124] Having determined a pixel that corresponds to a spot, and a dot that corresponds to the pixel, the systems and techniques (e.g., system 600) may determine a depth for the pixel based on the dot and the pixel (and / or the centroid of the spot). Returning to FIG. 1 as an example, distance 134 between center 114 of sensor 132 and position 116 of sensor 132 may be an example of a pixel position (or a centroid position). Similarly, distance 136 between center 114 and position 118 of sensor 132 may be another example of a pixel position (or a centroid position). Distance 134 may be used to determine a depth of object 106B in scene 106 for example, using three-dimensional geometry and baseline 112. Similarly, distance 136 and baseline 112 may be used to determine a depth of object 106A in scene 106.

[0125] System 600 may determine depths for points in a scene from which dots are reflected; for example, system 600 may determine a depth map. System 600 may determine the depths for various points in the scene based on a relationship between pixels (and / or centroids) and depths. For example, rather than calculating the distance based on three-dimensional geometry, system 600 may store a relationship between pixel positions (and / or centroid positions) and depths. Such relationships may be specific to each dot. For example, for each dot, system 600 may store a relationship between pixel positions (and / or centroid positions) in an image and depths. For instance, for a given dot, system 600 may store a relationship between a number of possible pixel positions of the spot (in an image of the pattern) (and / or possible positions of the centroid) and depths that would cause the dot to appear as a spot at that pixel position (or centroid position).

[0126] The relationship may be determined through a calibration process. For example, while capturing images of spots at various depths (as described above) the various depths may be recorded along with the pixel positions (and / or centroid positions) of the spots. The record of depths and pixel positions (and / or centroid positions) may be used as the relationship. In some cases, the calibration process may include extrapolating between pixel positions (and / or centroid positions) and depth to generate the relationship. Additionally or alternatively, the calibration process may include determining a polynomial that describes the relationship between pixels positions and depths.

[0127] For example, as described above, a spot centroid trace may be determined for each dot. Each point of spot centroid trace may correspond to a depth of a point from which the dot was reflected. For example, the spot centroid trace may be a continuous function of the depth. For example, there may be one depth for each centroid position of a spot. Line 1010 of FIG. 10B and line 1212 of FIG. 12B are examples of spot centroid traces. There may be a depth corresponding to each point of line 1010 and there may be a different depth corresponding to each point of line 1212.

[0128] A relationship between pixel positions (or centroid positions) of a spot and three-dimensional depths may be determined by a polynomial fitting. For example, during the calibration process, a depth may be recorded for each of the spots on which a spot centroid trace is based. The depths may be mapped, for example, through a polynomial fitting, to the spot centroid trace. The polynomial fitting may produce a continuous function (e.g., relating pixel positions with depths). The continuous function may be stored in the memory and used for depth determinations (e.g., as relationship 624 of FIG. 6). Therefore, the determination of depth is limited neither to a finite number of calibration steps nor to the finite number of pixels of the x-y trajectory.

[0129] FIG. 13 is a diagram of a system 1300 for determining an association between pixels and dots and / or a relationship between centroid positions and depths, according to various aspects of the present disclosure. During a calibration process an association between pixels and dots and / or a relationship between centroid positions and depths may be determined for each dot. For example, a pattern 1304 of dots (e.g., including one dot 1318 for descriptive purposes) may be projected into a scene 1308 (e.g., onto a flat surface) by a projector 1302. Dot 1322 is a representation of dot 1318 in scene 1308. An image of the scene may be captured. The image may include spots representative of the dots (e.g., spot 1316 representative of dot 1318). A depth 1328 of the scene (e.g., a distance between the flat surface and the projector 1302 and sensor 1310) may be changed a number of timesand a respective number of images may be captured. Image 1312 is a representation of several of such images composited. In image 1312, spot 1316 appears as a streak because image 1312 is a composite of several images.

[0130] Each dot of pattern 1304 may be associated with pixels of image 1312. For example, spot 1316 may be associated with dot 1318 (e.g., by matching a pattern of dots in the neighborhood of dot 1318 with a pattern of spots in the neighborhood of spot 1316). Pixels representative of spot 1316 may then be associated with dot 1318. In some aspects, pixels including centroids of spot 1316 may be associated with dot 1318. In some aspects, a polynomial (e.g., y = a + b*x + c*xA2 + d*xA3 + .. .) may be fit to spot 1316. The polynomial may predict pixel positions for spot 1316 that are between and beyond what was measured during the calibration process. For example, the polynomial may describe a line between a central vertical axis of image 1312 and an outer edge of image 1312. The line may pass through pixels that included spots as measured during calibration and through pixels that that did not include spots (e.g., based on the depths at which the images of the spots were captured).

[0131] Further, during the calibration process, for each dot, for each image, the pixel position of a centroid of a spot corresponding to the dot may be stored along with the depth at which the image was captured. For example, for each image, a pixel location of spot 1316 may be stored along with the depth 1328 at which the image was captured. Then, for each dot, a polynomial (e.g., Z = a + b*x + c*xA2 + d*xA3 + ...) may be fit to the centroid positions such that the polynomial relates centroid positions to depths. For example, the polynomial may describe depth 1328 as a function of centroid positions of spot 1316.

[0132] When deployed, the systems and techniques may use association between pixels and dots to determine a correspondence between spots, as they appear in images, and the dots of the pattern that resulted in the spots. Additionally, the systems and techniques may use the relationship between depths and centroids to determine depths of points from which dots are reflected based on the centroid position in the image and the dot that appeared as a spot at the pixel position. For example, the systems and techniques may, as described above, determine a given dot that appears as a spot at a given pixel position based on the association between pixels and dots. The systems and techniques may use the relationship of the given dot between centroid positions and depths to determine a depth of a point in the scene that reflected the given dot.

[0133] The systems and techniques may apply to any pattern. Some patterns may result in more dots being related to the same pixel than other patterns. For example, patterns including many dots that are along a single streak line (e.g., a streak line illustrated by any of the streaks in composite image 800, composite image 900, composite image 1000, or composite image 1200) may result in more pixels that are related to multiple dots than patterns that avoid such aligning of dots. Accordingly, an aspect disclosed herein is a pattern that decreases the number of dots that will relate to the same image pixel. For example, such a pattern may include dots arranged such that few (or no) dots are in line across a streak line. Image 900 of FIG. 9 is used as an example to describe a relationship between a pattern and streaks. For example, the central horizontal row of a pattern which resulted in image 900 resulted in more “spot trace” overlapping than other regions. The overlapping results in brighter streaks based on the compositing of images. Specifically, because the pattern which resulted in image 900 included several dots along a center horizontal axis of the pattern, image 900 includes overlapping streaks in a center of image 900. An alternative dot pattern including fewer dots on the center horizontal axis would decrease the overlapping seen in image 900.

[0134] FIG. 14 is a flow diagram illustrating a process 1400 for active depth sensing based on spots, in accordance with aspects of the present disclosure. One or more operations of process 1400 may be performed by a computing device (or apparatus) or a component (e.g., a chipset, codec, etc.) of the computing device. The computing device may be a mobile device (e.g., a mobile phone), a network- connected wearable such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or augmented reality (AR) device, a vehicle or component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and / or any other computing device with the resource capabilities to perform the process 1400. The one or more operations of process 1400 may be implemented as software components that are executed and run on one or more processors.

[0135] At block 1402, the computing device (or one or more components thereof) may obtain an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern. For example, system 600 of FIG. 6 (or processor 630 of system 600) may obtain image 612 of scene 608 (e.g., using sensor 610). Scene 608 may be illuminated by pattern 604 (e.g., by projector 602). Pattern 604 may include a number of dots 606.Image 612 may include a number of spots 614. Each of spots 614 may correspond to a respective one of dots 606.

[0136] In some aspects, the computing device (or one or more components thereof) may cause a projector to project the pattern comprising the plurality of dots to illuminate the scene with the pattern. For example, processor 630 may cause projector 602 to project pattern 604 onto scene 608 to illuminate scene 608 with pattern 604. In some aspects, the projector may be, or may include, a light source and / or a diffractive optical element. In some aspects, the light source may be, or may include, one or more lasers. For example, projector 602 may be, or may include, a light source (e.g., one or more lasers) and / or a diffractive optical element. In some aspects, the computing device (or one or more components thereof) may cause an image sensor to capture the image, for example, processor 630 may cause sensor 610 to capture image 612.

[0137] At block 1404, the computing device (or one or more components thereof) may determine a pixel of the image that corresponds to a spot of the plurality of spots in the image. For example, system 600 may determine pixel 618 of image 612 that corresponds to spot 616 (which is one of spots 614).

[0138] In some aspects, to determine the pixel of the image that corresponds to the spot, computing device (or one or more components thereof) may determine a position of a centroid of the spot based on values of pixels of the image corresponding to the spot and determine the pixel based on the centroid. For example, system 600 may determine centroid 619 of spot 616 and determine dot 620 based on centroid 619. In some aspects, the pixels of the image corresponding to the spot include a window of pixels representative of the spot, and wherein determining the position of the centroid is based on moments of the window of pixels. For example, window 706 of FIG. 7 may be representative of spot 710. Centroid 708 may be determined based on moments of the pixels of window 706. For example, system 600 may determine centroid 619 based on pixels of a window representative of spot 616. In some aspects, the position of the centroid may have a sub-pixel resolution. For example, the coordinates of the position of centroid 619 may have a resolution finer than the size of the pixels of image 612. In some aspects, to determine the pixel based on the centroid, the computing device (or one or more components thereof) may identify the pixel based on the pixel including the centroid. For example, system 600 may determine pixel 618 based on pixel 618 including centroid 619.

[0139] At block 1406, the computing device (or one or more components thereof) may determine a dot of the plurality of dots of the pattern that corresponds to the pixel. For example, system 600 may determine that dot 620 corresponds to pixel 618.

[0140] In some aspects, the dot of the plurality of dots of the pattern that corresponds to the pixel may be determined based on an association between pixels and dots. For example, system 600 may determine that dot 620 corresponds to pixel 618 (and / or to centroid 619) based on association 622 (which may be an association between dots pixels and dots). In some aspects, the association between pixels and dots may describe which pixels may represent which dots. For example, association 622 may include a description of which pixels of image 612 may represent each of dots 606. In some aspects, the association between pixels and dots may be, or may include, a polynomial. For example, association 622 may be, or may include, a polynomial (e.g., describing a line such as line 1010 of FIG. 10 or line 1212 of FIG. 12). In some aspects, the association between pixels and dots may be stored in a look-up table (LUT). For example, association 622 may be stored in memory 632, for example, in the form of a LUT.

[0141] In some aspects, the association between pixels and dots may be based on a calibration process involving the pattern and an image sensor. For example, association 622 may be generated through a calibration process involving pattern 604. The calibration process may involve a system that is substantially similar to system 600. For example, the pattern used in the calibration process may be substantially similar to pattern 604. Further, the projector used in the calibration process may be substantially similar to projector 602 and the sensor used in the calibration process may be substantially similar to sensor 610. Further still, a distance between the projector used in the calibration process and the sensor used in the calibration process may be substantially the same as the distance between projector 602 and sensor 610.

[0142] In some aspects, the computing device (or one or more components thereof) may determine the dot from among two or more dots based on the two or more dots being associated with the pixel. For example, system 600 may distinguish dot 620 from among two or more dots of dots 606 that may be described by a line that passes through pixel 618. In some aspects, the dot is determined from among the two or more dots based on a pattern of dots proximate to the dot. For example, system 600 may distinguish dot 620 from the two or more dots based on a pattern of dots in a neighborhood around dot 620, for example, by comparing the neighborhood of dots around dot 620 to a neighborhood ofspots around spot 616 in image 612. For example, system 600 may match a pattern of spots around spot 616 with a pattern dots of dots 606. To perform the pattern matching, system 600 may only compare the pattern of dots proximate to spot 616 with dots that are associated with pixel 618 (e.g., based on association 622). In some aspects, the pattern may be designed to limit a number of dots that relate to each pixel. For example, pattern 604 may have been designed to limit a number of dots 606 that relate to each pixel of image 612. For example, pattern 604 may have been designed to limit a number of dots 606 that will trace a line through each pixel of image 612.

[0143] At block 1408, the computing device (or one or more components thereof) may determine a depth for the spot based on the dot and the spot. For example, system 600 may determine depth 628 for spot 616. Depth 628 may be a depth of a point in scene 608 (e.g., the point from which spot 616 was reflected).

[0144] In some aspects, the depth for the spot is determined based on a relationship between pixels and depths. For example, system 600 may determine depth 628 for spot 616 based on relationship 624 (which may be, or may include, a relationship between positions of spots in image 612 and depths). In some aspects, the relationship may be specific to the dot. For example, memory 632 may store a relationship for each dot of dots 606. In some aspects, the relationship may be, or may include, a polynomial. In some aspects, the relationship may relate a position of the pixel in the image to the depth. For example, relationship 624 may be, or may include, a function relating positions of pixels in image 612 to depths. In some aspects, the relationship may be based on a calibration process involving the pattern and an image sensor. For example, relationship 624 may be developed through a calibration process involving a system that is the same as, or substantially similar to, system 600.

[0145] In some aspects, the depth for the spot is determined based on a relationship between pixels and centroids. For example, system 600 may determine depth 628 for spot 616 based on relationship 624 (which may be, or may include, a relationship between positions of centroids of spots in image 612 and depths). In some aspects, the computing device (or one or more components thereof) may determine a centroid of the spot and determine the depth based on the centroid of the spot. For example, system 600 may determine centroid 619 of spot 616 and determine depth 628 based on centroid 619 (e.g., based on relationship 624). In some aspects, the relationship may be specific to the dot. For example, memory 632 may store a relationship for each dot of dots 606. In some aspects, the relationship may be, or may include, a polynomial. In some aspects, the relationship may relate aposition of a centroid in the image to the depth. For example, relationship 624 may be, or may include, a function relating positions of centroids in image 612 to depths. Isa, the position of the centroid in the image has a sub-pixel resolution. For example, the position of centroid 619 in image 612 may have a finer resolution than the pixels of image 612. In some aspects, the relationship may be based on a calibration process involving the pattern and an image sensor. For example, relationship 624 may be developed through a calibration process involving a system that is the same as, or substantially similar to, system 600.

[0146] In some aspects, the computing device (or one or more components thereof) may, for each respective spot of the plurality of spots as captured in the image: determine a respective pixel of the image that corresponds to the respective spot; determine a respective dot of the plurality of dots of the pattern that corresponds to the respective pixel; and determine a respective depth for the respective spot based on the respective dot and the respective spot. For example, system 600 may determine a depth for each of spots 614 in image 612 according to process 1400.

[0147] In some examples, as noted previously, the methods described herein (e.g., process 1400 of FIG. 14, and / or other methods described herein) can be performed, in whole or in part, by a computing device or apparatus. In one example, one or more of the methods can be performed by active depthsensing system 100 of FIG. 1, system 200 of FIG. 2, structured-light system 300 of FIG. 3, system 600 of FIG. 6, or by another system or device. In another example, one or more of the methods (e.g., process 1400 of FIG. 14, and / or other methods described herein) can be performed, in whole or in part, by the computing-device architecture 1500 shown in FIG. 15. For instance, a computing device with the computing-device architecture 1500 shown in FIG. 15 can include, or be included in, the components of the active depth-sensing system 100, system 200, structured-light system 300, or system 600 and can implement the operations of process 1400, and / or other process described herein. In some cases, the computing device or apparatus can include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) that are configured to carry out the steps of processes described herein. In some examples, the computing device can include a display, a network interface configured to communicate and / or receive the data, any combination thereof, and / or other component(s). The network interface can be configured to communicate and / or receive Internet Protocol (IP) based data or other type of data.

[0148] The components of the computing device can be implemented in circuitry. For example, the components can include and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or can include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

[0149] Process 1400, and / or other process described herein are illustrated as logical flow diagrams, the operation of which represents a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the processes.

[0150] Additionally, process 1400, and / or other process described herein can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non- transitory.

[0151] FIG. 15 illustrates an example computing-device architecture 1500 of an example computing device which can implement the various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or computing device of a vehicle), or other device. For example, the computing-device architecture 1500 may include, implement, or be included in any or all of active depth-sensing system 100 of FIG. 1, system 200 of FIG. 2, structured-light system 300 of FIG. 3, and / or system 600 of FIG. 6. Additionally or alternatively, computingdevice architecture 1500 may be configured to perform process 1400, and / or other process described herein.

[0152] The components of computing-device architecture 1500 are shown in electrical communication with each other using connection 1512, such as a bus. The example computing-device architecture 1500 includes a processing unit (CPU or processor) 1502 and computing device connection 1512 that couples various computing device components including computing device memory 1510, such as read only memory (ROM) 1508 and random-access memory (RAM) 1506, to processor 1502.

[0153] Computing-device architecture 1500 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 1502. Computing-device architecture 1500 can copy data from memory 1510 and / or the storage device 1514 to cache 1504 for quick access by processor 1502. In this way, the cache can provide a performance boost that avoids processor 1502 delays while waiting for data. These and other modules can control or be configured to control processor 1502 to perform various actions. Other computing device memory 1510 may be available for use as well. Memory 1510 can include multiple different types of memory with different performance characteristics. Processor 1502 can include any general-purpose processor and a hardware or software service, such as service 1 1516, service 2 1518, and service 3 1520 stored in storage device 1514, configured to control processor 1502 as well as a special-purpose processor where software instructions are incorporated into the processor design. Processor 1502 may be a self- contained system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0154] To enable user interaction with the computing-device architecture 1500, input device 1522 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. Output device 1524 can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display, projector, television, speaker device, etc. In some instances, multimodal computing devices can enable a user to provide multiple types of input to communicate with computing-device architecture 1500. Communication interface 1526 can generally govern and manage the user input and computing device output. There is no restriction on operating on anyparticular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0155] Storage device 1514 is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, randomaccess memories (RAMs) 1506, read only memory (ROM) 1508, and hybrids thereof. Storage device 1514 can include services 1516, 1518, and 1520 for controlling processor 1502. Other hardware or software modules are contemplated. Storage device 1514 can be connected to the computing device connection 1512. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 1502, connection 1512, output device 1524, and so forth, to carry out the function.

[0156] The term “substantially,” in reference to a given parameter, property, or condition, may refer to a degree that one of ordinary skill in the art would understand that the given parameter, property, or condition is met with a small degree of variance, such as, for example, within acceptable manufacturing tolerances. By way of example, depending on the particular parameter, property, or condition that is substantially met, the parameter, property, or condition may be at least 90% met, at least 95% met, or even at least 99% met.

[0157] Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to specific devices.

[0158] The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system and so on). As used herein, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term “device” to describe various aspects of this disclosure, the term “device” is not limited to a specific configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or specific aspects. Forexample, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. While the below description and examples use the term “system” to describe various aspects of this disclosure, the term “system” is not limited to a specific configuration, type, or number of objects.

[0159] Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by one of ordinary skill in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks including devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the aspects.

[0160] Individual aspects may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.

[0161] Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general-purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0162] The term “computer-readable medium” includes, but is not limited to, portable or nonportable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and / or data. A computer-readable medium may include a non- transitory medium in which data can be stored and that does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, magnetic or optical disks, USB devices provided with non-volatile memory, networked storage devices, any suitable combination thereof, among others. A computer-readable medium may have stored thereon code and / or machineexecutable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.

[0163] In some aspects the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non- transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0164] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also beimplemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

[0165] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.

[0166] In the foregoing description, aspects of the application are described with reference to specific aspects thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative aspects of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate aspects, the methods may be performed in a different order than that described.

[0167] One of ordinary skill will appreciate that the less than (“<“) and greater than (“>“) symbols or terminology used herein can be replaced with less than or equal to (“<”) and greater than or equal to (“>”) symbols, respectively, without departing from the scope of this description.

[0168] Where components are described as being “configured to” perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.

[0169] The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and / or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and / or other suitable communication interface) either directly or indirectly.

[0170] Claim language or other language reciting “at least one of’ a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any duplicate information or data (e.g., A and A, B and B, C and C, A and A and B, and so on), or any other ordering, duplication, or combination of A, B, and C. The language “at least one of’ a set and / or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” may mean A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases “at least one” and “one or more” are used interchangeably herein.

[0171] Claim language or other language reciting “at least one processor configured to,” “at least one processor being configured to,” “one or more processors configured to,” “one or more processors being configured to,” or the like indicates that one processor or multiple processors (in any combination) can perform the associated operation(s). For example, claim language reciting “at least one processor configured to: X, Y, and Z” means a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a certain subset of operations X, Y, and Z such that together the multiple processors perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting “at least one processor configured to: X, Y, and Z” can mean that any single processor may only perform at least a subset of operations X, Y, and Z.

[0172] Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performs the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g., an apparatus) to perform functions, one element may be configured to cause the other element to perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions.

[0173] Where reference is made to an entity (e.g., any entity or device described herein) performing functions or being configured to perform functions (e.g., steps of a method), the entity may be configured to cause one or more elements (individually or collectively) to perform the functions. The one or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more (or all) of the functions, and / or any combination thereof. Where reference to the entity performing functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to collectively perform the functions. When the entity is configured to cause more than one component to collectively perform the functions, each function need not be performed by each of those components (e.g., different functions may be performed by different components) and / or each function need not be performed in whole by only one component (e.g., different components may perform different sub-functions of a function).

[0174] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0175] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general-purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer- readable data storage medium including program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage mediummay form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random-access memory (RAM) such as synchronous dynamic random-access memory (SDRAM), read-only memory (ROM), non-volatile random-access memory (NVRAM), electrically erasable programmable readonly memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0176] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration). Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.

[0177] Illustrative aspects of the disclosure include:

[0178] Aspect 1. An apparatus for determining depths, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determine a pixel of the image that corresponds to a spot of the plurality of spots in the image; determine a dot of the plurality of dots of the pattern that corresponds to the pixel; and determine a depth for the spot based on the dot and the spot.

[0179] Aspect 2. The apparatus of aspect 1, wherein, to determine the pixel of the image that corresponds to the spot, the at least one processor is configured to: determine a position of a centroid of the spot based on values of pixels of the image corresponding to the spot; and determine the pixel based on the centroid.

[0180] Aspect 3. The apparatus of aspect 2, wherein the pixels of the image corresponding to the spot include a window of pixels representative of the spot, and wherein determining the position of the centroid is based on moments of the window of pixels.

[0181] Aspect 4. The apparatus of any one of aspects 2 or 3, wherein the position of the centroid has a sub-pixel resolution.

[0182] Aspect 5. The apparatus of any one of aspects 2 to 4, wherein, to determine the pixel based on the centroid, the at least one processor is configured to identify the pixel based on the pixel including the centroid.

[0183] Aspect 6. The apparatus of any one of aspects 1 to 5, wherein the dot of the plurality of dots of the pattern that corresponds to the pixel is determined based on an association between pixels and dots.

[0184] Aspect 7. The apparatus of aspect 6, wherein the association between pixels and dots describes which pixels may represent which dots.

[0185] Aspect 8. The apparatus of any one of aspects 6 or 7, wherein the association between pixels and dots comprises a polynomial.

[0186] Aspect 9. The apparatus of any one of aspects 6 to 8, wherein the association between pixels and dots is stored in a look-up table (LUT).

[0187] Aspect 10. The apparatus of any one of aspects 6 to 9, wherein the association between pixels and dots is based on a calibration process involving the pattern and an image sensor.

[0188] Aspect 11. The apparatus of any one of aspects 6 to 10, wherein the at least one processor is further configured to determine the dot from among two or more dots based on the two or more dots being associated with the pixel.

[0189] Aspect 12. The apparatus of aspect 11, wherein the dot is determined from among the two or more dots based on a pattern of dots proximate to the dot.

[0190] Aspect 13. The apparatus of any one of aspects 1 to 12, wherein the pattern is designed to limit a number of dots that relate to each pixel.

[0191] Aspect 14. The apparatus of any one of aspects to 13, wherein the depth for the spot is determined based on a relationship between pixels and depths.

[0192] Aspect 15. The apparatus of any one of aspects 1 to 14, wherein the depth for the spot is determined based on a relationship between pixels and centroids.

[0193] Aspect 16. The apparatus of aspect 15, wherein the at least one processor is further configured to determine a centroid of the spot and determine the depth based on the centroid of the spot.

[0194] Aspect 17. The apparatus of any one of aspects 15 or 16, wherein the relationship is specific to the dot.

[0195] Aspect 18. The apparatus of any one of aspects 15 to 17, wherein the relationship comprises a polynomial.

[0196] Aspect 19. The apparatus of any one of aspects 15 to 18, wherein the relationship relates a position of a centroid in the image to the depth.

[0197] Aspect 20. The apparatus of aspect 19, wherein the position of the centroid in the image has a sub-pixel resolution.

[0198] Aspect 21. The apparatus of any one of aspects 15 to 20, wherein the relationship is based on a calibration process involving the pattern and an image sensor.

[0199] Aspect 22. The apparatus of aspect 1, wherein the at least one processor is further configured to cause a projector to project the pattern comprising the plurality of dots to illuminate the scene with the pattern.

[0200] Aspect 23. The apparatus of aspect 22, further comprising the projector.

[0201] Aspect 24. The apparatus of any one of aspects 22 or 23, wherein the projector comprises at least one of a light source or a diffractive optical element.

[0202] Aspect 25. The apparatus of aspect 24, wherein the light source comprises one or more lasers.

[0203] Aspect 26. The apparatus of any one of aspects 1 to 25, further comprising causing an image sensor to capture the image.

[0204] Aspect 27. The apparatus of aspect 26, further comprising the image sensor.

[0205] Aspect 28. The apparatus of any one of aspects 1 to 27, wherein the at least one processor is further configured to: for each respective spot of the plurality of spots as captured in the image: determining a respective pixel of the image that corresponds to the respective spot; determining a respective dot of the plurality of dots of the pattern that corresponds to the respective pixel; and determining a respective depth for the respective spot based on the respective dot and the respective spot.

[0206] Aspect 29. A method for determining depths, the method comprising: obtaining an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determining a pixel of the image that corresponds to a spot of the plurality of spots in the image; determining a dot of the plurality of dots of the pattern that corresponds to the pixel; and determining a depth for the spot based on the dot and the spot.

[0207] Aspect 30. The method of aspect 29, wherein determining the pixel of the image that corresponds to the spot comprises: determining a position of a centroid of the spot based on values of pixels of the image corresponding to the spot; and determining the pixel based on the centroid.

[0208] Aspect 31. The method of aspect 30, wherein the pixels of the image corresponding to the spot include a window of pixels representative of the spot, and wherein determining the position of the centroid is based on moments of the window of pixels.

[0209] Aspect 32. The method of any one of aspects 30 or 31, wherein the position of the centroid has a sub-pixel resolution.

[0210] Aspect 33. The method of any one of aspects 30 to 32, wherein determining the pixel based on the centroid comprises identifying the pixel based on the pixel including the centroid.

[0211] Aspect 34. The method of any one of aspects 29 to 33, wherein the dot of the plurality of dots of the pattern that corresponds to the pixel is determined based on an association between pixels and dots.

[0212] Aspect 35. The method of aspect 34, wherein the association between pixels and dots describes which pixels may represent which dots.

[0213] Aspect 36. The method of any one of aspects 34 or 35, wherein the association between pixels and dots comprises a polynomial.

[0214] Aspect 37. The method of any one of aspects 34 to 36, wherein the association between pixels and dots is stored in a look-up table (LUT).

[0215] Aspect 38. The method of any one of aspects 34 to 37, wherein the association between pixels and dots is based on a calibration process involving the pattern and an image sensor.

[0216] Aspect 39. The method of any one of aspects 34 to 38, further comprising determining the dot from among two or more dots based on the two or more dots being associated with the pixel.

[0217] Aspect 40. The method of aspect 39, wherein the dot is determined from among the two or more dots based on a pattern of dots proximate to the dot.

[0218] Aspect 41. The method of any one of aspects 29 to 40, wherein the pattern is designed to limit a number of dots that relate to each pixel.

[0219] Aspect 42. The method of any one of aspects 29 to 41, wherein the depth for the spot is determined based on a relationship between pixels and depths.

[0220] Aspect 43. The method of any one of aspects 29 to 42, wherein the depth for the spot is determined based on a relationship between pixels and centroids.

[0221] Aspect 44. The method of aspect 43, further comprising determining a centroid of the spot and determining the depth based on the centroid of the spot.

[0222] Aspect 45. The method of any one of aspects 43 or 44, wherein the relationship is specific to the dot.

[0223] Aspect 46. The method of any one of aspects 43 to 45, wherein the relationship comprises a polynomial.

[0224] Aspect 47. The method of any one of aspects 43 to 46, wherein the relationship relates a position of a centroid in the image to the depth.

[0225] Aspect 48. The method of aspect 47, wherein the position of the centroid in the image has a sub-pixel resolution.

[0226] Aspect 49. The method of any one of aspects 43 to 48, wherein the relationship is based on a calibration process involving the pattern and an image sensor.

[0227] Aspect 50. The method of any one of aspects 29 to 49, further comprising projecting, using a projector, the pattern comprising the plurality of dots to illuminate the scene with the pattern.

[0228] Aspect 51. The method of aspect 50, wherein the projector comprises at least one of a light source or a diffractive optical element.

[0229] Aspect 52. The method of aspect 51, wherein the light source comprises one or more lasers.

[0230] Aspect 53. The method of any one of aspects 29 to 52, further comprising capturing the image.

[0231] Aspect 54. The method of any one of aspects 29 to 53, further comprising: for each respective spot of the plurality of spots as captured in the image: determining a respective pixel of the image that corresponds to the respective spot; determining a respective dot of the plurality of dots of the pattern that corresponds to the respective pixel; and determining a respective depth for the respective spot based on the respective dot and the respective spot.

[0232] Aspect 55. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations according to any of aspects 29 to 54.

[0233] Aspect 56. An apparatus for providing virtual content for display, the apparatus comprising one or more means for perform operations according to any of aspects 29 to 54.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. An apparatus for determining depths, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determine a pixel of the image that corresponds to a spot of the plurality of spots in the image; determine a dot of the plurality of dots of the pattern that corresponds to the pixel; and determine a depth for the spot based on the dot and the spot.

2. The apparatus of claim 1 , wherein, to determine the pixel of the image that corresponds to the spot, the at least one processor is configured to: determine a position of a centroid of the spot based on values of pixels of the image corresponding to the spot; and determine the pixel based on the centroid.

3. The apparatus of claim 2, wherein the pixels of the image corresponding to the spot include a window of pixels representative of the spot, and wherein determining the position of the centroid is based on moments of the window of pixels.

4. The apparatus of claim 2, wherein the position of the centroid has a sub-pixel resolution.

5. The apparatus of claim 2, wherein, to determine the pixel based on the centroid, the at least one processor is configured to identify the pixel based on the pixel including the centroid.

6. The apparatus of claim 1, wherein the dot of the plurality of dots of the pattern that corresponds to the pixel is determined based on an association between pixels and dots.

7. The apparatus of claim 6, wherein the association between pixels and dots describes which pixels may represent which dots.

8. The apparatus of claim 6, wherein the association between pixels and dots comprises a polynomial.

9. The apparatus of claim 6, wherein the association between pixels and dots is stored in a look-up table (LUT).

10. The apparatus of claim 6, wherein the association between pixels and dots is based on a calibration process involving the pattern and an image sensor.

11. The apparatus of claim 6, wherein the at least one processor is further configured to determine the dot from among two or more dots based on the two or more dots being associated with the pixel.

12. The apparatus of claim 11, wherein the dot is determined from among the two or more dots based on a pattern of dots proximate to the dot.

13. The apparatus of claim 1, wherein the pattern is designed to limit a number of dots that relate to each pixel.

14. The apparatus of claim 1, wherein the depth for the spot is determined based on a relationship between pixels and depths.

15. The apparatus of claim 1, wherein the depth for the spot is determined based on a relationship between pixels and centroids.

16. The apparatus of claim 15, wherein the at least one processor is further configured to determine a centroid of the spot and determine the depth based on the centroid of the spot.

17. The apparatus of claim 15, wherein the relationship is specific to the dot.

18. The apparatus of claim 15, wherein the relationship comprises a polynomial.

19. The apparatus of claim 15, wherein the relationship relates a position of a centroid in the image to the depth.

20. The apparatus of claim 19, wherein the position of the centroid in the image has a subpixel resolution.

21. The apparatus of claim 15, wherein the relationship is based on a calibration process involving the pattern and an image sensor.

22. The apparatus of claim 1, wherein the at least one processor is further configured to cause a projector to project the pattern comprising the plurality of dots to illuminate the scene with the pattern.

23. The apparatus of claim 22, further comprising the projector.

24. The apparatus of claim 22, wherein the proj ector comprises at least one of a light source or a diffractive optical element.

25. The apparatus of claim 24, wherein the light source comprises one or more lasers.

26. The apparatus of claim 1, further comprising causing an image sensor to capture the image.

27. The apparatus of claim 26, further comprising the image sensor.

28. The apparatus of claim 1, wherein the at least one processor is further configured to: for each respective spot of the plurality of spots as captured in the image: determining a respective pixel of the image that corresponds to the respective spot; determining a respective dot of the plurality of dots of the pattern that corresponds to the respective pixel; and determining a respective depth for the respective spot based on the respective dot and the respective spot.

29. A method for determining depths, the method comprising: obtaining an image of a scene, wherein the scene is illuminated with a pattern comprising a plurality of dots, and wherein the image comprises a plurality of spots, each spot of the plurality of spots corresponding to a respective dot of the plurality of dots of the pattern; determining a pixel of the image that corresponds to a spot of the plurality of spots in the image; determining a dot of the plurality of dots of the pattern that corresponds to the pixel; and determining a depth for the spot based on the dot and the spot.

30. The method of claim 29, wherein the depth for the spot is determined based on a relationship between pixels and centroids.

Images