System and method for structured light depth computation using single photon avalanche diodes

Abstract

A system for structured light depth computation using single-photon avalanche diodes (SPADs) can be configured to selectively activate an illuminator to perform interleaved structured light illumination operations over a frame capture time period. The interleaved structured light illumination operations include alternately emitting at least a first structured light pattern from the illuminator and emitting at least a second structured light pattern from the illuminator. The system can also be configured to perform a plurality of sequential shutter operations to configure each SPAD pixel of a SPAD array to enable photon detection over the frame capture time period. The plurality of sequential shutter operations generate a plurality of binary counts for each SPAD pixel of the SPAD array indicating whether a photon was detected during each shutter operation of the plurality of sequential shutter operations.

Description

Background Technology

[0001] Depth maps include information indicating the distance between a viewpoint (e.g., the viewpoint of a camera capturing an image of the environment) and the surfaces of objects within the scene (e.g., objects within the environment captured by the camera). Depth maps are used as tools to facilitate a variety of computer vision tasks and / or user experiences, such as simultaneous localization and mapping, object tracking, pass-through imaging (e.g., capturing images and reprojecting them using depth maps to correspond to another viewpoint), photogrammetry, surface reconstruction, and / or others.

[0002] Depth maps can be generated in various ways. One exemplary technique for generating depth maps includes structured light depth imaging. One form of structured light depth imaging involves projecting a series of different light patterns (e.g., a stripe pattern with different numbers and / or widths of stripes) onto an environment and using a camera to capture a series of images of the environment. Each image captures the environment while the environment is illuminated by a different light pattern.

[0003] When capturing images, the camera is typically kept still, so that each image sensing pixel of the camera captures the same part of the environment during the projection and capture of different light patterns.

[0004] Selecting a light pattern from the series of different light patterns allows for the determination of a pixel signature for each image-sensing pixel of the camera. For example, the series of different light patterns may include an arrangement of vertical stripes formed by illuminated and unilluminated vertical portions arranged in an alternating pattern. Different light patterns may include vertical stripes of different densities. For example, a first light pattern may include a single illuminated vertical portion (occupying half of the pattern) and a single unilluminated vertical portion (occupying the other half of the pattern). A second light pattern may include four vertical stripes (two illuminated, two unilluminated, arranged in an alternating pattern), a third light pattern may include eight vertical stripes (four illuminated, four unilluminated, arranged in an alternating pattern), and so on.

[0005] Based on the above example, a pixel signature for a specific image sensing pixel can be defined by tracking whether a portion of the environment captured by the specific image sensing pixel is illuminated during each different light pattern projected into the environment. For example, a first value can be recorded indicating whether the specific image sensing pixel detected light pattern illumination when a first light pattern was projected into the environment (e.g., a binary "1" or "0", where "1" indicates detected illumination); a second value can be recorded indicating whether the specific image sensing pixel detected light pattern illumination when a second light pattern was projected into the environment; a third value can be recorded indicating whether the specific image sensing pixel detected light pattern illumination when a third light pattern was projected into the environment; and so on. In other words, the first value can indicate whether a portion of the environment captured by the specific image sensing pixel is within the illumination stripes of the first pattern, the second value can indicate whether a portion of the environment captured by the specific image sensing pixel is within the illumination stripes of the second pattern, the third value can indicate whether a portion of the environment captured by the specific image sensing pixel is within the illumination stripes of the third pattern, and so on. Various values ​​for a particular image-sensing pixel (e.g., light pattern illumination or unillumination during the projection of a series of different light patterns) can be combined (e.g., arranged as a sequence of values) to generate a pixel signature for that particular image-sensing pixel.

[0006] Pixel signatures of the camera's image-sensing pixels can be determined based on captured images of various structured light patterns projected into the environment, as discussed above. The system can also determine virtual pixel signatures based on structured light patterns projected into the environment. For example, virtual pixel signatures can each be associated with pixel coordinates, and can be determined based on whether the pixel coordinates correspond to illuminated or unilluminated portions across multiple patterns.

[0007] Therefore, to perform depth calculations, the system can search along scan lines (e.g., horizontal scan lines or scan directions orthogonal to the stripes of the structured light pattern) to identify the location of camera pixels associated with pixel signatures that match virtual pixel signatures on the same scan line. The system can use the pixel coordinates associated with the matching pixel signatures and virtual pixel signatures to determine parallax and / or depth values. Such matching can be performed or attempted to calculate depth values ​​for all camera pixels, thereby forming a depth map of the captured environment. To facilitate ideal structured light depth imaging as discussed above, a light pattern of a series of structured light patterns should be selected to allow the pixel signature of each image-sensing pixel for the camera to be unique to other pixel signatures at least along the same scan line (e.g., along the same horizontal scan line).

[0008] In some cases, structured light imaging, as described in the examples above, can provide high-precision depth information and reduce errors for the captured environment compared to other conventional techniques for depth imaging, such as using single-point patterning to facilitate active stereo imaging. However, environments including moving objects present many challenges to structured light imaging utilizing a series of light patterns as discussed above. For example, structured light images are typically captured using complementary metal-oxide-semiconductor (CMOS) and / or charge-coupled device (CCD) image sensors. Such sensors may include an array of image-sensing pixels, where each pixel is configured to generate electron-hole pairs in response to detected photons. Electrons may be converted into charges stored in a capacitor per pixel, and the charge stored in the capacitor can be read out to provide image data (e.g., by converting the stored charge into a voltage).

[0009] CMOS and / or CCD image sensors typically operate by performing an exposure operation to allow charge to be collected in a capacitor at each pixel and then performing a readout operation to generate image data based on the collected charge at each pixel. Therefore, a moving object in the captured environment often occupies one location (or set of locations) during the exposure and / or readout of a first light pattern image and a different location (or different set of locations) during the exposure and / or readout of a second light pattern image. Consequently, different light pattern images can include spatially misaligned representations of the same object in the environment, leading to errors in depth calculation.

[0010] Therefore, at least for the reasons mentioned above, there is a continued need and expectation for improved systems and methods to facilitate structured optical depth computation.

[0011] The subject matter claimed herein is not limited to embodiments that address any shortcomings or operate only in environments such as those described above. Rather, this background is provided merely to illustrate an exemplary technical field in which some of the embodiments described herein can be practiced. Summary of the Invention

[0012] The disclosed embodiments include systems, methods, and apparatus for facilitating structured optical depth calculations using single-photon avalanche diodes (SPADs).

[0013] Some embodiments provide a system comprising: a SPAD array including a plurality of SPAD pixels; an illuminator configured to emit one or more structured light patterns; one or more processors; and one or more hardware storage devices storing instructions executable by the one or more processors to configure the system to perform various actions. The actions include: selectively activating the illuminator to perform interleaved structured light illumination operations during a frame capture period. The interleaved structured light illumination operations include alternately emitting at least a first structured light pattern from the illuminator and emitting at least a second structured light pattern from the illuminator. The actions also include: performing a plurality of sequential shutter operations during the frame capture period to configure each SPAD pixel of the SPAD array to enable photon detection. The plurality of sequential shutter operations generate a plurality of binary counts for each SPAD pixel of the SPAD array, indicating whether a photon was detected during each of the plurality of sequential shutter operations.

[0014] This overview is provided to present a simplified selection of concepts, which will be further described in the detailed description below. This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0015] Additional features and advantages will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practice of the teachings herein. The features and advantages of the invention can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. The features of the invention will become more apparent from the description which follows and the appended claims, or may be learned by practice of the invention as set forth below. Attached Figure Description

[0016] To describe how the above and other advantages and features can be obtained, the subject matter briefly described above will be described in more detail with reference to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings merely depict typical embodiments and therefore should not be considered as limiting the scope. The embodiments will be described and explained with additional features and details using the drawings, wherein:

[0017] Figure 1 The illustrations depict exemplary components that may include or be used to implement one or more of the disclosed embodiments;

[0018] Figure 2 An exemplary implementation of a single-photon avalanche diode (SPAD) sensor in a head-mounted display (HMD) is illustrated.

[0019] Figures 3A-3E The illustration shows examples of SPAD exposure operations performed during the frame capture time period under illumination by different structured light patterns;

[0020] Figure 3F An exemplary image is illustrated, generated based on a binary count obtained via shutter operations performed during the frame capture period.

[0021] Figures 4A-4C Figures 5A-5B illustrate examples of determining pixel signatures associated with SPAD pixels;

[0022] Figure 6 The illustration shows an example of determining a virtual pixel signature;

[0023] Figure 7 The illustration shows an example of generating disparity values ​​based on pixel signatures and corresponding virtual pixel signatures; and

[0024] Figure 8 An exemplary flowchart depicting the actions associated with structured optical depth calculations using SPAD is illustrated. Detailed Implementation

[0025] The disclosed embodiments generally relate to systems, methods, and apparatus for facilitating structured optical depth calculations using single-photon avalanche diodes (SPADs).

[0026] Examples of technological benefits, improvements, and practical applications

[0027] In view of this disclosure, those skilled in the art will recognize that at least some of the embodiments disclosed can be implemented to address various drawbacks associated with at least some conventional structured optical depth computing techniques. The following sections outline some exemplary improvements and / or practical applications provided by the disclosed embodiments. However, it will be appreciated that the following are merely examples and the embodiments described herein are by no means limited to the exemplary improvements discussed herein.

[0028] Compared to conventional CMOS or CCD sensors, SPADs operate at a bias voltage that enables them to detect individual photons. Upon detection of a single photon, an electron-hole pair is formed, and the electron is accelerated across a high electric field, causing avalanche multiplication (e.g., generating additional electron-hole pairs). Therefore, each detected photon can potentially trigger an avalanche event. SPADs can be operated in a gated manner (each gate corresponding to a separate shutter operation), where each gated shutter operation can be configured to result in a binary output. The binary output can include a "1" indicating that an avalanche event was detected during exposure (e.g., a photon was detected there), or a "0" indicating that no avalanche event was detected.

[0029] Individual shutter operations can be integrated within a frame capture period. The binary output of shutter operations within the frame capture period can be counted, and intensity values ​​can be calculated based on the counted binary output.

[0030] An array of SPADs can form an image sensor, where each SPAD forms an individual pixel in the SPAD array. To capture an image of the environment, each SPAD pixel can detect avalanche events and provide a binary output for sequential shutter operations in a manner described herein. The per-pixel binary output of multiple shutter operations within a frame capture time period can be counted, and a per-pixel intensity value can be calculated based on the counted per-pixel binary output. This per-pixel intensity value can be used to form an intensity image of the environment.

[0031] As will be described in more detail below, the techniques disclosed herein include using SPAD pixels of a SPAD array to perform interleaved ambient exposures. During a first light pattern exposure, sequential shutter operations are performed using the SPAD pixels to generate binary counts, where each binary count indicates whether a photon was detected during the shutter operation (e.g., "1" indicates a photon was detected, and "0" indicates no photon was detected). The sequential shutter operations of the first light pattern exposure are performed while the scene is captured according to the first light pattern illumination. During a second light pattern exposure, sequential shutter operations are performed using the SPAD pixels while the illuminator emits the second light pattern into the captured scene. These shutter operations also provide binary counts. Different light pattern exposures and corresponding light pattern illuminations can be performed in an interleaved manner (e.g., first light pattern illumination and exposure, second light pattern illumination and exposure, first light pattern illumination and exposure, etc.). Additional light pattern exposures under additional light pattern illumination can also be performed and interleaved (e.g., third light pattern illumination and exposure, fourth light pattern illumination and exposure, nth light pattern illumination and exposure).

[0032] Interleaved light pattern exposures are performed alternately (e.g., one after another) within the frame capture period, thereby providing multiple subsets of the binary count. For example, a first subset of the binary count may be associated with a first light pattern exposure, a second subsequent subset of the binary count may be associated with a second light pattern exposure, a third subsequent subset of the binary count may be associated with the first light pattern exposure, a fourth subsequent subset of the binary count may be associated with the second light pattern exposure, a fifth subsequent subset of the binary count may be associated with the first light pattern exposure, and so on. As described above, additional subsets of the binary count may correspond to additional light pattern illumination and exposure, and may be interleaved with the above.

[0033] Therefore, it is possible to generate subsets of binary counts associated with the same light pattern exposure that are temporally discontinuous. Furthermore, a subset of binary counts associated with one light pattern exposure (e.g., the second light pattern exposure) can be temporally between two subsets of binary counts associated with another light pattern exposure (e.g., the first light pattern exposure), and vice versa.

[0034] The system can then use all subsets of the binary counts associated with each specific light pattern exposure (e.g., the first light pattern exposure, the second light pattern exposure, etc.) to generate a corresponding light pattern image for that specific light pattern exposure, even if the subsets of binary counts associated with the specific light pattern exposure are not sequential in time. The different corresponding light pattern images can then be used to generate pixel signatures for calculating depth information.

[0035] Using a SPAD image sensor to capture light pattern images, as described herein, offers numerous advantages over conventional systems and techniques for capturing light pattern images (e.g., for structured optical depth calculations). As mentioned above, conventional techniques for capturing light pattern images involve fully exposing and reading out a first light pattern image using a CMOS or CCD sensor before proceeding with full exposure and readout of subsequent light pattern images. This can cause motion artifacts and / or spatial misalignment between different light pattern images. Conversely, by staggering the exposure of different light patterns according to this disclosure, motion (and / or motion of one or more image sensors) in the captured scene will affect all light pattern images in a similar manner, thereby mitigating motion artifacts and / or spatial misalignment between light pattern images. Therefore, the accuracy and / or usability of structured optical depth imaging can be improved.

[0036] Having just described some of the various advanced features and benefits of the disclosed embodiments, we will now turn our attention to... Figures 1 to 8 These figures illustrate various conceptual representations, architectures, methods, and supporting descriptions related to the disclosed embodiments.

[0037] Exemplary systems and techniques for structured optical depth computation using SPAD

[0038] Figure 1 Various exemplary components that can be used to implement one or more of the disclosed embodiments of system 100 are illustrated. For example, Figure 1 The diagram illustrates that system 100 may include one or more processors 102, storage devices 104, one or more sensors 110, one or more SPAD arrays 112, one or more input / output systems 114 (I / O systems 114), and one or more communication systems 116. Although Figure 1System 100 is illustrated as including specific components, but in view of this disclosure, one will appreciate that system 100 may include any number of additional or alternative components.

[0039] One or more processors 102 may include one or more sets of electronic circuitry comprising any number of logic units, registers, and / or control units to facilitate the execution of computer-readable instructions (e.g., instructions forming a computer program). Such computer-readable instructions may be stored within storage device 104. Storage device 104 may include physical system memory and may be volatile, non-volatile, or some combination thereof. Furthermore, storage device 104 may include local storage, remote storage (e.g., accessible via one or more communication systems 116 or otherwise), or some combination thereof. Additional details relating to processors (e.g., one or more processors 102) and computer storage media (e.g., storage device 104) will be provided below.

[0040] In some implementations, processor(s)102 may include or be configured to execute any combination of software and / or hardware components operable to facilitate processing using machine learning models or other artificial intelligence-based architectures. For example, processor(s) 102 may include and / or utilize hardware components or computer-executable instructions operable for executing function blocks and / or processing layers, which are configured in a non-limiting example manner as: single-layer neural network, feedforward neural network, radial basis function network, deep feedforward network, recurrent neural network, long short-term memory (LSTM) network, gated recurrent unit, autoencoder neural network, variational autoencoder, denoising autoencoder, sparse autoencoder, Markov chain, Hopfield neural network, Boltzmann machine network, restricted Boltzmann machine network, deep belief network, deep convolutional network (or convolutional neural network), deconvolutional neural network, deep convolutional inverse graph network, generative adversarial network, liquid state machine, extreme learning machine, echo state network, deep residual network, Kohonen network, support vector machine, neural Turing machine and / or others.

[0041] As will be described in more detail, processor(s) 102 may be configured to execute instructions 106 stored within storage device 104 to perform specific actions associated with imaging using a SPAD array. These actions may depend at least in part on data 108 (e.g., avalanche event counting or tracking, etc.) stored on storage device 104 in a volatile or non-volatile manner.

[0042] In some cases, the operation may rely at least in part on one or more communication systems 116 for receiving data from one or more remote systems 118, which may include, for example, separate systems or computing devices, sensors, and / or others. The communication systems 118 may include any combination of software or hardware components operable to facilitate communication between components / devices on the system and / or with components / devices outside the system. For example, the communication systems 118 may include ports, buses, or other physical connection means for communicating with other devices / components. Additionally or alternatively, the communication systems 118 may include systems / components operable to wirelessly communicate with external systems and / or devices via any suitable communication channel (e.g., Bluetooth, ultra-wideband, WLAN, infrared communication, and / or others, as non-limiting examples).

[0043] Figure 1 The system 100 illustrated may include or communicate with one or more sensors 110. The one or more sensors 110 may include any device for capturing or measuring data representing a perceptible phenomenon. As a non-limiting example, the one or more sensors 110 may include one or more image sensors, microphones, thermometers, barometers, magnetometers, accelerometers, gyroscopes, and / or others.

[0044] Figure 1 The diagram also illustrates that sensor(s) 110 may include SPAD array(s) 112. For example... Figure 1 As depicted, the SPAD array 112 may include an arrangement of SPAD pixels 120, each SPAD pixel 120 being configured to facilitate avalanche events in response to sensed photons, as described above. One or more SPAD arrays 112 may be implemented on system 100 (e.g., MR HMD) to facilitate various functions such as image capture and / or computer vision tasks.

[0045] also, Figure 1 The illustration shows that system 100 may include or communicate with one or more I / O systems 114. The one or more I / O systems 114 may include any type of input or output device, such as, by way of non-limiting example, a touchscreen, mouse, keyboard, controller, and / or others, but are not limited thereto. For example, the one or more I / O systems 114 may include a display system, which may include any number of display panels, optics, laser scanning display components, and / or other components.

[0046] Figure 1The components of system 100 can conceptually represent various types of devices, such as mobile electronic devices 100A (e.g., smartphones), personal computing devices 100B (e.g., laptop computers), mixed reality head-mounted displays 100C (HMD 100C), aircraft 100D (e.g., drones), and / or other devices. Although this description focuses on utilizing HMDs to implement the techniques of this disclosure in at least some aspects, additional or alternative types of systems may be used.

[0047] Figure 2 An exemplary implementation of a single-photon avalanche diode (SPAD) sensor in a head-mounted display (HMD) is illustrated. Specifically, Figure 2 An exemplary HMD 200 is illustrated, which may correspond in at least some respects to the references above. Figure 1 The system 100 described. In Figure 2 In the example shown, HMD 200 includes a SPAD sensor 202. SPAD sensor 202 can be configured to capture intensity image frames as described above (e.g., by counting photons or avalanche events detected during the frame capture period on a per-pixel basis). The intensity images captured using SPAD sensor 202 can be used for a variety of purposes, such as facilitating image / video capture, pass-through imaging, depth computation (e.g., structured light depth imaging), object tracking, object segmentation, surface reconstruction, simultaneous localization and mapping (SLAM), and / or others. HMD 200 can implement any number (one or more) of other sensors 208 for the same, additional, or alternative purposes.

[0048] Figure 2 The HMD 200, including illuminator 204, is also illustrated. Illuminator 204 can take the form of any suitable light-emitting device. As will be described herein, a system (e.g., HMD 200 or other device) can be combined with one or more SPAD sensors (e.g., SPAD sensor 202) to operate the illuminator (e.g., illuminator 204) to capture images that can be used to facilitate structured light depth calculations. For example, the system can selectively (or iteratively) activate the illuminator (e.g., in a pulsed manner) to allow one or more SPAD sensors 202A-202E to capture images of the environment while illuminating the environment with different structured light patterns. The system can combine SPAD sensor 202 with illuminator 204 as a “virtual camera” to facilitate structured light depth calculations.

[0049] According to this disclosure, illuminator 204 can take various forms to facilitate various types of illumination for capturing various types of illuminated images. For example, illuminator 204 can be configured to emit light in the visible, infrared, ultraviolet, or combinations thereof and / or other spectral ranges. In some cases, as noted above, illuminator 204 can be configured to emit one or more structured light patterns, such as striped structured light patterns. Although this disclosure focuses in at least some aspects on utilizing illuminators to emit striped structured light patterns to facilitate structured light depth calculation, in view of this disclosure, one will appreciate that additional or alternative types of illumination can be used. For example, grayscale illumination, sinusoidal patterns, random patterns, and / or others can be used.

[0050] In view of this disclosure, one will realize that, despite Figure 2 Only a single illuminator 204 on the HMD 200 is illustrated, but any number of illuminators can be used with any type of system to practice the techniques disclosed herein.

[0051] Figures 3A-3E The illustration shows an example of a SPAD exposure operation performed during a frame capture period. Specifically, Figure 3A The illustration shows a SPAD array 300, which corresponds to the reference above. Figure 1 The described SPAD array(s) 112. In this aspect, each SPAD pixel (e.g., SPAD pixel 302) of the SPAD array 300 can be configured to trigger an avalanche event in response to the detection of a photon. The SPAD pixel 302 can be operated in a controlled gating manner to facilitate different exposure operations for image acquisition.

[0052] Figure 3A An ellipsis 340 is used to depict the SPAD pixels of the SPAD array 300 and the representation of the exposure operation, which will be described in more detail below. The ellipsis 340 indicates that any number of SPAD pixels (e.g., all SPAD pixels) of the SPAD array 300 can be used to perform the exposure operation described below. For clarity and simplicity, it is intended that... Figure 3A A single representation of the exposure operation is shown in (and in the subsequent figures). Figure 3A Additionally, a time axis t is provided to illustrate the temporal relationships between the different exposure operations described below.

[0053] Figure 3A As shown, to facilitate the capture of image frames usable for structured optical depth calculations, the SPAD pixels of the SPAD array 300 are configured to perform multiple exposure operations in an interleaved manner. Specifically, Figure 3A The illustration shows the ambient exposure operation performed during frame capture time period 304 (in Figure 3AThe image is marked "A" in the middle, and the exposure operation of Pattern 1 is shown in the middle. Figure 3A The pattern is marked "P1" and referred to below as "P1 exposure operation"), and the pattern 2 exposure operation (in Figure 3A The pattern is marked "P2" and referred to below as "P2 exposure operation" and the pattern 3 exposure operation (in Figure 3A This is marked as "P3" and referred to below as "P3 Exposure Operation". Figure 3A In the example shown, the ambient exposure operation, the P1 exposure operation, the P2 exposure operation, and the P3 exposure operation are executed in an interleaved manner, wherein the set of exposure operations includes the ambient exposure operation, the P1 exposure operation, the P2 exposure operation, and the P3 exposure operation executed sequentially.

[0054] The ellipsis 390 indicates that any number of ambient exposure operations, P1 exposure operations, P2 exposure operations, and / or P3 exposure operations can be performed within the frame capture time period 304. Furthermore, although this example focuses at least in some respects on utilizing ambient exposure operations, P1 exposure operations, P2 exposure operations, and P3 exposure operations, ambient exposure operations can be omitted from the frame capture time period 304, and / or additional or fewer pattern exposure operations can be interleaved within the frame capture time period 304.

[0055] Figure 3A The illustration also shows (via dashed lines extending downwards from the first ambient exposure operation) an ambient exposure operation comprising a set of applied shutter operations 306. As described above, the SPAD pixel is configured for photon detection by applying a shutter operation to it in response to a detected photon triggering an avalanche event. Therefore, the presence of an avalanche event during the shutter operation indicates that the SPAD pixel detected a photon during the shutter operation. Whether the SPAD pixel experiences an avalanche event (and thus detects a photon) during the shutter operation can be represented by a binary "1" or "0", where "1" indicates that an avalanche event occurred during the shutter operation, and "0" indicates that no avalanche event occurred during the shutter operation. Therefore, Figure 3A A binary count 310 is depicted associated with each shutter operation 306. The ellipsis 392 indicates that an ambient exposure operation may include any suitable number of shutter operations 306.

[0056] Figure 3A It also depicts the execution of an individual shutter operation 306 within a specific gating time 308. The gating time 308 is the duration during which the SPAD pixel becomes configured to trigger an avalanche event in response to the detection of a photon.

[0057] although Figure 3AOnly the set of shutter operations 306 associated with the first ambient exposure operation is explicitly described, but other sets of shutter operations can be applied to facilitate other exposures (i.e., P1, P2, and / or P3 exposure operations). For each P1 exposure operation, shutter operation 306 is performed while selectively activating (or pulsed) an illuminator (e.g., illuminator 204) to project pattern 1 into the environment. Figure 3A As depicted, pattern 1 includes illuminated vertical stripes (shown in white on the left) and unilluminated vertical stripes (shown in black on the right). For each P2 exposure operation, shutter operation 306 is performed while selectively activating (or pulsed) the illuminator (e.g., illuminator 204) to project pattern 2 into the environment. Figure 3A As depicted, pattern 2 comprises two illuminated vertical stripes and two unilluminated vertical stripes arranged in an alternating manner. Furthermore, for each P3 exposure operation, shutter operation 306 is performed while selectively activating (or pulsed) the illuminator (e.g., illuminator 204) to project pattern 3 into the environment. Figure 3A As depicted, pattern 3 comprises four illuminated vertical stripes and four unilluminated vertical stripes arranged in an alternating manner. Conversely, for each ambient exposure operation, shutter operation 306 is performed while one or more illuminators selectively refrain from illuminating the environment. At this point, during the frame capture period, one or more illuminators (e.g., including illuminator 204) alternately emit different light patterns and refrain from refraining from illuminating light.

[0058] As described above, ellipsis 390 indicates that any number of ambient exposure operations and pattern exposure operations can be performed within the frame capture time period 304. As will be described in more detail below, the results of ambient exposure operations can be combined to form an ambient image, and the results of different pattern exposure operations can be combined to form different pattern images. To provide the desired image, in some cases, at least two pattern exposure operations are performed for each light pattern used within the frame capture time period 304. Furthermore, in some cases, the system suppresses the pause execution of shutter operation 306 within the frame capture time period 304 to avoid motion artifacts. For example, the system can suppress the pause execution of shutter operation to perform a readout operation (although, in some configurations, the readout operation can be passively performed during the frame capture time period 304 without pausing the shutter operation).

[0059] The shutter operation 306 performed during each individual ambient exposure operation (i.e., (one or more) illuminators do not emit light) can provide a separate subset of binary counts. For example, Figure 3BThe diagram illustrates various subsets of the binary count, 312A, 312B, and 312C. Each subset of the binary count, 312A, 312B, and 312C, is associated with a different corresponding environmental exposure operation (in...). Figure 3B (This is illustrated by arrows representing various subsets of 312A, 312B, and 312C, extending from various environmental exposure operations to binary counting). As indicated above, and as in... Figure 3B As depicted, the subsets 312A, 312B, and 312C of binary counts are not generated sequentially in time (i.e., because the pattern exposure operation is between the environment exposure operation). The subsets 312A, 312B, and 312C of binary counts can be combined to form a set 314 of binary counts, and the set 314 of binary counts can be used to generate an environment image, even though the subsets 312A, 312B, and 312C of binary counts forming the set 314 are not sequential in time (see [reference]). Figure 3F ).

[0060] Similarly, the shutter operation 306 performed during each individual P1 exposure operation (i.e., during the emission of pattern 1 by one or more illuminators) can provide a separate subset of binary counts. For example, Figure 3C The diagram illustrates various subsets of the binary count, 316A, 316B, and 316C. Each subset of the binary count, 316A, 316B, and 316C, is associated with a different corresponding P1 exposure operation (in...). Figure 3C (This is illustrated by arrows representing various subsets of the 316A, 316B, and 316C, extending from various P1 exposure operations to binary counting). As indicated above, and as in... Figure 3C As depicted, the various subsets 316A, 316B, and 316C of binary counts are not generated sequentially in time (i.e., because the ambient exposure operation, the P2 exposure operation, and the P3 exposure operation are between the P1 exposure operation). The various subsets 316A, 316B, and 316C of binary counts can be combined to form a set 318 of binary counts, and the set 318 of binary counts can be used to generate the image of Pattern 1, even though the various subsets 316A, 316B, and 316C of binary counts forming the set 318 are not sequential in time (see [reference]). Figure 3F ).

[0061] Similarly, the shutter operation 306 performed during each individual P2 exposure operation (i.e., during the emission of pattern 2 by one or more illuminators) can provide a separate subset of binary counts. For example, Figure 3DThe diagram illustrates various subsets of the binary count, 320A, 320B, and 320C. Each subset of the binary count, 320A, 320B, and 320C, is associated with a different corresponding P2 exposure operation (in...). Figure 3D (This is illustrated by arrows representing various subsets of the 320A, 320B, and 320C, extending from various P2 exposure operations to binary counting). As indicated above, and as in... Figure 3D As depicted, the various subsets 320A, 320B, and 320C of binary counts are not generated sequentially in time (i.e., because the ambient exposure operation, the P1 exposure operation, and the P3 exposure operation are between the P2 exposure operation). The subsets 320A, 320B, and 320C of binary counts can be combined to form a set 322 of binary counts, and this set 322 can be used to generate the image of Pattern 2, even though the subsets 320A, 320B, and 320C of binary counts forming the set 322 are not sequential in time (see [reference]). Figure 3F ).

[0062] Additionally, the shutter operation 306 performed during each individual P3 exposure operation (i.e., during the emission of pattern 3 by one or more illuminators) can provide a separate subset of binary counts. For example, Figure 3E The diagram illustrates various subsets of the binary count, 324A, 324B, and 324C. Each subset of the binary count, 324A, 324B, and 324C, is associated with a different corresponding P3 exposure operation (in...). Figure 3E (This is illustrated by arrows representing various subsets of 324A, 324B, and 324C, extending from various P3 exposure operations to binary counting). As indicated above, and as in... Figure 3E As depicted, the various subsets 324A, 324B, and 324C of binary counts are not generated sequentially in time (i.e., because the ambient exposure operation, the P1 exposure operation, and the P2 exposure operation are between the P3 exposure operation). The various subsets 324A, 324B, and 324C of binary counts can be combined to form a set 326 of binary counts, and the set 326 of binary counts can be used to generate the pattern 3 image, even though the various subsets 324A, 324B, and 324C of binary counts forming the set 3182 of binary counts are not sequential in time (see [link to documentation]). Figure 3F ).

[0063] although Figure 3BOnly a single set 314 of binary counts obtained from shutter operation 306 performed by a single SPAD pixel 302 during the ambient exposure operation within the frame capture time period 304 is illustrated, but a separate set of binary counts from the ambient exposure operation can be generated for each SPAD pixel of the SPAD array 300. Similarly, although Figure 3C Only a single set 318 of binary counts obtained during the P1 exposure operation within the frame capture time period 304 from the shutter operation 306 performed by a single SPAD pixel 302 is illustrated, but a separate set of binary counts from the P1 exposure operation can be generated for each SPAD pixel of the SPAD array 300. Similarly, although Figure 3D Only a single set 322 of binary counts obtained from the shutter operation 306 performed by a single SPAD pixel 302 during the P2 exposure operation within the frame capture time period 304 is illustrated, but a single set of binary counts from the P2 exposure operation can be generated for each SPAD pixel of the SPAD array 300. Furthermore, although Figure 3E Only a single set 326 of binary counts obtained from shutter operation 306 performed by a single SPAD pixel 302 during the P3 exposure operation within the frame capture time period 304 is illustrated, but a separate set of binary counts from the P3 exposure operation can be generated for each SPAD pixel of the SPAD array 300.

[0064] Figure 3F The illustration shows a set 328 of ambient exposure binary counts, which includes each set of binary counts generated by each SPAD pixel of the SPAD array 300 during ambient exposure operation (see [link]). Figure 3B ). Each set of binary counts in the set of 328 binary counts of environmental exposure (e.g., including those from Figure 3B The set of binary counts 314 includes corresponding subsets of binary counts generated by SPAD pixels (e.g., SPAD pixel 302) during ambient exposure operations within the frame capture time period 304 (e.g., subsets 312A-312C for the set of binary counts 314).

[0065] Figure 3F The illustration also illustrates ambient readout 330 performed using a set 328 of ambient exposure binary counts. For a SPAD array, "readout" involves determining or outputting the number of photons detected per pixel within the shutter operation set. Pixels capturing the brighter portions of the captured environment will count more photons than pixels capturing the darker portions of the captured environment. Therefore, the number of photons per pixel detected in the shutter operation set can be used to form an intensity image.

[0066] Therefore, ambient readout 330 may include determining or outputting the number of photons represented by each set of binary counts in a set 328 of ambient exposure binary counts, where each set corresponds to a different SPAD pixel. The per-pixel intensity value may be determined based on the number of photons from each set, and the per-pixel intensity value may be used to generate the ambient image 32.

[0067] Similarly, Figure 3F The illustration shows a set 334 of binary counts for P1 exposure, which includes each set of binary counts generated by each SPAD pixel of the SPAD array 300 during the P1 exposure operation (see [link]). Figure 3C P1 exposes each set of binary counts in the set 334 of binary counts (e.g., including those from...). Figure 3C The set of binary counts 318 includes corresponding subsets of binary counts generated by SPAD pixels (e.g., SPAD pixel 302) during the P1 exposure operation within the frame capture time period 304 (e.g., subsets 316A-316C for the set of binary counts 318).

[0068] Figure 3F The diagram also illustrates P1 readout 336 performed using a set 334 of P1 exposure binary counts. P1 readout 328 may include determining or outputting the number of photons represented by each set of binary counts in the set 334 of P1 exposure binary counts, where each set corresponds to a different SPAD pixel. A per-pixel intensity value may be determined based on the number of photons from each set, and the per-pixel intensity value may be used to generate Pattern 1 image 338.

[0069] also, Figure 3F The illustration shows a set 340 of binary counts for P2 exposure, which includes each set of binary counts generated by each SPAD pixel of the SPAD array 300 during the P2 exposure operation (see [link]). Figure 3D P2 exposes each set of binary counts in the set 340 of binary counts (e.g., including those from...). Figure 3D The set of binary counts 322 includes corresponding subsets of binary counts generated by SPAD pixels (e.g., subsets 320A-320C of the set of binary counts 322) during the P2 exposure operation within the frame capture time period 304.

[0070] Figure 3FThe diagram also illustrates P2 readout 342 performed using a set 340 of P2 exposure binary counts. P1 readout 342 may include determining or outputting the number of photons represented by each set of binary counts in the set 340 of P2 exposure binary counts, where each set corresponds to a different SPAD pixel. The per-pixel intensity value may be determined based on the number of photons from each set, and the per-pixel intensity value may be used to generate Pattern 2 image 344.

[0071] Similarly, Figure 3F The illustration shows a set 346 of P3 exposure binary counts, which includes each set of binary counts generated by each SPAD pixel of the SPAD array 300 during the P3 exposure operation (see [link]). Figure 3E P3 exposes each set of binary counts in the set of 346 binary counts (e.g., including those from...). Figure 3E The set of binary counts 326 includes corresponding subsets of binary counts generated by SPAD pixels (e.g., SPAD pixel 302) during the P3 exposure operation within the frame capture time period 304 (e.g., subsets 324A-324C for the set of binary counts 326).

[0072] Figure 3F The diagram also illustrates a P1 readout 348 performed using a set 346 of P3 exposure binary counts. The P1 readout 348 may include determining or outputting the number of photons represented by each set of binary counts in the set 348 of P3 exposure binary counts, where each set corresponds to a different SPAD pixel. A per-pixel intensity value may be determined based on the number of photons from each set, and the per-pixel intensity value may be used to generate pattern 3 image 350.

[0073] Figure 3F The illustration shows an environmental image 332 representing a moving ball captured by the SPAD array 300. Figure 3F Pattern 1 image 338, Pattern 2 image 344, and Pattern 3 image 350 are also illustrated, as each image includes a corresponding representation of the same moving ball captured by the SPAD array 300 under its respective lighting conditions. Because the ambient exposure operation used to generate ambient image 332 is interleaved with the pattern illumination operation used to generate pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350 within the same frame capture time period 304, the moving ball is depicted in a spatially aligned manner in ambient image 324, pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350. In other words, the movement of the ball will affect the capture of ambient image 324, pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350 in substantially the same way.

[0074] Because the representations of the moving ball are essentially spatially aligned in Pattern 1 image 338, Pattern 2 image 344, and Pattern 3 image 350, these images are well-suited for determining pixel signatures to facilitate depth calculations. Figures 4A-4C and Figures 5A-5B The illustration shows an example of determining the pixel signature associated with a SPAD pixel. Specifically, Figure 4A The illustrations show environmental image 322, pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350. Figure 4A Depicting the reference above Figures 3A-3E The SPAD pixel 302 discussed (in Figure 4A The image pixel of the environment image 332 (marked as "302") is described below. The following discussion describes how to obtain the pixel signature of SPAD pixel 302 (or its corresponding image pixel).

[0075] Figure 4A The illustration shows a set 314 of binary counts generated by shutter operation 306 based on SPAD pixel 302 during an ambient exposure operation (see reference above). Figure 3B (As described). Figure 4A An ambient light measurement result 402, determined based on a set 314 of binary counts, is also illustrated. The ambient light measurement result 402 includes a representation of the number of detected photons indicated in the set 314 of binary counts. In this example, the set 314 of binary counts indicates that 6 photons were detected at SPAD pixel 302 during an ambient exposure operation, providing an ambient light measurement result 402 of 6 photons (as shown in...). Figure 4A (As shown in the illustration).

[0076] As in Figure 4A As shown, the ambient light measurement result 402 can be used to determine the photon threshold number 404. In some cases, the photon threshold number 404 indicates the number of photons that can be used to determine whether the SPAD pixel 302 captures a portion of the scene illuminated by the structured light pattern when capturing different pattern images. In some cases, the photon threshold number 402 is greater than the ambient light measurement result 402.

[0077] As will be described below, the number of photon thresholds 404 can be used to determine the signature value of the SPAD pixel 302 associated with the capture of each pattern image (i.e., pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350), and the pixel signature for the SPAD pixel 302 can be based on the signature value.

[0078] Figure 4BThe illustration shows pattern 1 image 338, pattern 2 image 344 and pattern 3 image 350, and a representation of SPAD pixels 302 on each pattern image to visually depict portions of various pattern images captured by SPAD pixels 302. Figure 4B Includes a set 318 of binary counts captured by SPAD pixels 302 during the P1 exposure operation used to generate pattern 1 image 338. Figure 4B The diagram also illustrates the number of photons 406 (i.e., 15 photons) determined based on the set 318 of binary counts, and shows the signature value 408 determined based on whether the number of photons 406 satisfies the photon threshold number 404 discussed above.

[0079] As in Figure 4B As depicted, the arrow extends from photon count 406 to the decision box within signature value 408. Figure 4B In the example shown, if the number of photons 406 meets the photon threshold number 404, the signature value 408 is defined as the value "1", and if the number of photons 406 fails to meet the photon threshold number 404, the signature value 408 is defined as the value "0". Figure 4B An example is illustrated where, during the P1 exposure operation performed within frame capture time period 304 to generate pattern 1 image 338, SPAD pixel 302 captures a portion of the environment illuminated by pattern 1 projected by the illuminator (see [link]). Figure 4B Pattern 1 (image 338) in the image. Therefore, Figure 4B The number of photons 406, represented by the set 318 of binary counts, is illustrated as 15 photons, which far exceeds the ambient light measurement result 402, and therefore, in this example, the photon threshold number 404 is satisfied. Therefore, Figure 4B The “Yes” and “1” elements within the signature value 408 are shown in bold and underlined format, indicating that the photon quantity 406 satisfies the photon threshold quantity 404, resulting in the signature value 408 for the pattern 1 portion of the SPAD pixel 302 being “1”.

[0080] Figure 4B Similarly, a set 322 of binary counts captured by SPAD pixels 302 during the P2 exposure operation used to generate pattern 2 image 344 is shown. Figure 4B The diagram also illustrates the number of photons 410 (i.e., 6 photons) determined based on a set 322 of binary counts, and shows the signature value 412 determined based on whether the number of photons 410 satisfies the photon threshold number 404 discussed above. (As in...) Figure 4B As shown, the number of photons 410 represented by the set 322 of binary counts is 6 photons, which does not exceed the ambient light measurement result 402, and therefore, in this example, does not exceed the photon threshold number 404. Figure 4BThe “No” and “0” elements within the signature value 412 are shown in bold and underlined format, indicating that the photon quantity 412 failed to meet the photon threshold quantity 404, resulting in a signature value 408 of “0” for the pattern 2 portion of the SPAD pixel 302.

[0081] also, Figure 4B The illustration shows a set 326 of binary counts captured by SPAD pixels 302 during the P3 exposure operation used to generate pattern 3 image 350. Figure 4B The diagram also illustrates the number of photons 414 (i.e., 15 photons) determined based on a set 326 of binary counts, and shows the signature value 416 determined based on whether the number of photons 414 satisfies the photon threshold number 404 discussed above. (As in...) Figure 4B As shown, the number of photons 414, represented by the set 326 of binary counts, is 15 photons, which far exceeds the ambient light measurement result 402, and therefore, in this example, the photon threshold number 404 is satisfied. Therefore, Figure 4B The “yes” and “1” elements within the signature value 416 are shown in bold and underlined format, indicating that the photon quantity 416 satisfies the photon threshold quantity 404, resulting in a signature value 408 of “1” for the pattern 3 portion of the SPAD pixel 302.

[0082] Although the aforementioned example uses the measured number of photons to determine the signature value, other metrics based on a set of binary counts (e.g., the average number of photons) can be used.

[0083] Signature values ​​408, 412, and 416 can be combined to form a pixel signature associated with SPAD pixel 302. Because signature values ​​408, 412, and 416 are determined based on sets of binary counts 318, 322, and 326, respectively, the pixel signature is also based on binary counts 318, 322, and 326. Figure 4C The illustration shows a pixel signature 418 formed from signature values ​​408, 412, and 416 using a sequence of signature values ​​408, 412, and 416 as identifiers. Given this disclosure, it will be appreciated that pixel signatures can take other forms, and signature values ​​(which can also take other forms) can be combined and / or transformed in various ways to generate pixel signatures.

[0084] It will be appreciated that, in view of this disclosure, pixel signatures can be determined for any number of SPAD pixels (or image pixels associated with SPAD pixels). Figures 4A-4C At least some aspects focus on techniques for generating pixel signatures that utilize the number of photon thresholds per pixel determined based on an environmental image 332 (e.g., based on per-pixel ambient light measurements 402). However, other techniques can be used to generate pixel signatures.

[0085] The following text is for reference only. Figure 5A and Figure 5B Additional techniques for generating pixel signatures are discussed. Figure 5A The illustrations show environmental image 332, pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350. Figure 5A The illustration shows ambient light subtraction 502 performed on pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350, instead of determining the number of photon thresholds per pixel. Ambient light subtraction 502 may include subtracting an intensity value, ambient light measurement result, or binary count associated with ambient image 332 from each of pattern 1 image 338, pattern 2 image 344, and pattern 3 image 350 on a per-pixel basis. Figure 5A In the example shown, the ambient light subtraction operation 502 provides pattern 1 lighting image 504, pattern 2 lighting image 506, and pattern 3 lighting image 508.

[0086] In some cases, by subtracting ambient light from a patterned image to generate a patterned illumination image, one or more common photon thresholds can be used to determine the signature value of all SPAD pixels used to capture the image (e.g., a threshold of 1 photon). For example, Figure 5B The illustration shows the image pixels in pattern 1 illumination image 504, pattern 2 illumination image 506, and pattern 3 illumination image 508 corresponding to SPAD pixel 302. Figure 5B The illumination value 510 (i.e., 9 photons) for SPAD pixel 302 is also illustrated, which in some cases is provided by subtracting an ambient light measurement (e.g., ambient light measurement result 402) from the number of photons associated with the pattern 1 image 338 (e.g., photon number 406). Figure 5B In the example shown, the illumination value 510 for SPAD pixel 302 is 9 photons, and the signature value 512 of pattern 1 portion of SPAD pixel 302 is determined to be the value "1" based on the illumination value 510 satisfying a photon count threshold shared with other SPAD pixels (e.g., a threshold of 1 photon).

[0087] Figure 5B The illustration also shows the illumination value of 514 for SPAD pixel 302. Figure 5B In the example shown, the illumination value 514 for SPAD pixel 302 is 0 photons, and because the illumination value 514 fails to meet the photon counting threshold shared with other SPAD pixels (e.g., the threshold is 1 photon), the signature value 516 for the pattern 2 portion of SPAD pixel 302 is determined to be "0". Furthermore, Figure 5B The illustration shows the illumination value 518 for SPAD pixel 302. Figure 5BIn the example shown, the illumination value 518 for SPAD pixel 302 is 9 photons, and based on the fact that illumination value 518 meets a photon counting threshold shared with other SPAD pixels (e.g., a threshold of 1 photon), the signature value 520 for pattern 3 portion of SPAD pixel 302 is determined to be the value "1". Similar to pixel signature 418 discussed above, pixel signature 522 can be determined based on signature values ​​512, 516, and 520.

[0088] This example focuses on using three light patterns to generate pixel signatures. It will be appreciated that, given this disclosure, any number of light patterns can be used to facilitate unique pixel signatures for pixels along the same scan line.

[0089] In some cases, SPAD pixels can be configured to perform any of the processing described herein on the sensor to provide a sensor output corresponding to the pixel signature, which can facilitate computationally efficient depth processing.

[0090] Furthermore, in some cases, parameters associated with the set of captured binary counts used to facilitate depth processing can be dynamically modified to take into account the amount of ambient light present in the captured scene, the distance between the image sensor and the captured object, and / or other factors. For example, in cases where the captured scene includes high ambient light, the illuminator may need to emit additional light to provide sufficient distinction between scenes illuminated by patterned light and those not illuminated by patterned light. Therefore, in some cases, based on the detected ambient light (e.g., determined by capturing an image of the ambient light), the system can modify the pulse duration for pulsed illuminator activation, the number of pulses associated with emitting one or more light patterns, the amount of light emitted by the illuminator (e.g., light intensity), the frame capture duration, the shutter timing associated with shutter operation (e.g., gate time 308), and / or other parameters.

[0091] As described above, an illuminator (e.g., illuminator 204) can be combined with a SPAD sensor (e.g., SPAD sensor 202) as a “virtual camera” to facilitate structured light depth calculation. Figure 6 Patterns 1, 2, and 3 according to examples of this disclosure are illustrated, which are projected onto a captured environment to generate pixel signatures 418 and / or 522. Figure 6 The diagram also illustrates virtual pixels 602 associated with specific pixel coordinates relative to patterns 1, 2, and 3. The system can determine whether a specific pixel coordinate corresponds to an illuminated or unilluminated portion for each structured light pattern (e.g., pattern 1, pattern 2, and pattern 3).

[0092] therefore, Figure 6Virtual pixels 602 are depicted overlaid on various patterns (i.e., pattern 1, pattern 2, and pattern 3), and arrows extending from virtual pixels 602 to corresponding decision boxes associated with corresponding illumination values ​​(i.e., illumination value 604 for pattern 1, illumination value 606 for pattern 2, and illumination value 608 for pattern 3). The decision boxes associated with illumination values ​​604, 606, and 608 conceptually depict the determination of whether the coordinates of virtual pixel 602 correspond to the illuminated or unilluminated portion of the various patterns. Figure 6 It is evident that the pixel coordinates of virtual pixel 602 correspond to the illuminated portion of pattern 1, and it is in Figure 6 In the example, the lighting value "1" is represented by the lighting value 604 (where "1" and "yes" are represented in bold and underline format). Figure 6 The pixel coordinates of the virtual pixel 602 corresponding to the unlit portion of pattern 2 are also depicted, which are in Figure 6 In the example, it is represented by a lighting value of "0" for a lighting value of 606. Furthermore, Figure 6 The illustration shows the pixel coordinates of the virtual pixel 602 corresponding to the lighting portion of pattern 2, which is located in... Figure 6 In the example, the lighting value is represented by "1" for lighting value 608.

[0093] Figure 6 The illustration shows the virtual pixel signature 610 generated based on the illumination values ​​604-608 discussed above. Figure 6 In the example, the virtual pixel signature 610 is represented by a sequence of values ​​including “1”, “0” and “1”, which matches the sequence of values ​​for the pixel signatures 418 and 522 discussed above for SPAD pixel 302.

[0094] The system can determine the depth for a SPAD pixel by identifying a corresponding virtual pixel (on the same scan line) that has a virtual pixel signature that matches the pixel signature for the SPAD pixel and by using the pixel coordinates of the SPAD pixel and the corresponding virtual value to determine the disparity value (which can be used to calculate the depth). Figure 7 This functionality is conceptually depicted, illustrating the use of pixel coordinates 702 and pixel signature 522 associated with SPAD pixel 302, and pixel coordinates 704 associated with corresponding virtual pixel 602 and matching virtual pixel signature 610, to determine disparity value 706. This functionality can be performed for all SPAD pixels in the SPAD array to determine per-pixel disparity and / or depth values ​​for generating a depth map.

[0095] An exemplary method for structured optical depth calculation using SPAD

[0096] The following discussion now relates to a variety of methods and method actions that can be performed by the disclosed system. Although the method actions are discussed in a certain order and shown in the flowcharts as occurring in a specific order, the specific order is not required unless specifically stated otherwise, or because an action depends on another action completed before performing that action. It will be appreciated that some embodiments of this disclosure may omit one or more of the actions described herein.

[0097] Figure 8 An exemplary flowchart 800 depicts actions associated with structured optical depth calculations using SPAD. Discussion of the various actions represented in flowchart 800 includes references to… Figure 1 and / or Figure 2 More detailed descriptions of various hardware components are provided.

[0098] Action 802 of flowchart 800 includes: selectively activating an illuminator during a frame capture time period to perform interleaved structured illumination operations, said interleaved structured illumination operations including alternately emitting at least a first structured light pattern and at least a second structured light pattern from the illuminator. In some cases, action 802 is performed using one or more components of system 100 (e.g., processor(s) 102, storage device 104, sensor(s) 110, SPAD array(s) 112, I / O system(s) 114, communication system(s) 116), an illuminator (e.g., illuminator 204), and / or other components. In some cases, the first and second structured light patterns include striped structured light patterns. Furthermore, in some cases, the second structured light pattern includes more stripes than the first structured light pattern. Additionally, in some cases, performing interleaved structured illumination operations includes causing the first and second structured light patterns to be emitted from the illuminator at least twice during the frame capture time period. In addition, in some cases, performing multiple sequential shutter operations includes suppressing the pause of shutter operations during the frame capture period to perform readout operations.

[0099] Action 804 of flowchart 800 includes: performing a plurality of sequential shutter operations during a frame capture time period to configure each SPAD pixel of the SPAD array to enable photon detection, the plurality of sequential shutter operations generating a plurality of binary counts for each SPAD pixel of the SPAD array indicating whether a photon was detected during each shutter operation of the plurality of sequential shutter operations. In some cases, action 804 is performed using one or more components of system 100 (e.g., one or more processors 102, storage device 104, one or more sensors 110, one or more SPAD arrays 112, one or more I / O systems 114, one or more communication systems 116) and / or other components.

[0100] Action 806 of flowchart 800 includes: for each SPAD pixel of the SPAD array, determining a corresponding pixel signature based at least on (i) a first set of binary counts generated via the SPAD pixel during illumination by a first structured light pattern within a frame capture time period, and (ii) a second set of binary counts generated via the SPAD pixel during illumination by a second structured light pattern within a frame capture time period. In some cases, action 806 is performed using one or more components of system 100 (e.g., one or more processors 102, storage device 104, one or more sensors 110, one or more SPAD arrays 112, one or more I / O systems 114, one or more communication systems 116) and / or other components. In some cases, the first set of binary counts includes a first plurality of subsets of binary counts generated via the SPAD pixel during illumination by the first structured light pattern within a frame capture time period, and each subset of the first plurality of subsets of binary counts is not generated sequentially with each other in time. In some cases, the second set of binary counts includes a second plurality of subsets of binary counts generated via SPAD pixels during illumination by the second structured light pattern within a frame capture time period, and each subset of the second plurality of subsets of binary counts is not generated sequentially with each other in time. Furthermore, in some cases, at least one subset of the binary counts in the second plurality of subsets of binary counts is temporally located between at least two subsets of the binary counts in the first plurality of subsets of binary counts.

[0101] In some cases, for each SPAD pixel in the SPAD array, the corresponding pixel signature is based at least on a corresponding first signature value and a corresponding second signature value. In some implementations, the corresponding first signature value is determined by subtracting a corresponding ambient light measurement result from the number of photons represented by a first set of binary counts; and determining whether the first illumination value satisfies a photon threshold number. Furthermore, in some cases, the corresponding second value is determined by subtracting a corresponding ambient light measurement result from the number of photons represented by a second set of binary counts; and determining whether the second illumination value satisfies a photon threshold number.

[0102] Furthermore, in some cases, for each SPAD pixel of the SPAD array, the corresponding pixel signature is based at least on a corresponding first signature value and a corresponding second signature value, wherein the corresponding first signature value is based on whether the number of photons represented by a first set of binary counts satisfies a corresponding photon threshold number, and the corresponding second signature value is based on whether the number of photons represented by a second set of binary counts satisfies the photon threshold number. In some implementations, for each SPAD pixel of the SPAD array, the photon threshold number is determined based on a corresponding ambient light measurement result. The corresponding ambient light measurement result is determined based on an ambient light image frame captured using the SPAD array while suppressing the emission of structured light patterns from the illuminator. In some cases, the ambient light image frame is generated based on photons detected by the SPAD pixels of the SPAD array during the frame capture time period while suppressing the emission of structured light patterns from the illuminator.

[0103] Action 808 of flowchart 800 includes: determining a plurality of virtual pixel signatures using at least a first structured light pattern and a second structured light pattern. In some cases, action 808 is performed using one or more components of system 100 (e.g., one or more processors 102, storage device 104, one or more sensors 110, one or more SPAD arrays 112, one or more I / O systems 114, one or more communication systems 116) and / or other components. In some cases, each virtual pixel signature in the plurality of virtual pixel signatures is associated with a corresponding pixel coordinate relative to the first structured light pattern and the second structured light pattern. Each particular virtual pixel signature in the plurality of virtual pixel signatures is associated with (i) whether the corresponding pixel coordinate for the particular virtual pixel signature is associated with an illuminated or unilluminated portion of the first structured light pattern, and (ii) whether the corresponding pixel coordinate for the particular virtual pixel signature is associated with an illuminated or unilluminated portion of the second structured light pattern.

[0104] Action 810 of flowchart 800 includes generating a depth map by identifying (i) the corresponding pixel signatures of the SPAD pixels of the SPAD array and (ii) the correspondence between multiple virtual pixel signatures. In some cases, action 810 is performed using one or more components of system 100 (e.g., one or more processors 102, storage device 104, one or more sensors 110, one or more SPAD arrays 112, one or more I / O systems 114, one or more communication systems 116) and / or other components.

[0105] Action 812 of flowchart 800 includes dynamically modifying, based on ambient light image frames captured using a SPAD array while suppressing the emission of structured light patterns from an illuminator: (i) the pulse duration associated with the emission of a first or second structured light pattern, (ii) the number of pulses associated with the emission of the first or second structured light pattern within the frame capture duration, (iii) the amount of emitted light associated with the emission of the first or second structured light pattern, (iv) the frame capture duration, or (v) the shutter timing associated with performing multiple sequential shutter operations. In some cases, action 814 is performed using one or more components of system 100 (e.g., one or more processors 102, storage device 104, one or more sensors 110, one or more SPAD arrays 112, one or more I / O systems 114, one or more communication systems 116) and / or other components.

[0106] The disclosed embodiments may include or utilize dedicated or general-purpose computers including computer hardware, as discussed in more detail below. The disclosed embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or dedicated computer system. A computer-readable medium storing computer-executable instructions in data form is one or more “physical computer storage media” or “hardware storage devices.” A computer-readable medium that carries only computer-executable instructions without storing them is a “transmission medium.” Therefore, by way of example and not limitation, the present embodiments may include at least two distinct types of computer-readable media: computer storage media and transmission media.

[0107] Computer storage media (also known as “hardware storage devices”) are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, RAM-based solid-state drives (“SSDs”), flash memory, phase-change memory (“PCM”), or other types of memory, or other optical disc storage devices, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in hardware in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0108] A “network” is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or other communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer correctly regards the connection as a transmission medium. Transmission media may include networks and / or data links, which may be used to carry program code in the form of computer-executable instructions or data structures and may be accessible by general-purpose or special-purpose computers. Combinations of the above are also included within the scope of computer-readable media.

[0109] Furthermore, upon arrival at various computer system components, program code units in the form of computer-executable instructions or data structures can be automatically transferred from the transmission computer-readable medium to the physical computer-readable storage medium (or vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or a less lossy computer-readable physical storage medium at the computer system. Therefore, computer-readable physical storage media can be contained within computer system components that also (or even primarily) utilize the transmission medium.

[0110] Computer-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. Computer-executable instructions can be, for example, binary, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0111] The disclosed embodiments may include or utilize cloud computing. The cloud model may consist of various features (e.g., on-demand self-service, broad network access, resource pooling, rapid elasticity, measurable services, etc.), service models (e.g., Software as a Service (“SaaS”), Platform as a Service (“PaaS”), Infrastructure as a Service (“IaaS”)), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).

[0112] Those skilled in the art will understand that this invention can be practiced in networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, pagers, routers, switches, wearable devices, etc. This invention can also be practiced in distributed system environments, where multiple computer systems (e.g., local and remote systems) linked via a network (or via hardwired data links, wireless data links, or a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules can reside in local and / or remote memory storage devices.

[0113] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), central processing units (CPUs), graphics processing units (GPUs), and / or others.

[0114] As used herein, the terms “executable module,” “executable component,” “component,” “module,” or “engine” can refer to a hardware processing unit or a software object, routine, or method that can be executed on one or more computer systems. The various components, modules, engines, and services described herein can be implemented as objects or processors that execute on one or more computer systems (e.g., as separate threads).

[0115] Those who wish to understand will also understand how any feature or operation disclosed herein can be combined with any or a combination of other features and operations disclosed herein. Furthermore, the content or feature in any of the figures can be combined or used in conjunction with any content or feature used in any other figure. In this regard, the content disclosed in any of the figures is not mutually exclusive, but can be combined with content from any other figure.

[0116] The invention may be embodied in other specific forms without departing from the spirit or characteristics thereof. The described embodiments should be considered illustrative in all respects and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All modifications falling within the meaning and scope of equivalents of the claims are included within their scope.

Claims

1. A system for structured optical depth calculation using a single-photon avalanche diode (SPAD), the system comprising: A SPAD array, which includes multiple SPAD pixels; An illuminator configured to emit one or more structured light patterns; One or more processors; as well as One or more hardware storage devices storing instructions executable by the one or more processors to configure the system to facilitate structured optical depth computation using SPAD by configuring the system to perform the following operations: During a frame capture period, the illuminator is selectively activated to perform interleaved structured illumination operations, the interleaved structured illumination operations comprising alternately emitting at least a first structured light pattern and at least a second structured light pattern from the illuminator, wherein the first structured light pattern is different from the second structured light pattern, and wherein performing the interleaved structured illumination operations comprises: emitting the first structured light pattern and the second structured light pattern from the illuminator at least twice during the frame capture period and within the same frame capture period; and During the frame capture time period, multiple sequential shutter operations are performed to configure the SPAD array to enable photon detection. The multiple sequential shutter operations generate multiple binary counts for each SPAD pixel of the SPAD array, indicating whether a photon was detected during each of the multiple sequential shutter operations.

2. The system according to claim 1, wherein, Performing the plurality of sequential shutter operations includes suppressing the pause execution of the shutter operations during the frame capture time period to perform a readout operation.

3. The system according to claim 1, wherein, The instructions can be executed by the one or more processors to configure the system to: for each SPAD pixel of the SPAD array, determine a corresponding pixel signature based at least on (i) a first set of binary counts generated via the SPAD pixel during illumination of the first structured light pattern within the frame capture time period and (ii) a second set of binary counts generated via the SPAD pixel during illumination of the second structured light pattern within the frame capture time period.

4. The system according to claim 3, wherein, For each SPAD pixel in the SPAD array: The first set of binary counts includes a first plurality of subsets of binary counts generated via the SPAD pixels during illumination by the first structured light pattern within the frame capture time period. Each subset in the first plurality of subsets of the binary count is not generated sequentially with each other in time. The second set of binary counts includes a second subset of binary counts generated via the SPAD pixels during illumination by the second structured light pattern within the frame capture time period, and Each subset in the second plurality of subsets of the binary count is not generated sequentially from one another in time.

5. The system according to claim 4, wherein, At least one subset of the binary counts in the second plurality of subsets of the binary counts is temporally located between at least two subsets of the binary counts in the first plurality of subsets of the binary counts.

6. The system according to claim 3, wherein, For each SPAD pixel in the SPAD array: The corresponding pixel signature is based at least on the corresponding first signature value and the corresponding second signature value. The corresponding first signature value is determined by the following operations: A first illumination value is determined by subtracting the corresponding ambient light measurement from the number of photons represented by the first set of binary counts; and Determine whether the first illumination value meets the photon threshold quantity, and The corresponding second value is determined by the following steps: The second illumination value is determined by subtracting the corresponding ambient light measurement from the number of photons represented by the second set of binary counts; and Determine whether the second illumination value meets the photon threshold number.

7. The system according to claim 3, wherein, For each SPAD pixel in the SPAD array: The corresponding pixel signature is based at least on the corresponding first signature value and the corresponding second signature value. The corresponding first signature value is based on whether the number of photons represented by the first set of binary counts meets the corresponding photon threshold number, and The corresponding second signature value is based on whether the number of photons represented by the second set of binary counts satisfies the corresponding photon threshold number.

8. The system according to claim 7, wherein, For each SPAD pixel in the SPAD array, the number of photon thresholds is determined based on a corresponding ambient light measurement, which is based on an ambient light image frame captured using the SPAD array while suppressing the emission of structured light patterns from the illuminator.

9. The system according to claim 8, wherein, The ambient light image frame is generated based on photons detected by the SPAD pixels of the SPAD array during the frame capture time period while suppressing the emission of structured light patterns from the illuminator.

10. The system according to claim 3, wherein, The instructions can be executed by the one or more processors to further configure the system to: determine a plurality of virtual pixel signatures using at least the first structured light pattern and the second structured light pattern, each of the plurality of virtual pixel signatures being associated with a corresponding pixel coordinate relative to the first structured light pattern and the second structured light pattern, each particular virtual pixel signature being based on: (i) whether the corresponding pixel coordinate for the particular virtual pixel signature is associated with an illuminated or unilluminated portion of the first structured light pattern, and (ii) whether the corresponding pixel coordinate for the particular virtual pixel signature is associated with an illuminated or unilluminated portion of the second structured light pattern.

11. The system according to claim 10, wherein, The instructions can be executed by the one or more processors to further configure the system to generate a depth map by identifying the correspondence between the corresponding pixel signatures of the SPAD pixels in the SPAD array and the plurality of virtual pixel signatures.

12. The system according to claim 1, wherein, The instructions can be executed by the one or more processors to configure the system to dynamically modify the pulse time period associated with the emission of the first or second structured light pattern based on ambient light image frames captured by the SPAD array while suppressing the emission of structured light patterns from the illuminator.

13. The system according to claim 1, wherein, The instructions can be executed by the one or more processors to configure the system to dynamically modify the number of pulses associated with emitting the first or second structured light pattern during the frame capture time period, based on ambient light image frames captured using the SPAD array while suppressing the emission of structured light patterns from the illuminator.

14. The system according to claim 1, wherein, The instructions can be executed by the one or more processors to configure the system to dynamically modify the amount of emitted light associated with the emission of the first structured light pattern or the second structured light pattern, based on ambient light image frames captured by the SPAD array while suppressing the emission of structured light patterns from the illuminator.

Images