Time-of-flight rangefinder for structured light systems

By combining structured light and Time-of-Flight (TOF) systems and using a TOF receiver to sense structured light reflection pulses, the limitations of the range and accuracy of structured light systems and the low resolution of TOF systems are solved, enabling greater range and higher accuracy depth measurement while reducing the impact of component movement and resource consumption.

CN112204420BActive Publication Date: 2024-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980035494.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-05-30
Filing Date
2019-05-08
Publication Date
2024-11-19
Estimated Expiration
2039-05-08

AI Technical Summary

Technical Problem

Structured light systems have limited effective range and accuracy, while Time-of-Flight (TOF) systems have low resolution over long distances and are susceptible to component movement. Combining the two results in excessive power and space consumption.

Method used

By combining structured light systems and time-of-flight (TOF) systems, the reflected pulses from the structured light emitter are sensed by the TOF receiver, the depth is determined using triangulation and round-trip time, and the ranging mode is switched as needed to optimize performance.

Benefits of technology

It expands the effective range of depth measurement, improves measurement accuracy over long distances, reduces sensitivity to component movement, and optimizes power and space utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112204420B_ABST
    Figure CN112204420B_ABST
Patent Text Reader

Abstract

Aspects of the present disclosure relate to systems and methods for determining one or more depths. An example system includes a time-of-flight receiver configured to sense pulses of reflections of light from a structured light emitter. An example method includes sensing, by the time-of-flight receiver, pulses of reflections of light from a structured light emitter.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Non-Provisional Application No. 15 / 992,746, filed on May 30, 2018, entitled “TIME OF FLIGHT RANGE FINDER FOR A STRUCTURED LIGHT SYSTEM,” assigned to the assignee of the present invention, and is expressly incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to systems and methods for depth detection, and particularly to a time-of-flight rangefinder for a structured light system. Background Art

[0004] The device can use different depth detection systems to determine the distance of objects around it. When determining the depth, the device can generate a depth map by sending (emit) one or more wireless signals and measuring the reflection of the wireless signals from the scene, and the depth map shows or otherwise indicates the depth of the object from the device. Two types of depth detection systems include time-of-flight (TOF) systems and structured light systems.

[0005] For a TOF system, pulses of light are sent and reflections of the pulses are received. The round trip time of the light from the transmitter to the receiver is determined by the pulses, and the distance or depth of the object that sent the light from the TOF system is determined by the reflections.

[0006] For structured light systems, a known spatial pattern of light is transmitted and reflections of the spatial pattern are received. The transmitter and receiver are separated by a distance and as a result, a displacement and distortion of the spatial pattern occurs at the receiver. Triangulation using the displacement and distortion of the spatial pattern and the distance between the transmitter and receiver is used to determine the distance or depth of the object that sent the light from the reflection from the structured light system.

[0007] The patterning intensity of a structured light system may be limited, thus limiting the range of depth detection. In addition, the distance between the transmitter and receiver used for triangulation corresponds to the accuracy of the depth measurement. The smaller the distance, the less accurate the depth measurement of the object, and the greater the impact on accuracy as the depth of the object increases from the structured light system. Summary of the invention

[0008] This Summary is provided to introduce selected concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0009] Aspects of the present disclosure relate to systems and methods for determining one or more depths.An example system includes a time-of-flight receiver configured to sense pulses of reflections of light from a structured light emitter.

[0010] In another example, a method for determining one or more depths is disclosed. The method includes sensing, by a time-of-flight receiver, a pulse of reflection of light from a structured light emitter.

[0011] In yet another example, a non-transitory computer-readable medium storing one or more programs containing instructions is disclosed. Execution of the instructions by one or more processors of a device causes the device to sense, via a time-of-flight receiver, a pulse of reflection of light from a structured light emitter.

[0012] In yet another example, a device for determining one or more depths is disclosed. The device includes a device for emitting pulsed light of a spatial pattern, a device for sensing reflections of the pulsed light of the spatial pattern, and a device for sensing pulses from the reflections of the pulsed light. The device for sensing the pulses is separate from the device for sensing the reflections of the spatial pattern of light. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Various aspects of the disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

[0014] Figure 1 is a schematic diagram of a structured light system.

[0015] Figure 2 is a schematic diagram of a TOF system.

[0016] Figure 3 is a block diagram of an example device including a structured light system.

[0017] Figure 4 is a schematic diagram of spatially patterned light sensed by a structured light sensor.

[0018] Figure 5 is a schematic diagram of a pulse of at least a portion of spatially patterned light sensed by a TOF receiver.

[0019] Figure 6 is a flow chart describing example operations for TOF ranging for structured light emission.

[0020] Figure 7 is a flow chart describing another example operation for TOF ranging for structured light emission.

[0021] Figure 8is a flow chart describing example operations for calibrating a structured light receiver based on TOF measurements. DETAILED DESCRIPTION

[0022] Aspects of the present disclosure can be used in depth detection systems. In some aspects, a TOF sensor can receive a portion of a structured light emission to determine the depth of an object that reflects the portion of the structured light emission. In this way, ranging using TOF information can be performed outside the effective range of the structured light system.

[0023] Figure 1 is a schematic diagram of a structured light system 100. The structured light system 100 may be used to generate a depth map (not shown) of a scene (having objects 106A and 106B at different depths in the scene), or may be used for other applications of ranging of objects 106A and 106B or other parts of the scene. The structured light system 100 may include a transmitter 102 and a receiver 108.

[0024] The emitter 102 can be configured to project the spatial pattern 104 onto a scene (including objects 106A and 106B). The emitter 102 can include one or more light sources 124 (such as laser sources), a lens 126, and a light modulator 128. In some embodiments, the light modulator 128 includes one or more diffractive optical elements (DoE) to diffract the transmission from the one or more light sources 124 (which can be directed to the light modulator 128 by the lens 126) into additional transmissions. The light modulator 128 can also adjust the intensity of the transmission. Additionally or alternatively, the light source 124 can be configured to adjust the intensity of the transmission.

[0025] In some other embodiments of the transmitter 102, the DOE can be directly coupled to the light source (without the lens 126) and configured to diffuse the light transmitted from the light source into at least a portion of the spatial pattern 104. The spatial pattern 104 can be a fixed pattern of transmitted light that the transmitter projects onto the scene. For example, the DOE can be manufactured so that the black spots in the spatial pattern 104 correspond to locations in the DOE that prevent light from the light source 124 from being transmitted by the transmitter 102. In this way, the spatial pattern 104 can be known when analyzing any reflections received by the receiver 108. The transmitter 102 can transmit the spatial pattern of light onto the scene (including objects 106A and 106B) through the aperture 122 of the transmitter 102.

[0026] Receiver 108 may include an aperture 120 through which reflections of the transmitted light may pass, be directed by lens 130, and strike sensor 110. Sensor 110 may be configured to detect (or "sense") one or more reflections of spatially patterned light from a scene. As shown, transmitter 102 may be located on the same reference plane as receiver 108, and transmitter 102 and receiver 108 may be separated by a distance 112 referred to as a "baseline."

[0027] The sensor 110 may include an array of photodiodes (such as avalanche photodiodes) to measure or sense reflections. The array may be coupled to a complementary metal oxide semiconductor (CMOS) sensor including a plurality of pixels or regions corresponding to the number of photodiodes in the array. A plurality of electrical pulses generated by the array may trigger corresponding pixels or regions of the CMOS sensor to provide a measurement of the reflection sensed by the array. Alternatively, the sensor 110 may be a photosensitive CMOS sensor to sense or measure reflections including a reflection codeword pattern. The CMOS sensor may be logically divided into groups of pixels corresponding to the bit size or codeword (patch of bit) size of the spatial pattern 104.

[0028] The reflections may include multiple reflections of spatially patterned light from different objects or portions of the scene at different depths, such as objects 106A and 106B. Based on the baseline 112, the displacement and distortion of the light of the sensed spatial pattern 104, and the intensity of the reflections, the structured light system 100 may be used to determine one or more depths and positions of objects, such as objects 106A and 106B, from the structured light system 100. Using triangulation based on the baseline and the distance, the structured light system 100 may be used to determine different distances between objects 106A and 106B. For example, a first distance between the center 114 and a location 116 where light reflected from object 106B strikes the sensor 110 is less than a second distance between the center 114 and a location 118 where light reflected from object 106A strikes the sensor 110. The distances from the center to the location 116 and the location 118 of the sensor 110 may indicate the depths of objects 106A and 106B, respectively. The first distance being less than the second distance may indicate that object 106B is farther from the emitter 102 than object 106A. In addition to determining the distance from the center of the sensor 110, the calculation may also include determining the displacement or distortion of the spatial pattern 104 in the light striking the sensor 110 to determine the depth or distance.

[0029] A structured light system included in a power-constrained device (such as a smartphone, tablet, or other battery-powered device) has a limit on the emission intensity of the spatially patterned light (which corresponds to the power supplied to the light source used to emit the light). In addition, eye safety issues may prompt structured light system manufacturers or device manufacturers to limit the light intensity of the emitter. As a result, the effective range of the structured light system may be limited. For example, a structured light system of a smartphone may have a maximum effective range of 4 meters, and the depth of an object beyond 4 meters from the smartphone cannot be consistently determined.

[0030] For a structured light system, the total power used to emit spatially patterned light is for multiple light spots in the spatial pattern. If the total power is dedicated to one light source, the light intensity from the one light source will be greater than the intensity of each light spot in the spatial pattern. Assuming that the field of light from one light source and the field of light from each light spot in the spatial pattern are the same, the light from the one light source can propagate farther than each light spot in the spatial pattern. In this way, TOF ranging using one light source can have a larger effective range than determining depth using a structured light system. In addition, since the structured light system recognizes codewords in the reflection, the accuracy of the structured light system can decrease as the depth of the object increases until the structured light system can no longer reliably measure the depth (which can be considered the upper limit of the effective range). In contrast, as long as the TOF system can recognize reflections, the measurement accuracy of the TOF system will not decrease with increasing distance.

[0031] Another advantage of the TOF system is that the system is not very sensitive to component movement during operation. For example, when a user holds a smartphone (such as by a frame), pressure may occur by the user's hand squeezing the smartphone to maintain a grip on the smartphone. The pressure of squeezing the smartphone to maintain a grip may cause the smartphone to bend slightly, and the internal components of the smartphone to move slightly. As a result, the light source of the transmitter may move, thereby distorting the spatial pattern. The distorted spatial pattern may have a negative impact on depth measurement. In contrast, a TOF system that measures round-trip time is not affected by such pressure or force. For example, a TOF system in a smartphone is not affected by the user maintaining a grip on the smartphone by applying pressure that may slightly bend the smartphone.

[0032] Figure 22 is a schematic diagram of a TOF system 200. The TOF system 200 may be used to generate a depth map (not shown) of a scene (including an object 206 in the scene shown as reflecting light from a transmitter 202), or may be used for other applications of ranging. The TOF system 200 may include a transmitter 202 and a receiver 208. The transmitter 202 may be referred to as a "transmitter", "projector", "emitter", etc., and should not be limited to a specific transmitting component. Similarly, the receiver 208 may be referred to as a "detector", "sensor", "sensing element", "photodetector", etc., and should not be limited to a specific receiving component.

[0033] The emitter 202 may be configured to emit, transmit, or project a signal (such as a light field) onto a scene (including emitting, transmitting, or projecting onto an object 206). The emitter 202 emits light 204 toward the scene. Although the emitted light 204 is only shown as being directed toward the object 206, the transmission or emission field of the emitter 202 may extend beyond the object 206 (such as toward the entire scene including the object 206). For example, a conventional TOF system emitter has a fixed focal length lens for transmission, which limits the transmission field to propagate away from the emitter.

[0034] The transmitted light 204 includes light pulses 214 at known time intervals (such as periodically). The receiver 208 includes a sensor 210 to sense reflections 212 of the transmitted light 204. The reflections 212 include reflected light pulses 216, and the TOF system determines the round trip time 222 of the light by comparing the timing 218 of the transmitted light pulses and the timing 220 of the reflected light pulses. The distance of the object 206 from the TOF system can be calculated as half of the round trip time multiplied by the transmission speed (such as the speed of light at which the light is transmitted).

[0035] The sensor 210 may include a photodiode array to measure or sense reflections. Alternatively, the sensor 210 may include a CMOS sensor or other suitable light-sensitive sensor including a plurality of pixels or regions for sensing. The TOF system 200 identifies a reflected light pulse 216 sensed by the sensor 210 when the pulse amplitude is greater than a threshold. For example, the TOF system measures the amplitude of ambient light and other interference in the absence of a signal, and then determines whether a further measurement is greater than a threshold value than a previous measurement. The upper limit of the effective range of the TOF system may be the distance at which the noise or degradation of the signal causes the signal-to-noise ratio (SNR) to be too large before the reflection is sensed, so that the sensor cannot accurately sense the reflected light pulse 216. To reduce interference, the receiver 208 may include a bandpass filter before the sensor 210 to filter out some incident light of different wavelengths from the emitted light 204.

[0036] The resolution (number of distances that can be determined simultaneously) of a TOF system may be lower than that of a structured light system. In the case where objects in the scene are at different depths, reflections of the emitted light can be received at different points in time. Structured light systems do not rely on round trip times and are therefore not affected by receiving reflections at multiple points in time. For a TOF system, reflections at different points in time may cause interference in attempts to determine round trip times. To reduce interference, many TOF systems can focus a pulsed beam and use a receiver to determine a round trip time of the pulsed light. To compensate for the use of one beam, the speed of the TOF system can be increased to determine the round trip time of different points in the sequence more quickly, but there may be limitations in the pulse spacing of the emitted light. As a result, although TOF systems may be effective at longer distances than structured light systems and are not affected by component movement (which may cause distortions in the spatial pattern of structured light systems), many TOF systems may have lower resolution than structured light systems.

[0037] Since a TOF system may have a lower resolution than a structured light system but a higher effective range, a TOF system may be used to measure the distance to an object that is farther away than the system can determine using a structured light system or to determine the depth of the object. For example, a TOF system may be used for ranging applications. Conversely, a structured light system may be used for object recognition or applications that require more depth points than a TOF system can provide. For example, a structured light system may be used for facial recognition and identification. In addition, if two systems exist for a device, a TOF system may be used to calibrate the structured light system or to determine if the depth determined using the structured light system is incorrect.

[0038] The device may include a TOF system and a structured light system. However, two transmitters and other components may be required. As a result, the two systems may consume more power and space than the preferred system in the device. Alternatively, the structured light receiver sensor can be configured to sense the pulse to determine the round trip time. However, since the receiver is separated from the transmitter by a baseline distance, the calculation of the depth based on the measured round trip time is complicated by the triangulation calculation. In addition, the structured light sensor will need to be configured to recognize the pulse and recognize the codeword of the spatial pattern in the reflection, which may cause the design and manufacture of the structured light receiver to be very complicated.

[0039] For example, a structured light system may operate in the near-infrared (NIR) range, and the wavelength of the light from the structured light emitter may be 825 nanometers (nm) or 940 nm (although other frequency ranges and wavelengths may be used). A conventional structured light receiver sensor (which may be similar to a camera sensor) may have a frame rate of 30 frames per second (fps), and an exposure length of approximately 33 ms to capture a spatial pattern of light intensity. With a speed of light of approximately 300 million meters per second, a wavelength of 825 nm, and an exposure window of 33 ms, a large number of light cycles occur during the exposure window of the structured light sensor. Conventional structured light sensors with conventional frame rates cannot distinguish between cycles of emitted light.

[0040] If the emitted light is pulsed (in addition to the natural waveform of the light signal) so that a conventional structured light sensor can sense the pulses, the exposure length of the structured light sensor (such as 33ms) requires a pulse rate that is slower than the exposure length. As a result, the number of round trip times that can be determined in a period of time is limited by the sensor exposure length. A TOF sensor (which may include an avalanche photodiode) can have a shorter exposure length than a structured light sensor. For example, the exposure length of an avalanche photodiode can be 10 picoseconds (compared to 33ms for the example structured light system). Therefore, the pulses used for the TOF sensor can be closer than the pulses of the structured light sensor used to determine the round trip time. If the pulses are slowed down so that the structured light sensor can be used to determine the round trip time, the number of round trip times that can be determined using the example structured light system with an exposure length of 33ms is a small fraction of the number of round trip times that can be determined using the example TOF system with an exposure length of 10 picoseconds (such as the theoretical limit of the TOF system is up to 3 billion pulses for every 1 pulse of the structured light system).

[0041] In some example embodiments, a TOF sensor separate from a structured light sensor may be used to identify pulses of emitted light in at least a portion of a spatial pattern. The structured light receiver may be a conventional structured light receiver. Conventional spatial patterns may also be used, so there is no need to change the structured light emitter. In this way, the additional component from a conventional structured light system may be only a TOF sensor. In some example embodiments, the TOF sensor may be a pulsed light sensor. For example, the sensor may be a photodiode (such as an avalanche photodiode), and the photodiode may be juxtaposed with the structured light emitter (similar to the position of the receiver relative to the emitter in a conventional TOF system). Compared to conventional structured light sensors (such as those with a frame rate of 30fps), avalanche photodiodes may have a high time of arrival resolution. For example, the time of arrival resolution of a photon captured by an avalanche photodiode may be less than 10 picoseconds. Although the TOF sensor is described as a photodiode in the following examples, any number of photodiodes (such as a photodiode array or row) may be used. Additionally or alternatively, the TOF sensor may be other configurations (such as a continuous wave sensor), and the present disclosure should not be limited to the examples provided.

[0042] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a comprehensive understanding of the present disclosure. The term "coupled" used herein means directly connected to or connected through one or more intermediate components or circuits. And, in the following description, for the purpose of explanation, specific terms are set forth to provide a comprehensive understanding of the present disclosure. However, it is obvious to those skilled in the art that these specific details may not be needed to practice the teachings disclosed herein. In other cases, well-known circuits, systems and devices are shown in block diagram form to avoid blurring the teachings of the present disclosure. Some parts of the following detailed description are presented in terms of processes, logic blocks, processing and other symbolic representations of operations on data bits in computer memory. In the present disclosure, processes, logic blocks, processes, etc. are considered to be self-consistent sequences of steps or instructions that lead to desired results. These steps require physical operations on physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared and otherwise manipulated in a computer system.

[0043] It should be remembered, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, it will be apparent from the following discussion that throughout this application, discussions using terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "establish," etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the computer system's registers and memories into other data similarly represented as physical quantities in the computer system's memories or registers or other such information storage, transmission, or display devices.

[0044] In the accompanying drawings, a single block may be described as performing one or more functions; however, in actual practice, one or more functions performed by the block may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described below according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present disclosure. In addition, the example systems and devices may include components other than the components shown, including well-known components such as processors, memories, etc.

[0045] Aspects of the present disclosure are applicable to TOF ranging of structured light systems and may be included in or coupled to any suitable electronic device or system (such as a security system, a smartphone, a tablet, a laptop, a digital camera, a vehicle, a drone, a virtual reality device, or other device that can utilize depth sensing). Although described below with respect to a device having or coupled to one structured light system, aspects of the present disclosure may be applicable to devices having any number of structured light systems.

[0046] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. In addition, the term "system" is not limited to multiple components or specific embodiments. For example, a system can be implemented on one or more printed circuit boards or other substrates, have one or more housings, be one or more objects integrated into another device, and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0047] Figure 3 is a block diagram of an example device 300 including a structured light system. In some other examples, the structured light system can be coupled to the device 300. The example device 300 can include or be coupled to a transmitter 301 (such as Figure 1 ), a receiver 302 (such as a transmitter 102 in FIG. 1 ) separated from the transmitter by a baseline 303 Figure 1 108), a TOF receiver 305, a processor 304, a memory 306 storing instructions 308, and a controller 310 (which may include one or more signal processors 312). The device 300 may optionally include (or be coupled to) a display 314 and multiple input / output (I / O) components 316. The device 300 may include additional features or components not shown. For example, the wireless communication device may include a wireless interface, which may include multiple transceivers and a baseband processor. The device 300 may also include a power supply 318, which may be coupled to or integrated into the device 300.

[0048] Transmitter 301 and receiver 302 may be structured light systems (such as signal processor 312) controlled by controller 310 and / or processor 304. Figure 1 The device 300 may include or be coupled to an additional structured light system or a different configuration of a structured light system. The present disclosure should not be limited to any particular example or illustration, including the example device 300.

[0049] Memory 306 may be a non-transitory or non-temporary computer-readable medium storing computer-executable instructions 308 to perform all or a portion of one or more operations described in the present disclosure. Processor 304 may be one or more suitable processors capable of executing scripts or instructions (such as instructions 308) of one or more software programs stored in memory 306. In some aspects, processor 304 may be one or more general-purpose processors that execute instructions 308 to cause device 300 to perform any number of functions or operations. In additional or alternative aspects, processor 304 may include integrated circuits or other hardware to perform functions or operations without the use of software. Although in Figure 3 304, but the processor 304, memory 306, controller 310, optional display 314, and optional I / O components 316 may be coupled to each other in various arrangements. For example, the processor 304, memory 306, controller 310, optional display 314, and optional I / O components 316 may be coupled to each other via one or more local buses (not shown for simplicity).

[0050] The display 314 may be any suitable display or screen that allows user interaction and / or presents items for the user to view, such as a depth map or a preview image of a scene. In some aspects, the display 314 may be a touch-sensitive display. The I / O component 316 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from a user and provide output to the user. For example, the I / O component 316 may include, but is not limited to, a graphical user interface, a keyboard, a mouse, a microphone and a speaker, a squeezable border or border of the device 300, a physical button located on the device 300, and the like. The display 314 and / or the I / O component 316 may provide a preview image or depth map of a scene to the user and / or receive user input for adjusting one or more settings of the device 300, such as controlling when to use a structured light system or when to perform TOF ranging, controlling the pulses of spatially patterned light, and the like.

[0051] The controller 310 may include a signal processor 312, which may be one or more processors, to process measurements provided by the receiver 302 and the TOF receiver 305, and / or to control the transmitter 301. In some aspects, the signal processor 312 may execute instructions from a memory (such as instructions 308 from the memory 306 or instructions stored in a separate memory coupled to the signal processor 312). In other aspects, the signal processor 312 may include specific hardware for operation. The signal processor 312 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

[0052] The transmitter 301 may transmit spatially patterned light, at least a portion of which includes pulses for TOF ranging. In some example embodiments, the transmitter 301 may include or be coupled to one or more power supplies for adjusting power to induce pulses in the spatially patterned light. For example, the transmitter 301 may include multiple lasers coupled to multiple DOEs for emitting spatially patterned light. The lasers may be coupled to one or more power supplies (e.g., each laser may be coupled to a separate power supply). The one or more power supplies may increase and decrease the power of one or more lasers so that at least a portion of the spatially patterned light includes pulses for TOF ranging. In some example embodiments, the one or more power supplies are configured to change the power of one or more light sources (such as lasers) in the TOF ranging mode and maintain a constant power of one or more light sources (such as lasers) in the structured light depth sensing mode. In some other example embodiments, the one or more power supplies are configured to always change the power of the one or more light sources.

[0053] The TOF receiver 305 may be collocated with the transmitter 301 and configured to sense reflections of at least a portion of the spatially patterned light from the transmitter 301. The reflections may then be used to identify pulses and determine the round trip time of the spatially patterned light (such as the controller 310 or the signal processor 312). In some example embodiments, the TOF receiver 305 is or includes a photodiode (such as an avalanche photodiode). In some other example embodiments, the TOF receiver 305 is a photodiode array. The field of view of the photodiode or photodiode array may be narrowed (such as by an aperture or other funnel) so that the TOF receiver 305 is configured to sense pulses of a specific portion of the spatially patterned light. In some example embodiments, the TOF receiver 305 may include one or more lenses to focus light from a specific portion of the spatially patterned light onto the TOF sensor (such as onto a photodiode or a photodiode array). If the TOF receiver 305 includes a photodiode, a depth measurement of a point of a scene may be performed using TOF ranging.

[0054] In some example embodiments, the portion of the spatially patterned light corresponding to the TOF receiver 305 may be light from a zero-order portion of the spatial pattern. For example, one or more DOEs may diffuse light from one or more light sources to produce a spatial pattern. The zero-order portion of the spatial pattern may be a portion of the light from the light source that the DOE does not diffuse. In one example, the zero-order portion may be a point of light from a DOE where the light is not bent from a source laser. In another example, the zero-order portion may be a gap between DOEs (e.g., present in conventional defects in manufacturing structured light emitters) where light from a laser source escapes.

[0055] Figure 4 Schematic diagram of spatial patterned light 400 sensed by a structured light sensor. As shown, point 402 of spatial pattern 400 may have a higher intensity than the rest of spatial pattern 400. In some example embodiments, TOF receiver 305 may be configured to sense light pulses at point 402 of spatial pattern 400. The portion of the spatial pattern including point 402 may have light pulsed for TOF ranging. Alternatively, the light of the entire spatial pattern may be pulsed. In some example embodiments, the TOF receiver lens may focus reflected light from the zero-order portion of the spatial pattern (which may be point 402) onto one or more photodiodes, and the one or more photodiodes may sense light pulses from the zero-order portion of the spatial pattern.

[0056] The pulses can be frequent enough not to affect the operation of the structured light receiver. For example, if the exposure length of the structured light sensor is 33 ms, multiple light pulses (such as tens or hundreds of light pulses) may occur during the exposure length. Each pixel of the structured light sensor measures the light intensity accumulated over the exposure window and is therefore not affected by the pulses.

[0057] Figure 5 is a schematic diagram of a pulse of at least a portion of spatially patterned light sensed by TOF receiver 305. The reflected intensity of emitted light 502 may be determined by the exposure length 504 during a frame corresponding to a frame start 506 and a frame end 508. Figure 3 The structured light receiver 302 senses. In some examples, the intensity of the light can vary from 90% to 100% of the intensity of the light used for the pulse. Any suitable intensity range for the pulse can be used, and the present disclosure should not be limited to a specific range or level of light intensity for the pulse.

[0058] Example portion 510 of emitted light 502 depicts multiple pulses occurring during an exposure window of a frame of a structured light receiver. Reflections 512 of emitted light depict the reflections of the structured light receiver. Figure 3 The phase difference 514 between the emitted light 502 (shown as portion 510) and the reflected light 512 corresponds to the pulse from Figure 3 The transmitter 301 and Figure 3 The round trip time of the TOF receiver 305. In some example embodiments, Figure 3 The device 300 (such as the controller 310, the signal processor 312 or the processor 304) determines the phase difference 514 and determines the round trip time based on the phase difference. The depth of the object from the device 300 and corresponding to the reflection 512 can be about half of the round trip time multiplied by the speed of the transmitted signal (such as the speed of light of the transmitted light).

[0059] Figure 6 600 is a flow chart describing example operations of TOF ranging of structured light emission. Although the following example operations (including example operations 600) are described as being performed by Figure 3 The example operations are performed by the example device 300 in the present invention, but other devices or systems may also be used to perform the example operations, and the present disclosure should not be limited to the examples provided.

[0060] Starting from 602, Figure 3 The structured light emitter 301 may emit pulsed light in a spatial pattern. Alternatively, the pulsed light in a spatial pattern may be emitted by an emitter separate from the system performing TOF ranging. Some embodiments of the present disclosure do not require a structured light emitter. Figure 3 The TOF transmitter 305 can then sense pulses of reflections of the pulsed light from the structured light transmitter (604). For example, a portion of the pulsed light can reflect from an object in the scene and propagate to the TOF receiver 305, and the TOF receiver 305 can sense pulses from such reflections.

[0061] Figure 7 700 is a flow chart describing example operations of TOF ranging of structured light emission. Starting at 702, Figure 3 The structured light emitter 301 may emit pulsed light in a spatial pattern. In some example embodiments, the power or current of one or more lasers of the emitter 301 may be varied to change the intensity of light from the one or more lasers. Figure 3 The device 300 (such as the controller 310) can control the pulses of the pulsed light from the emitter 301 to prevent Figure 3 The structured light receiver 302 senses the pulses (704). In some example embodiments, the device 300 may control the pulses so that multiple pulses occur during an exposure window of each frame captured by the structured light receiver 302. As a result, the structured light receiver 302 does not sense a single pulse. An increase in the pulse frequency may provide increased accuracy and / or increased points when measuring the depth of an object based on round trip time.

[0062] Figure 3 The TOF receiver 305 can sense a pulse (706) of the reflection of the pulsed light from the transmitter 301. In some example embodiments, the photodiode can be configured to sense when a pulse occurs in the zero-order portion of the spatial pattern. In addition, the structured light receiver 302 can sense the reflection of the pulsed light of the spatial pattern (708). As previously described, the exposure window of the structured light receiver 302 can be long so that the structured light receiver 302 does not sense the pulse in the reflection.

[0063] Referring back to step 706 regarding sensing the pulse, the controller 310 may determine a round trip time of the light (710). In some example embodiments, the controller 310 may compare the time of the pulse emission from the transmitter 301 and the time of the pulse being received or sensed by the TOF receiver 305 to determine the round trip time. In some other example embodiments, the controller 310 may determine a phase difference between the emission signal and the signal sensing the pulse and determine the round trip time based on the phase difference. The controller 310 may then determine the depth of the object based on the round trip time (712). For example, the depth may be half the round trip time multiplied by the speed of light.

[0064] Referring back to step 708 regarding sensing the spatial pattern of light, the controller 310 may identify one or more portions of the spatial pattern in the sensed reflection (714). In some example embodiments, the spatial pattern includes a plurality of codewords, and the controller 310 may identify one or more codewords in the sensed reflection. The controller 310 may then determine one or more depths of the scene based on the identified portions of the spatial pattern (716). For example, based on the distortion of the identified codeword and / or the position on the sensor sensing the identified codeword, the controller 310 may determine the depth of the object. Each identified codeword may correspond to a determined scene depth. In this manner, the device 300 may determine the depth using TOF ranging, and may determine the depth using structured light depth sensing based on structured light emission. Although the example operation 700 depicts parallel TOF ranging and structured light depth detection, these operations may be performed sequentially. Alternatively, only one of the processes for determining the depth may be used.

[0065] In some example embodiments, TOF ranging of structured light emission may have a higher effective range than structured light depth detection. For example, for a smartphone including an exemplary structured light system utilizing TOF ranging, TOF ranging may be effective up to 10 meters, while structured light depth detection may be effective up to 4 meters. Device 300 may determine whether to use TOF ranging or structured light to determine the depth of an object. Device 300 may perform structured light depth detection in a first mode corresponding to a first range of depths from device 300 (e.g., up to 4 meters from device 300), and device 300 may perform TOF ranging in a second mode corresponding to a range higher than the first range (e.g., greater than 4 meters from device 300).

[0066] In some example embodiments, if the device 300 in the second mode determines that the object is within the first range of depths based on TOF ranging, the device 300 may switch modes to use structured light for depth detection (such as being able to determine more depth points than using a TOF system). Additionally or alternatively, if the device 300 in the first mode determines that the object is outside the first range of depths or the error in identifying one or more parts of the spatial pattern is greater than a threshold (e.g., the device 300 cannot accurately determine the distance using structured light), the device 300 may switch to the second mode to use TOF ranging.

[0067] Although the device 300 can determine whether to use TOF ranging or structured light depth detection based on, for example, the depth of an object, both the TOF receiver and the structured light receiver can be operable. In some example embodiments, the device 300 can use both TOF ranging and structured light depth detection to determine the depth. The device 300 can then determine which depth to use. In some other example embodiments, the device 300 can ignore the measurement from one receiver based on whether to use TOF ranging or structured light depth detection to determine the depth. In some further example embodiments, the TOF receiver can be enabled when performing TOF ranging, and the structured light receiver can be enabled when performing structured light depth detection.

[0068] For devices with structured light systems such as Figure 3 In a device 300 of the present invention, the spatial pattern may be deformed due to the movement of components in a transmitter (such as transmitter 301). For example, a user may squeeze a smartphone while holding it (in order to maintain a grip), causing the spatial pattern from the structured light transmitter to be distorted. The distortion of the spatial pattern at the time of emission may cause errors or inaccuracies in determining one or more depths based on the reflection of the spatial pattern. In some example embodiments, the device may use TOF ranging to calibrate a structured light receiver (such as receiver 302).

[0069] Figure 8 is a method for calibrating structured light receivers (such as Figure 3 Flowchart of example operations 800 of a receiver 302 of FIG. 802. Starting from 802, Figure 3 The device 300 can be based on the use of Figure 3 The TOF range measurement of the TOF receiver 305 is used to determine the depth measurement of the object from the device 300. For example, the device 300 can sense the TOF from Figure 3 The reflected light pulse is a zero-order portion of the spatial pattern of the emitter 301. The sensed pulse can then be used to determine a first depth measurement of the object.

[0070] Device 300 may also determine a second depth measurement of the object based on structured light depth probing using structured light receiver 302 (804). For example, device 300 may identify a codeword corresponding to a zero-order portion of the spatial pattern in the reflection sensed by structured light receiver 302, and device 300 determines the second depth measurement of the object based on the identified codeword.

[0071] After determining the first depth measurement and the second depth measurement, the device 300 may determine a difference between the first depth measurement and the second depth measurement (806). Assuming that the first depth measurement is accurate (and not susceptible to errors due to component movement), the device 300 may determine that the error in the second depth measurement corresponds to the difference. The device 300 may then adjust the depth measurements determined using the structured light receiver 302 based on the difference (808). For example, the device 300 may add or subtract the difference from each depth measurement determined using the structured light receiver. In some example embodiments, the depth measurement may correspond to any identified codeword in the spatial pattern (not just the codeword corresponding to the zero-order portion of the spatial pattern).

[0072] Alternatively, to compensate for errors through structured light depth sensing, device 300 may determine the correct displacement and / or distortion of the codeword identified in the reflection based on the depth measurement using TOF ranging, and device 300 may adjust the measured distortion and displacement to correct the depth measurement using structured light depth sensing.

[0073] In some example embodiments, the device 300 may periodically determine the difference value for up to each frame captured by the structured light receiver 302. In this manner, the device 300 may calibrate the structured light receiver in real time and / or at any frequency.

[0074] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules or components may also be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software (such as a device for changing the spatial pattern of an included structured light system), the techniques may be implemented at least in part by a non-transitory processor-readable storage medium (such as a processor-readable storage medium) including instructions 308. Figure 3 The example memory 306 in the device 300 of the embodiment of the present invention is implemented, when the instruction 308 is executed by the processor 304 (or the controller 310 or the signal processor 312), the device 300 or the depth detection system performs one or more of the above methods. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0075] Non-transitory processor-readable storage media may include random access memory (Random Access Memory, RAM) (such as synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM)), read only memory (Read Only Memory, ROM), non-volatile random access memory (Non-Volatile Random Access Memory, NVRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, these technologies may be implemented at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read and / or executed by a computer or other processor.

[0076] The various illustrative logical blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors such as Figure 3 The processor 304 or signal processor 312 in the example device 300 of the example device 300 is executed. Such (multiple) processors may include, but are not limited to, one or more digital signal processors (Digital Signal Processor, DSP), general microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), application specific instruction set processors (Application Specific Instruction Set Processor, ASIP), field programmable gate arrays (Field Programmable Gate Array, FPGA) or other equivalent integrated or discrete logic circuits. The term "processor" used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the technology described herein. In addition, in some aspects, the functions described herein may be provided in a dedicated software module or hardware module configured as described herein. And, these technologies may be fully implemented in one or more circuits or logic elements. A general processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0077] Although the present disclosure shows illustrative aspects, it should be noted that various changes and modifications may be made herein without departing from the scope of the appended claims. For example, although the examples of the depth detection system are described with respect to light transmission or NIR transmission, signals of other frequencies, such as microwaves, radio frequency signals, sound signals, etc., may also be used. In addition, unless otherwise expressly stated, the functions, steps, or actions of the method claims according to the aspects described herein do not need to be performed in any particular order. For example, if performed by the device 300, the controller 310, the processor 304, and / or the signal processor 312, the steps of the described example operations may be performed in any order. In addition, although an element may be described or claimed in the singular, the plural number is also contemplated unless a limitation to the singular is expressly stated. Therefore, the present disclosure is not limited to the examples shown, and any means for performing the functions described herein are included in the various aspects of the present disclosure.

Claims

1. A depth detection system, comprising: a time-of-flight (TOF) receiver configured to sense pulses of reflections of light from a structured light emitter, the light from the structured light emitter comprising a spatial pattern of light spots, wherein the spatial pattern of light spots comprises a pulsed light spot, wherein the pulsed light spot has a higher intensity than the rest of the spatially patterned light; and The controller is configured as: determining information associated with the sensed pulse; and Based on the information, a round trip time of the sensed pulse is determined.

2. The depth detection system according to claim 1, wherein: The controller is configured to: The depth of an object from the TOF receiver is determined based on the round trip time of the sensed pulses. 3 . The depth detection system according to claim 1 , further comprising a structured light emitter for emitting the spatial pattern of light spots.

4. The depth detection system of claim 3, further comprising a structured light receiver configured to sense reflections of the spatial pattern of light spots, wherein the controller is further configured to identify one or more portions of the spatial pattern in the sensed reflections from the structured light receiver.

5. The depth detection system according to claim 4, wherein: The controller is further configured to control the pulse to prevent the structured light receiver from sensing the pulse.

6. The depth detection system according to claim 4, wherein: The TOF receiver includes a photodiode.

7. The depth detection system according to claim 6, wherein: The photodiode is an avalanche photodiode.

8. The depth detection system according to claim 6, wherein: The photodiode is configured to sense a pulse of reflection of light from a zero-order portion of the spatial pattern of light spots.

9. The depth detection system according to claim 4, wherein: The controller is also configured to calibrate the structured light receiver based on the sensed pulses from the TOF receiver.

10. The depth detection system according to claim 4, wherein: The controller is also configured to determine the depth of the object using the structured light receiver when the object is within a first range of depths from the depth detection system, and is also configured to determine the depth of the object using the TOF receiver when the object is within a second range of depths, wherein the second range is higher than the first range.

11. A method for determining one or more depths, comprising: sensing, by a time-of-flight (TOF) receiver, reflected pulses of light from a structured light emitter, the light from the structured light emitter comprising a spatial pattern of light spots, wherein the spatial pattern of light spots comprises a pulsed light spot, wherein the pulsed light spot has a higher intensity than a remainder of the spatially patterned light; determining information associated with the sensed pulse; and Based on the information, a round trip time of the sensed pulse is determined.

12. The method of claim 11, further comprising determining a depth of an object from the TOF receiver based on the round trip time of the sensed pulse.

13. The method of claim 11, further comprising emitting the spatial pattern of light spots by the structured light emitter.

14. The method according to claim 13, further comprising: sensing reflections of the spatial pattern of light spots by a structured light receiver; as well as One or more portions of the spatial pattern are identified in the sensed reflections from the structured light receiver.

15. The method of claim 14, further comprising controlling a pulse to prevent the structured light receiver from sensing the pulse.

16. The method according to claim 14, wherein: Sensing the pulse from the reflection includes sensing the pulse from the reflection of light from a zero order portion of the spatial pattern of light spots.

17. The method of claim 14, further comprising calibrating the structured light receiver based on the sensed pulses from the TOF receiver.

18. The method according to claim 14, further comprising: determining the depth of the object using the structured light receiver when the object is within a first range of depths; as well as The depth of the object is determined using the TOF receiver when the object is within a second range of depths, wherein the second range is higher than the first range.

19. A non-transitory computer-readable medium storing one or more programs containing instructions that, when executed by one or more processors of a device, cause the device to determine one or more depths by performing operations comprising: sensing, by a time-of-flight (TOF) receiver, reflected pulses of light from a structured light emitter, the light from the structured light emitter comprising a spatial pattern of light spots, wherein the spatial pattern of light spots comprises a pulsed light spot, wherein the pulsed light spot has a higher intensity than a remainder of the spatially patterned light; determining information associated with the sensed pulse; and Based on the information, a round trip time of the sensed pulse is determined.

20. The non-transitory computer readable medium of claim 19, wherein: Execution of the instructions causes the device to perform operations further comprising the following steps: The depth of the object from the TOF receiver is determined based on the round trip time of the sensed pulses.

21. The non-transitory computer readable medium of claim 19, wherein: Execution of the instructions causes the device to perform operations further comprising the following steps: The spatial pattern of light spots is emitted using the structured light emitter.

22. The non-transitory computer readable medium of claim 21, wherein: Execution of the instructions causes the device to perform operations further comprising the following steps: sensing reflections of the spatial pattern of light spots by a structured light receiver; as well as One or more portions of the spatial pattern are identified in the sensed reflections from the structured light receiver.

23. The non-transitory computer readable medium of claim 22, wherein: Execution of the instructions causes the device to perform operations further comprising the following steps: The pulses are controlled to prevent the structured light receiver from sensing the pulses.

24. The non-transitory computer readable medium of claim 22, wherein: Execution of the instructions causes the device to perform operations further comprising the following steps: A pulse of reflection of light from the zero-order portion of the spatial pattern of light spots is sensed.

25. The non-transitory computer readable medium of claim 22, wherein: Execution of the instructions causes the device to perform operations further comprising the following steps: The structured light receiver is calibrated based on the sensed pulses from the TOF receiver.

26. The non-transitory computer readable medium of claim 22, wherein: Execution of the instructions causes the device to perform operations further comprising the following steps: determining the depth of the object using the structured light receiver when the object is within a first range of depths; as well as The depth of the object is determined using the TOF receiver when the object is within a second range of depths, wherein the second range is higher than the first range.

27. An apparatus for determining one or more depths, comprising: means for emitting pulsed light in a spatial pattern of light spots, at least a portion of the spatial pattern of light spots comprising pulsed light spots; means for sensing reflections of said spatial pattern of light spots; and means for sensing reflected pulses from said at least a portion of said spatial pattern of light spots including said pulsed light spots, wherein the means for sensing pulses is separate from means for sensing reflections of light in said spatial pattern, wherein said pulsed light spots have a higher intensity than the rest of the spatially patterned light; and Means for determining a round trip time of the sensed pulse based on information associated with the sensed pulse.

28. The apparatus of claim 27, further comprising: Means for identifying in the sensed reflections of the spatial pattern of pulsed light one or more portions of the spatial pattern.

29. The apparatus of claim 27, further comprising means for controlling pulses to prevent the means for sensing the spatial pattern of pulsed light from sensing the pulses.

30. The apparatus of claim 27, further comprising means for sensing a pulse of reflection of light from a zero-order portion of the spatial pattern.