Projector with spatial light modulation

Abstract

To provide a favorable projector with spatial light modulation.SOLUTION: A time-of-flight based depth detection system is disclosed that includes a projector configured to sequentially emit multiple complementary illumination patterns. A sensor of the depth detection system is configured to capture the light from the illumination patterns reflecting off objects within the sensor's field of view. The data captured by the sensor can be used to filter out erroneous readings caused by light reflecting off multiple surfaces prior to returning to the sensor.SELECTED DRAWING: Figure 3A

Description

[Background technology]

[0001] BACKGROUND OF THE INVENTION Numerous techniques exist for range imaging, which can be very useful in several different applications. One specific type of range imaging can be performed using a time-of-flight camera. A time-of-flight camera measures the time it takes a pulse of light to travel to and return from an object in the sensor's field of view, allowing the distance between the sensor and the object in the sensor's field of view to be determined. Unfortunately, the light emitted by a depth-sensing system may not always travel directly to an object in the sensor's field of view and return to the sensor. If the light bounces off another object before reflecting from the object, the time it takes the light to return to the sensor is increased, thereby increasing the time-of-flight measured for the reflected pulse of light. A longer time-of-flight measurement can result in the depth-sensing system erroneously increasing the measured distance between the sensor and the object. As a result, a method for correcting this error is desirable. Summary of the Invention [Problem to be solved by the invention]

[0002] (Summary of the Invention) The present disclosure describes a time-of-flight camera that is configured to filter out erroneous readings resulting from pulses of light bouncing off multiple surfaces. [Means for solving the problem]

[0003] The present disclosure relates to a method for improving the performance of a depth detection system. The depth detection system can be configured to sequentially emit complementary illumination patterns onto an area monitored by an imaging sensor of the depth detection system. The imaging sensor can determine the distance between the depth detection system and an object within the area by acting as a time-of-flight sensor and measuring the time it takes for light to form the illumination pattern, reflect off the object, and return to the imaging sensor. Some of the light received at the imaging sensor may be indirect light that bounces off other surfaces before reaching the imaging sensor. This can be particularly problematic in room corners, where more indirect light is likely to return to the imaging sensor. Reflections increase the amount of time it takes for light to return to the imaging sensor, thereby reducing the accuracy of the sensor data. Some of this indirect light can be filtered out from consideration by the depth detection system when a first illumination pattern is active by identifying light that reflects from portions of the area monitored by the imaging sensor that are outside the first illumination pattern. This identified light can then be removed from consideration when a second illumination pattern is active. Similarly, when the second lighting pattern is active, light outside the second lighting pattern can be filtered out from the first lighting pattern. In this way, more accurate depth sensing information can be obtained.

[0004] The light sources, which emit complementary illumination patterns, can be mounted on a common substrate to prevent the light sources from moving out of alignment with one another. The common substrate can also help reduce any thermal effects that would result in the light sources being projected out of alignment.

[0005] A depth detection system is disclosed that includes at least the following: a projection system comprising a projector housing having a rigid substrate; a first light source configured to emit light through a first plurality of light shaping components, the first light source mounted to the rigid substrate; and a second light source configured to emit light through a second plurality of light shaping components, the second light source mounted to the rigid substrate adjacent to the first light source; an imaging sensor proximate to the projection system and configured to receive light emitted by the first and second light sources after being reflected from an object in a field of view of the imaging sensor; and a processor configured to calculate a distance between the depth detection system and an object in a field of view of the sensor by measuring the amount of time it takes for the light emitted by the first and second light sources to reflect from an object in the sensor field of view and return to the imaging sensor.

[0006] Another depth detection system is also disclosed, including: a plurality of light-shaping components, the plurality of light-shaping components comprising a collimating optical element, a refractive optical element, a diffractive optical element, and a microlens array; a light source configured to emit light through the plurality of light-shaping components; an imaging sensor configured to detect light emitted by the light source and reflected from an object within a field of view of the imaging sensor; and a processor configured to determine a distance between the depth detection system and the object by filtering out sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor.

[0007] A depth sensing system is disclosed, comprising: a projection system comprising: a projector housing having a rigid substrate; a first light source configured to emit light through a first plurality of light shaping components to produce a first illumination pattern, the first light source mounted to the rigid substrate; and a second light source configured to emit light through a second plurality of light shaping components to produce a second illumination pattern complementary to the first illumination pattern, the second light source mounted to the rigid substrate adjacent to the first light source; and a photography system. an image sensor proximate to the projection system and configured to receive light emitted by the first and second light sources after being reflected from objects within a field of view of the imaging sensor; and a processor configured to calculate a distance between the depth detection system and objects within the field of view of the sensor by measuring the amount of time it takes for the light emitted by the first and second light sources to reflect from objects within the field of view of the sensor and return to the imaging sensor, and filtering out sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor.

[0008] Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the described embodiments. The present specification also provides, for example, the following items: (Item 1) 1. A depth detection system, comprising: 1. A projection system comprising: a projector housing having a rigid substrate; a first light source configured to emit light through a first plurality of light shaping elements, the first light source being mounted to the rigid substrate; a second light source configured to emit light through a second plurality of light shaping elements, the second light source mounted to the rigid substrate adjacent to the first light source; a projection system comprising: an imaging sensor proximate the projection system and configured to receive light emitted by the first and second light sources after being reflected from objects within a field of view of the imaging sensor; a processor configured to calculate a distance between the depth detection system and the object in the sensor field of view by measuring the amount of time it takes for light emitted by the first and second light sources to reflect off the object in the sensor field of view and return to the imaging sensor; and A depth detection system comprising: (Item 2) Item 2. The depth detection system of item 1, wherein the first and second light sources are infrared laser diodes. (Item 3) Item 10. The depth detection system of item 1, wherein the imaging sensor has a global shutter. (Item 4) Item 10. The depth detection system of item 1, wherein the first and second light sources are configured to emit pulses in a non-overlapping pattern. (Item 5) Item 1. The depth detection system of item 1, wherein the first plurality of light shaping components comprises a diffractive optical element and a microlens array. (Item 6) 6. The depth detection system of item 5, wherein the first plurality of light shaping components shape the light emitted by the first light source into a first plurality of parallel bars of light that are distributed across the field of view of the imaging sensor. (Item 7) 7. The depth detection system of item 6, wherein the second plurality of light shaping components shape light emitted by the second light source into a second plurality of parallel bars of light that cover gaps between the first plurality of parallel bars of light. (Item 8) Item 10. The depth detection system of item 1, wherein the first plurality of light shaping components comprises a collimating lens with a folded optics. (Item 9) 9. The depth detection system of claim 8, wherein the second plurality of light shaping components comprises a collimating lens with the folded optical system. (Item 10) 10. The depth detection system of item 9, wherein the light projected by the first and second light sources is redirected at approximately 90 degrees by a reflective surface of the collimating lens. (Item 11) 2. The depth detection system of claim 1, wherein calculating the distance also includes filtering out sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor. (Item 12) 1. A depth detection system, comprising: a plurality of light shaping components, a collimating optical element; and a refractive optical element; and a diffractive optical element; and Microlens array and a plurality of light shaping components comprising: a light source configured to emit light through the plurality of light shaping components; an imaging sensor configured to detect light emitted by the light source and reflected from objects within a field of view of the imaging sensor; a processor configured to determine a distance between the depth detection system and the object by filtering out sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor; and A depth detection system comprising: (Item 13) Item 13. The depth detection system of item 12, wherein the plurality of light shaping components further comprise linearly actuated optics configured to be laterally offset relative to the diffractive optical element. (Item 14) The light source is a first light source, and the depth detection system further comprises: a printed circuit board; Second light source and Equipped with Item 13. The depth detection system of item 12, wherein the first and second light sources are electrically and mechanically coupled to the printed circuit board. (Item 15) Item 15. The depth detection system of item 14, wherein the plurality of light shaping components is a first plurality of light shaping components, and the depth detection system further comprises a second plurality of light shaping components through which the second light source is configured to emit light. (Item 16) Item 13. The depth detection system of item 12, wherein the collimating optical element comprises a folded optical system having a reflective surface configured to change the direction of the light emitted by the light source. (Item 17) 1. A depth detection system, comprising: 1. A projection system comprising: a projector housing having a rigid substrate; a first light source configured to emit light through a first plurality of light shaping elements to produce a first illumination pattern, the first light source mounted to the rigid substrate; a second light source configured to emit light through a second plurality of light shaping elements to produce a second illumination pattern complementary to the first illumination pattern, the second light source mounted to the rigid substrate adjacent to the first light source; a projection system comprising: an imaging sensor proximate the projection system and configured to receive light emitted by the first and second light sources after being reflected from objects within a field of view of the imaging sensor; a processor configured to calculate a distance between the depth detection system and the object within the sensor field of view by measuring the amount of time it takes for light emitted by the first and second light sources to reflect off the object within the sensor field of view and return to the imaging sensor, and by filtering out sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor; and A depth detection system comprising: (Item 18) Item 18. The depth detection system of item 17, wherein the processor filters sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor by identifying the light emitted by the first light source that reflects from an area within the field of view of the imaging sensor that corresponds to the second illumination pattern. (Item 19) Item 18. The depth detection system of item 17, wherein the first and second illumination patterns comprise a series of parallel bars. (Item 20) Item 18. The depth detection system of item 17, wherein the first plurality of light shaping components comprises a collimating optical element, a refractive optical element, a diffractive optical element, and a microlens array.

[0009] The present disclosure will be readily understood by the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals refer to like structural elements and in which: [Brief explanation of the drawings]

[0010] [Figure 1A] FIG. 1A shows an exemplary depth detection sensor in use. [Figure 1B] FIG. 1B illustrates how light incident on an object can be reflected by diffuse and / or specular reflection, according to some embodiments. [Figure 1C] FIG. 1C shows examples of different types of objects illuminated by a projection system, according to some embodiments. [Figure 2A] FIG. 2A shows a projection system 102 including two projectors, according to some embodiments. [Figure 2B] FIG. 2B shows exemplary illumination patterns A and B, according to some embodiments. [Figure 2C] FIG. 2C shows illumination patterns C and D, according to some embodiments. [Figure 2D]FIG. 2D shows illumination patterns E, F, and G, according to some embodiments. [Figure 2E] FIG. 2E illustrates how discrete pixels or sampling points may be distributed across multiple illumination patterns, according to some embodiments. [Figure 3A] 3A-3C show various optical assembly embodiments, each consisting of a group of light shaping components positioned in front of a light source, according to some embodiments. [Figure 3B] 3A-3C show various optical assembly embodiments, each consisting of a group of light shaping components positioned in front of a light source, according to some embodiments. [Figure 3C] 3A-3C show various optical assembly embodiments, each consisting of a group of light shaping components positioned in front of a light source, according to some embodiments. [Figure 4A] 4A-4B show a projector assembly with two light sources incorporating an optical assembly per light source similar to the optical assembly depicted in FIG. 3, according to some embodiments. [Figure 4B] 4A-4B show a projector assembly with two light sources incorporating an optical assembly per light source similar to the optical assembly depicted in FIG. 3, according to some embodiments. [Figure 5A] 5A-5C show diagrams of a multiple light source projector assembly utilizing folded optics, according to some embodiments. [Figure 5B] 5A-5C show diagrams of a multiple light source projector assembly utilizing folded optics, according to some embodiments. [Figure 5C] 5A-5C show diagrams of a multiple light source projector assembly utilizing folded optics, according to some embodiments. [Figure 6A] 6A-6B show side views of a projection assembly using a single light source, according to some embodiments. [Figure 6B]6A-6B show side views of a projection assembly using a single light source, according to some embodiments. [Figure 7] FIG. 7 shows a diagram depicting the interaction between different components of the depth detection system described above, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] Detailed Description of Specific Embodiments Representative applications of the methods and apparatus according to the present application are described in this section. These examples are provided solely to add context and aid in understanding the described embodiments. Thus, it will be apparent to one of ordinary skill in the art that the described embodiments may be practiced without some or all of these specific details. In other instances, well-known process steps have not been described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are also possible, and therefore the following examples should not be construed as limiting.

[0012] A depth detection system can be configured to characterize an environment within the field of view of the depth detection system. The resulting characterization can be used to determine the location of an object facing the depth detection system and the external shape of that portion. One type of depth detection system is a time-of-flight (TOF) camera. A TOF camera utilizes a projector for emitting modulated pulses of light and a sensor for receiving portions of each pulse of light that reflect off various objects within the field of view of the sensor. A processor receiving a reading from the sensor can determine the time it takes for the light to travel from the sensor, bounce off one of the objects in the field of view, and return to the sensor. Because the speed of light is known, the system can determine the distance between the depth detection sensor and the object based on that time. Unfortunately, while this method works well for determining distance when light bounces off an object and returns directly to the sensor, any light that first bounces off another object and returns to the sensor can cause inaccuracies in the depth data.

[0013] One solution to this problem is to filter out the indirectly reflected light received at the sensor to reduce inaccuracies. One way this can be accomplished is by adjusting the way the environment is illuminated with light. Light can be emitted by the projection system in an alternating illumination pattern to sequentially illuminate different portions of an object in the field of view. In some embodiments, the illumination pattern can be arranged in approximately parallel stripes, although different patterns are also possible. Each stripe can be separated by a gap having approximately the same thickness as each stripe. In this way, approximately half of the field of view can be illuminated each time an illumination pattern is emitted. It should be understood that different stripe and gap thicknesses can be used, but that at some point during a series of different illumination patterns, each portion of the field of view should not be illuminated. Any light returning from an area of ​​the frame that should not be illuminated by a particular pattern of light can be used to identify reflected light. When a different illumination pattern illuminates that portion of the object where reflected light was previously detected, the reflected light can be removed from the detected light, and only that portion of light that traveled directly from the projection system to the object and from there back to the sensor can be identified. Any other light can be ignored for the purposes of creating a depth map of the area with the field of view of the sensor. In this way, the accuracy of the depth data can be substantially improved.

[0014] A projection system for implementing the aforementioned methods can include two or more light sources to generate the illumination pattern. In some embodiments, the projection system can be configured to operate very quickly to keep up with changing conditions. For example, in some embodiments, the light source can be configured to emit more than 100 pulses per second. A sensor associated with the projection system can be configured to capture the light as it returns and can have a global shutter that allows each of the sensor's pixels to be read simultaneously. In this way, any errors introduced due to sequentially reading the pixels can be avoided.

[0015] In some embodiments, the light sources can be integrated into a single projector housing. Packaging the light sources into a single projector prevents a situation in which one of two or more separate projection units clashes or competes with the other units in different amounts, resulting in misaligned illumination patterns. Slight alignment changes in a single projector configured to project multiple illumination patterns can result in a portion of the sensor field of view not being covered by the illumination pattern, but a majority of the sensor field of view can remain covered without compromising the alignment of the illumination pattern. In some embodiments, the single projector housing can include a monolithic rigid substrate with a low coefficient of thermal expansion that keeps the separation between the light sources consistent over a wide range of temperatures. Each light source can have different optics that direct light into various illumination patterns. In some embodiments, a projection system with a single light source with shifting optics can be used. In such embodiments, the optics can oscillate between two or more positions, resulting in two or more illumination patterns from a single light source.

[0016] These and other embodiments are discussed below with reference to Figures 1A-7. However, those skilled in the art will readily understand that the detailed description provided herein with reference to these figures is for illustrative purposes only and should not be construed as limiting.

[0017] FIG. 1A illustrates an exemplary depth-sensing system 100 in use. The depth-sensing system 100 includes a projection system 102 and a sensor 104. The projection system 102 can be configured to emit light toward an object 106. In some embodiments, the light emitted by the projection system 102 can be infrared or near-infrared light. Because the light emitted by the projection system 102 can be configured to cover a wide area corresponding to the field of view of the sensor 104, an exemplary light wave 108 may bounce off a wall 110; however, due to the angle of the wall 110, the light wave 108 may bounce off the object 106 and then return to the sensor 104 instead of reflecting back from the wall 110 as depicted. This can be particularly problematic when the object 106 has an irregular surface (i.e., a curved or cylindrical surface) that scatters light incident on the object 106. The scattering of the reflected light increases the likelihood that the reflected light will return to the sensor 104 as depicted.

[0018] FIG. 1B illustrates how light incident on the object 106 can be reflected by diffuse and / or specular reflection. While a flat surface is generally required to produce specular reflection, flat surfaces also tend to produce some degree of diffuse reflection due to scattering centers located below the surface of the object 106. Curved or varying surfaces produce even more diffuse reflection, scattering in many directions. One reason that light reflected from the wall 110 can be difficult to distinguish from direct light is that when the surface of the wall 110 is relatively flat, a substantial amount of the light wave 108 is reflected as specular reflection from the wall 110, causing the resulting diffuse reflection at the object 106 from the light wave 108 to have a similar intensity to the resulting diffuse reflection at the object 106 from the light wave 112. Note that light traveling from the projector to the object 106, then bouncing off the wall 110 and back toward the sensor is not considered a problem if the wall 110 is not within the sensor's field of view. In such cases, high incidence angles of light entering the sensor will not be detected by the sensor because the sensor is configured to only receive light arriving from a particular field of view. High incidence angle light can be prevented from reaching the sensor using a shroud or focusing lens positioned over the sensor.

[0019] FIG. 1C shows examples of different types of objects illuminated by the projection system 102. The images in the first column show images generated using all of the light reflected from the object and captured by the sensor 104. The images in the second column show only light directly reflected from the object. The images in the third column show only light initially reflected from other objects (indirect light) prior to striking an object in the sensor's field of view. The egg photographs in the first row provide an example of diffuse interreflection. The spherical shape of the egg emphasizes the amount of diffuse reflection generated by light striking each surface of the egg. In particular, the indirect light image from the first row shows how the bottom edge of the egg captures a substantial amount of indirect light and, as a result, can appear farther away from the sensor. The wooden block photographs in the second row provide an example of both diffuse and specular interreflection. The flat surface of the wooden block provides a certain amount of specular reflection, while the underlying grain structure and corners of the block provide diffuse interreflection. Finally, the peppers in the third row show how subsurface scattering causes only a small amount of light to be reflected directly back to the sensor 104. This limited amount of direct light can make filtering out indirect light even more important for determining the actual distance between the sensor 104 and the peppers. Figure 1C was originally published as part of the article "Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination" by Krishnan.

[0020] FIG. 2A shows projection system 102, which includes projectors 202 and 204. Projectors 202 and 204 can be used to emit complementary illumination patterns A and B. Illumination patterns A and B can be pulsed sequentially so that only one of the illumination patterns is active at any given time. In some embodiments, the illumination patterns can be pulsed in an alternating pattern (e.g., in an A, B, A, B pattern). The pulsed radiation can also be modulated to help distinguish the pulsed radiation from other ambient light sources. As a result, when illumination pattern A is active, any areas outside illumination pattern A will be devoid of light. However, generally, portions of illumination pattern A that reflect off other surfaces, and in some environments, other ambient light, will initially reflect from areas not directly illuminated by illumination pattern A and can be detected by sensor 104. This reflected light detected in the unilluminated areas of object 106 can then be used to identify reflected light when illumination pattern B is active. Similarly, when lighting pattern B is active, reflected light arriving from outside lighting pattern B can subsequently be used to identify reflected light during the next pulse of lighting pattern A. Thus, in general, detected reflected light or indirect light (I) originating from outside the active lighting pattern can be used to identify reflected light during the next pulse of lighting pattern A. INDIRECT ) can be recorded. When the next lighting pattern is activated, the previously recorded indirect light (I INDIRECT ) is the total light received from the active illumination pattern (I TOTAL ) and can distinguish direct light. I INDIRECT =I TOTAL -I INDIRECT Equation (1)

[0021] It should be noted that in some embodiments, any ambient light that reflects off the object 106 and back into the sensor 104 can be filtered out by filtering out light that does not match the modulation associated with the illumination pattern.

[0022] FIG. 2B shows exemplary lighting patterns A and B. The intensities of lighting patterns A and B can be distributed in a sinusoidal pattern as a function of vertical position. As depicted, lighting pattern A can be 180 degrees out of phase with lighting pattern B, resulting in lighting pattern A having a maximum intensity value when lighting pattern B is at a minimum intensity value. In this way, when the two lighting patterns are emitted simultaneously, a substantially uniform light pattern will be produced. Graph 206 illustrates lighting pattern A, while graph 208 illustrates lighting pattern B. Mathematically, the intensity of the combined patterns will cause the intensity value to have a substantially constant value equal to 1. More generally, lighting intensity can be modeled using equation (2):

number

[0023] In equation (2), i denotes the illumination pattern out of a total of N illumination patterns being calculated; A0 is the amplitude of the illumination pattern; f is the spatial frequency of the light bars; β is the angle of the vertical field of view of the sensor; Φ i represents the phase shift for the illumination pattern, the value of which is determined by equation (3).

number

[0024] As can be seen, equation (3) clarifies that the phase shift can be 180 degrees for two patterns, 120 degrees for three patterns, 90 degrees for four patterns, and so on. In general, more illumination patterns can be used to achieve more accurate results. Furthermore, in some embodiments, the phase shift can also be varied in different ways.

[0025] FIG. 2C shows illumination patterns C and D. The intensity profiles of illumination patterns C and D are trapezoidal instead of sinusoidal. By having rapidly rising and falling intensities, sharper transitions between the bars of light of illumination patterns C and D can be achieved. Sharper transitions can be beneficial in minimizing ambiguity when filtering indirect light from direct light, as will be explained in more detail below.

[0026] FIG. 2D shows illumination patterns E, F, and G. The intensities of illumination patterns E, F, and G are vertically distributed, such that illumination pattern F is 120 degrees out of phase with illumination pattern E. In this manner, successive bars of light can be vertically shifted, but not complementary in nature. Graphs 214, 216, and 218 quantitatively illustrate the amount by which the individual illumination patterns E, F, and G vary with vertical position. A third illumination pattern can be generated by a third light source or by an optical system that is shifted and can result in both the second and third patterns.

[0027] FIG. 2E illustrates how discrete pixels or sampling points may be distributed across multiple illumination patterns. A close-up 220 shows three different sampling points p1, p2, and p3 distributed within illumination patterns A and B. The indirect light at each of the sampling points can be identified by performing several calculations for each pixel / sampling point. In particular, equation (4) shows that the light S collected by the sensor during each sequential illumination pattern is i can be used to sum up

number

[0028] Equation (5) can then be used to calculate the amount of direct light when the intensity of each illumination pattern varies sinusoidally.

number

[0029] Equation (5) sums the amplitude of each component of light received when each of the lighting patterns is active to represent the total amount of light emitted across the span of a set of lighting patterns. In a two-light pattern projection system, the subtracted image represents the reflected light detected from lighting pattern A when lighting pattern B is active and the reflected light detected from lighting pattern B when lighting pattern A is active. By adding the two sets of reflected light together, the distribution of reflected light across the entire field of view can be determined. In general, this calculation assumes that the reflected light remains substantially the same regardless of the lighting pattern that is active. As a result, the subtracted image is subtracted from the total light to identify the direct light in the field of view. Equation (6) shows how indirect light (I) can be calculated by subtracting the calculated direct light (D) from the total light (T). I=TD-const[GL] Equation (6)

[0030] In some embodiments, const[GL] can be subtracted from the total light. This constant can optionally be used to remove gray level bias when identifying indirect light within the sensor field of view. In some embodiments, removing the gray level bias can improve the accuracy of the depth data detected by the sensor. The gray level bias can be a factory setting or a value that can be periodically calibrated to keep the depth detection system working well.

[0031] FIG. 2E also demonstrates how depth detection at position p2 can be problematic for a system with only two lighting patterns. For p1 and p3, which are located safely away from the boundary between the lighting patterns, indirect light exclusion can be simple because there are only two lighting patterns to consider. For p1, when lighting pattern A is active, the received signal is equal to direct light plus any reflected light. When lighting pattern B is active, the received signal at p1 is equal to zero direct light plus any reflected light. Direct light can be calculated by taking the difference between the two signals. This yields only direct light because the reflected light cancels out and direct light during lighting pattern B is equal to zero. For p3, the calculation works in a similar manner, yielding only direct light. Unfortunately, at p2, which is located exactly at the interface between the lighting patterns, direct light from both patterns A and B will be detected at nearly the same intensity. This means that taking the difference in values ​​yields a zero value. Furthermore, areas near the interface will also suffer from some degree of inaccuracy whenever direct light from both lighting patterns is present in substantial amounts. As a result, lighting patterns with sharp boundaries between them will have less inaccuracy at the interface between them. However, direct light values ​​for points near the interface can still be calculated by interpolation. The direct light value for p3 can be calculated by interpolation from the direct light values ​​for p4 and p5. Generally, p4 and p5 should be as close to p2 as possible. For example, the processor can be configured to select interpolated point p4 with an amount of direct light from lighting pattern B below a predetermined threshold.

[0032] FIG. 3A shows a first optical assembly 300, which consists of a group of light-shaping components positioned in front of a light source 302. In some embodiments, the light source 302 can be an infrared laser diode. The light source 302 emits light that passes through a first light-shaping component, a collimating lens 304. The collimating lens 304 can be configured to focus the light 306 emitted by the light source 302 toward a second light-shaping component, a refractive optical element 308. The refractive optical element 308 tilts the focused light 306 by an angle θ, elongating the light vertically and generating a super-Gaussian beam 310 that is directed toward a third light-shaping component, a diffractive optical element 312. The diffractive optical element 312 then multiplies the super-Gaussian beam 310. The super-Gaussian 310 is depicted as being multiplied by a factor of five for illustrative purposes, although the number can vary. For example, in some embodiments, the diffractive optical element 312 can be configured to multiply the super-Gaussian beam by 25 times. The number and thickness of the multiplied super-Gaussian beams 310 can be selected to match the vertical field of view of the associated imaging sensor. As the super-Gaussian beams pass through the microlens array 314, the microlens array 314 spreads each super-Gaussian beam horizontally, resulting in an illumination pattern that illuminates region 316, as depicted. The microlens array 314 can be double-sided (as depicted), single-sided, or cylindrical. In some embodiments, region 318 and region 316 can be approximately the same size. Light from a second optical assembly can be configured to illuminate region 318. In some embodiments, the optical assemblies can emit light in complementary patterns such that one of horizontal regions 316 and 318 is illuminated at any given time.

[0033] FIG. 3B shows a second optical assembly 320, which consists of a group of light-shaping components positioned in front of a light source 322. In some embodiments, the light source 322 can be an infrared laser diode. The light source 322 emits light that passes through a first light-shaping component, a collimating lens 324. The collimating lens 324 can be configured to focus the light 326 emitted by the light source 322 toward a second light-shaping component, a refractive optical element 328. The refractive optical element 328 tilts the focused light 326 by an angle −θ, elongating the light vertically and generating a super-Gaussian beam 330 that is directed toward a third light-shaping component, a diffractive optical element 332. In some embodiments, orienting the super-Gaussian beam 330 in a direction opposite to that of the super-Gaussian beam 310 can reduce the risk of crosstalk between the light sources. The diffractive optical element 332 then magnifies the super-Gaussian beam 330. The super-Gaussian beams 330 are depicted as being multiplied by a factor of five for illustrative purposes, although the number can vary. For example, in some embodiments, the diffractive optical element 312 can be configured to multiply the super-Gaussian beams by a factor of 25. The number and thickness of the multiplied super-Gaussian beams 330 can be selected to match the vertical field of view of the associated imaging sensor. As the super-Gaussian beams pass through the microlens array 334, the microlens array 334 creates an illumination pattern that spreads each super-Gaussian beam horizontally and illuminates the region 318, as depicted. In this manner, the light sources 322 and 302 can cooperatively illuminate the regions 316 and 318. The illumination of the regions 316 and 318 can be staggered in different patterns. For example, the regions 316 and 318 can be illuminated sequentially such that light shines in both regions for approximately the same amount of time.

[0034] 3C shows another optical assembly 340, which consists of three light-shaping components positioned in front of a light source 342. In some embodiments, the light source 342 can be an infrared laser diode. The light source 342 emits light that passes through a first light-shaping component in the form of a collimating lens 344. The collimating lens 344 can be configured to collimate the light 346 emitted by the light source 342, which travels toward a second light-shaping component in the form of an optical element 348. The optical element 348 can include both a refractive surface 350 on a first side of the optical element 348 and a diffractive surface 352 on a second side of the optical element 348. The refractive surface 350 and the diffractive surface 352 can take the form of a polymer material molded onto opposite sides of a glass or polycarbonate substrate. When collimated light 336 passes through refractive surface 340, the light is tilted by angle θ and stretched into a super-Gaussian beam 354 within optical element 348. When super-Gaussian beam 354 passes through diffractive surface 352, it can be multiplied into multiple super-Gaussian beams 354. When super-Gaussian beam 354 passes through microlens array 356, microlens array 356 spreads each super-Gaussian beam 354 horizontally, resulting in an illumination pattern that illuminates area 316, as depicted. In this manner, light source 342 illuminates area 316.

[0035] 4A-4B show a projector assembly 400 with two light sources, each incorporating an optical assembly similar to optical assembly 300. FIG. 4A shows a top view of projection assembly 400. Projection assembly 400 includes light sources 402 and 404. Both light sources 402 and 404 can be mounted to a rigid substrate 406. In some embodiments, rigid substrate 406 can be formed from an alumina ceramic. Rigid substrate 406 keeps light sources 402 and 404 from shifting relative to one another. Rigid substrate 406 can also have a low coefficient of thermal expansion, which reduces shifting of light sources 402 and 404 relative to the optical assembly.

[0036] Light source 402 emits light through a first portion of a dual collimating lens 408, which focuses the light toward optical assembly 410. A second portion of dual collimating lens 408 focuses the light emitted by light source 404 toward optical assembly 412. In some embodiments, dual collimating lens 408 can be replaced by two separate collimating lenses that perform the same function. Optical assemblies 410 and 412 can each include a refractive optical element similar to 308, a diffractive optical element similar to 312, and a microlens array similar to 314 to spread the light from each light source in an illumination pattern. Optical assembly 410 differs slightly from optical assembly 412 in that the illumination pattern produced by light source 404 can be vertically offset from the illumination pattern produced by light source 402 so that the illumination patterns are complementary. This allows bars of light from one illumination pattern to be positioned between bars of light of the other illumination pattern. In this manner, the illumination patterns produced by light sources 402 and 404 cooperate to evenly cover the surface. In some embodiments, a refractive optical element can shift the light from light source 404 in an opposite direction to the light produced by light source 402.

[0037] Projector assembly 400 can also include a processor 414 mounted on PCB 416 and configured to synchronize the output from light sources 402 and 404. For example, processor 414 can be mounted on PCB 416 and configured to direct light sources 402 and 404 to emit staggered pulses of light such that neither lighting pattern is active at the same time. Processor 414 can also direct modulation of light source 404 and help the depth sensor distinguish pulses of light from other ambient light sources. In some embodiments, processor 414 can also be in communication with a sensor configured to receive pulses of light after being reflected from objects within the sensor's field of view.

[0038] 4B shows a side view of the projection assembly 400. In particular, the light source 404 is shown elevated by a rigid substrate 406. The rigid substrate can be inserted into a cutout defined by a PCB 416. The rigid substrate 406 can also form a base for a projector housing 418 of the projector assembly 400. The projector housing 418 can define a step 420 for supporting the dual collimating lens 408.

[0039] 5A-5B show diagrams of a multiple light source projector assembly 500 utilizing folded optics. Fig. 5A shows how projector assembly 500 includes two separate sets of optics, namely optical assemblies 410 and 412, which receive light emitted from folded optics 502 of collimating lens 408. Folded optics 502 allows light sources 402 and 404 to be positioned closer to collimating lens 408 by shifting light path 504 laterally, thereby allowing an overall reduction in the height of projector assembly 500.

[0040] 5B shows how, by laterally shifting the light path 504, the height of the projector assembly 500 can be reduced, thereby allowing the projector assembly 500 to be packaged in a smaller form factor device. In particular, the laterally shifted light path 504 allows the length of the light path to be divided into horizontal and vertical sections. The overall height of the projector assembly 500 is reduced because the portion of the light path in the horizontal section does not need to be incorporated into the overall height of the projector assembly 500. The direction of the light path through the folded optics 502 is redirected by an optically reflective surface 506, which redirects the light from a horizontal orientation to a vertical orientation. In some embodiments, the optically reflective surface 506 can be mirrored.

[0041] 5C shows projector assembly 510, which can have a lower overall height than projector assembly 500. Collimating lens 408 can include both folded optics 502 and cylindrical lens surface 508. Cylindrical lens surface 508 can partially collimate the light emitted by light source 404 by narrowing the width of the light entering collimating lens 408. Folded optics 502 can be shorter vertically due to the narrowed beam width of the light emitted by light source 404. The light then becomes fully collimated upon exiting collimating lens 408. In this manner, the height of collimating lens 408 can be reduced.

[0042] 6A-6B show side views of a projection assembly 600 using a single light source 602. FIG. 6A shows the projection assembly 600 in an inactive configuration. Because the projection assembly 600 only includes a single light source 602 to provide two different illumination patterns, the projection assembly 600 includes a linearly actuated optical system 606 configured to provide two complementary illumination patterns. The optical system 606 can be linearly actuated by a piezoelectric motor 608, which actuates the optical system 606 between two or more positions by rotating a linkage 610 between the two positions shown in FIG. 6B. The piezoelectric motor 608 can be configured to oscillate the optical system 606 back and forth at a rate that enables the light source 602 to sequentially project complementary illumination patterns 612 and 614. The light source 602 can be synchronized with the oscillation rate of the optical system 606 so that the light source 602 emits light only when the optical system 606 is in a position corresponding to one of the complementary illumination patterns. It should be noted that although only two lighting patterns are shown, the piezoelectric motor 608 can also be configured to define three or more different lighting patterns.

[0043] FIG. 7 shows a schematic diagram depicting the interaction between the different components of the depth detection system described above. The top of the flowchart indicates the start of the interaction, which progresses as you move down the flowchart. The projector of the depth detection system transmits alternating first and second illumination patterns. An object within the sensor field of view of the depth detection system reflects a portion of the first and second illumination patterns back into the sensor of the depth detection system. Light traveling directly from the projector to the object and returning (direct light) will arrive at the sensor before light bouncing off another surface (indirect light) arrives before returning to the sensor. As a result, a time-of-flight depth detection system will incorrectly increment the distance of the object from the sensor when indirect light is considered. The sensor then transmits the light received from the first and second illumination patterns to a processor. The processor can then be configured to filter out indirect light from the total light received so that only light that travels directly from the projector to the object and back to the sensor is considered when determining the distance between the sensor and objects in the sensor's field of view. The processor can then assign objects in the sensor's field of view to appropriate depth planes of a display associated with the depth-sensing sensor. Finally, the processor can transmit images to depth planes corresponding to various objects in the sensor's field of view.

[0044] Various aspects, embodiments, implementations, or features of the described embodiments can be used separately or in any combination. Various aspects of the described embodiments can be implemented by software, hardware, or a combination of hardware and software. The described embodiments can also be embodied as computer-readable code on a computer-readable medium for controlling manufacturing operations or for controlling a manufacturing line. A computer-readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of computer-readable media include read-only memory, random-access memory, CD-ROMs, HDDs, DVDs, magnetic tape, and optical data storage devices. The computer-readable medium can also be distributed across network-coupled computer systems so that the computer-readable code is stored and executed in a distributed fashion.

[0045] The foregoing description, for purposes of explanation, uses specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that specific details are not required to practice the described embodiments. Accordingly, the foregoing descriptions of specific embodiments are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the described embodiments to the precise forms disclosed. It will be apparent to those skilled in the art that numerous modifications and variations are possible in light of the above teachings.

Claims

1. 1. A method of operating a depth detection system, the method comprising: generating a first pulsed illumination pattern for a first time period by emitting light through a first plurality of light shaping elements with a first light source of a projection system of the depth detection system, the first light source being mounted to a rigid substrate of the projection system; generating a second pulsed illumination pattern complementary to the first pulsed illumination pattern during a second time period by emitting light through a second plurality of light shaping elements with a second light source of the projection system, the second light source being mounted to the rigid substrate adjacent to the first light source, the second time period not overlapping with the first time period, and the emission of light by the second light source following the emission of light by the first light source; receiving, by an imaging sensor proximate to the projection system, light emitted by the first light source after being reflected from an object within a field of view of the imaging sensor during the first time period; receiving, by the imaging sensor, light emitted by the second light source after being reflected from the object within the field of view of the imaging sensor during the second time period; calculating, by a processor of the depth detection system, a distance between the depth detection system and the object within the field of view of the imaging sensor by measuring the amount of time it takes for light emitted by the first and second light sources to reflect off the object within the field of view of the imaging sensor and return to the imaging sensor, and by filtering out sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor; A method of operating a depth detection system, comprising:

2. 2. The method of operating a depth detection system of claim 1, further comprising filtering, by the processor of the depth detection system, sensor readings associated with light reflected from surfaces outside the field of view of the imaging sensor by identifying the light emitted by the first light source that reflects from an area within the field of view of the imaging sensor that corresponds to the second pulsed illumination pattern.

3. The method of operating a depth sensing system of claim 1 , wherein the first and second pulsed illumination patterns comprise a series of parallel bars.

4. The method of operating a depth sensing system of claim 1 , wherein the first plurality of light shaping components comprises a collimating optical element, a refractive optical element, a diffractive optical element, and a microlens array.

Images