Depth sensing system and method

Abstract

The present invention relates to depth sensing systems and methods. A depth sensing system includes a sensor having a first sensor pixel and a second sensor pixel to receive light from a surface. The system also includes a filter that allows full spectrum light to be transmitted to the first sensor pixel and visible light to be transmitted to the second sensor pixel while blocking infrared light from being transmitted to the second sensor pixel. The system further includes a processor that is used to analyze the full spectrum light and the visible light to determine the depth of the surface. The filter is disposed between the sensor and the surface.

Description

[0001] This application is a divisional application of an application with a filing date of February 28, 2017, application number 201780014934.2, and invention name “Depth Sensing System and Method”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Application Serial No. 62 / 301,847, filed on March 1, 2016, attorney docket No. ML.30066.00, and entitled “DEPTH SENSING SYSTEMS AND METHODS.” This application is related to U.S. Provisional Patent Application Serial No. 62 / 012,273, filed on June 14, 2014 (attorney docket No. ML.30019.00), and U.S. Utility Patent Application Serial No. 14 / 738,877, filed on June 13, 2013 (attorney docket No. ML.20019.00). The entire contents of the foregoing patent applications are expressly and entirely incorporated herein by reference as if fully set forth. Described in the patent applications with attorney docket Nos. 62 / 012,273 and 14 / 738,877 are various embodiments of virtual reality and augmented reality systems in which projected light is used for depth sensing. Additional embodiments of projected light depth sensing systems and light sensors for depth sensing are described herein. Technical Field

[0004] The present disclosure relates to depth sensing systems and methods. Background Art

[0005] Depth sensing is the determination of the distance between a known point (e.g., a sensor) in three-dimensional ("3D") space and a point of interest ("POI") on the surface of an object. Depth sensing is also called texture sensing because the determination of the respective distances of multiple POIs on a surface determines the texture of the surface. Depth or texture sensing is useful for many computer vision systems, including mixed reality systems.

[0006] Modern computing and display technologies have facilitated the development of mixed reality systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears real or can be perceived as real. Virtual reality or "VR" scenarios generally involve presenting digital or virtual image information in a manner that is opaque to other actual real-world visual input. Augmented reality or "AR" scenarios generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user (i.e., transparent to other actual real-world visual input). Thus, AR scenarios involve presenting digital or virtual image information in a manner that is transparent to other actual real-world visual input.

[0007] Various optical systems generate images at different depths for displaying mixed reality (VR and AR) scenes. Some such optical systems are described in U.S. Utility Patent Application Serial No. 14 / 738,877 (Attorney Docket No. ML.20019.00), the contents of which have been previously incorporated herein by reference. Other such optical systems for displaying mixed reality scenes are described in U.S. Utility Patent Application Serial No. 14 / 555,585 (Attorney Docket No. ML.20011.00), filed on November 27, 2014, the entire contents of which are expressly and entirely incorporated herein by reference as if fully set forth.

[0008] AR scenes usually include the presentation of virtual image elements related to real-world objects. Figure 1 , depicting an augmented reality scene 100 in which a user of AR technology sees a real-world park-like setting 102 that features people, trees, buildings in the background, and a concrete platform 104. In addition to these items, the user of the AR technology also perceives that he "sees" a robot statue 106 standing on the real-world platform 104, and a cartoon-like avatar character 108 that appears to be an avatar of a bumblebee flying, even though these elements 106, 108 do not exist in the real world. In order to present a believable or conveyable AR scene 100, the depth of real-world objects (e.g., platform 104) must be determined to present virtual objects (e.g., robot statue 106) in relation to the real-world objects.

[0009] VR scenes that include partial reproductions of real-world environments may also benefit from determining the depth and texture of those portions of the real-world environment. Accurate depth and texture information will lead to more accurate VR scenes. Both AR and VR scenes may also include outward-facing cameras to capture portions of the real-world environment (e.g., for analysis or transmission). Determination of the depth of those portions of the real-world environment may help focus these outward-facing cameras.

[0010] One method of depth sensing includes measuring the corresponding angles between the optical axes of two images of a single POI on a surface and the POI on the corresponding image, the two images being separated by a known distance at a known orientation. The depth of the surface is then determined by triangulating the measured angles and the known distance between the image capture locations. Problems with this approach include: (1) identification of the POI in the first image (particularly on homogeneous surfaces) (the "identification problem"), and (2) identification of the corresponding POI in the second image (the "correspondence problem"). The systems and methods described herein are configured to address these challenges. Summary of the invention

[0011] Embodiments of the present invention relate to devices, systems, and methods for facilitating virtual reality and / or augmented reality interaction with one or more users.

[0012] In one embodiment, a depth sensing system includes a sensor having first and second sensor pixels to receive light from a surface. The system also includes a filter that allows full spectrum light to be transmitted to the first sensor pixel and allows visible light to be transmitted to the second sensor pixel while blocking infrared light from being transmitted to the second sensor pixel. The system further includes a processor that is used to analyze the full spectrum light and the visible light to determine the depth of the surface. The filter is disposed between the sensor and the surface.

[0013] In one or more embodiments, the sensor has a plurality of second sensor pixels including the second sensor pixel, each of the plurality of second sensor pixels being adjacent to the first sensor pixel, and the filter allowing visible light to be transmitted to each of the plurality of second sensor pixels while preventing infrared light from being transmitted to each of the plurality of second sensor pixels. Analyzing the full spectrum light and the visible light may include calculating an estimated visible light value for the first sensor pixel based on a plurality of detected visible light values ​​corresponding to the plurality of second sensor pixels. Calculating the estimated visible light value may include averaging the plurality of detected visible light values. Calculating the estimated visible light value may include performing edge detection on the plurality of detected visible light values.

[0014] In one or more embodiments, the sensor has a plurality of first sensor pixels including the first sensor pixel, each of the plurality of first sensor pixels being adjacent to the first sensor pixel, and the filter allowing visible light to be transmitted to each of the plurality of second sensor pixels while blocking infrared light from being transmitted to each of the plurality of second sensor pixels. Analyzing the full spectrum light and the visible light may include calculating a plurality of estimated visible light values ​​for each of the plurality of first sensor pixels, and calculating an estimated visible light value for the first sensor pixel based on at least some of the plurality of estimated visible light values.

[0015] In one or more embodiments, the system further comprises a spatially modulated light projection device, the spatially modulated light projection device being used to project the light toward the surface, wherein the light is reflected from the surface toward the sensor. Analyzing the full spectrum light and the visible light may include generating an infrared light image of the surface. Analyzing the full spectrum light and the visible light may include triangulating POIs in the infrared light image of the surface.

[0016] In another embodiment, a depth sensing system includes a spatially modulated light projection device, which is used to project light onto a surface. The system also includes a sensor, which is used to receive light reflected from the surface. The system further includes an actuator, which is used to control the spatial modulation of the light projection device and receive light information from the sensor. In addition, the system includes a processor, which is used to analyze the light reflected from the surface to determine the depth of the surface. The light includes infrared light having a wavelength from about 700nm to about 1mm.

[0017] In one or more embodiments, the light includes visible light having a wavelength from about 390nm to about 700nm. The actuator may control the spatial modulation of the light projection device to form a pattern including the infrared light and the visible light. The actuator may control the spatial modulation of the light projection device to form two intersecting line segments on the surface. The actuator may control the spatial modulation of the light projection device to form a static pattern including two intersecting line segments on the surface. The actuator may control the spatial modulation of the light projection device to form a dynamic pattern including two intersecting line segments on the surface. The actuator may control the spatial modulation of the light projection device to form a pattern including multiple discrete intersecting line segments on the surface.

[0018] In one or more embodiments, controlling the spatial modulation of the light projection device includes controlling the movement of at least a portion of the light projection device. Controlling the spatial modulation of the light projection device may include controlling the light projection device to project the light. The spatially modulated light projection device may include a fiber scanning display. The spatially modulated light projection device may include a laser source. The spatially modulated light projection device may include a laser scanning display.

[0019] In yet another embodiment, a depth sensing system includes a sensor having a first sensor pixel and a second sensor pixel to receive light from a surface. The system also includes a filter that allows a first proportion of visible light and a second proportion of infrared light to be transmitted to the first sensor pixel, and a third proportion of visible light and a fourth proportion of infrared light to be transmitted to the second sensor pixel. The first proportion of visible light and the second proportion of infrared light result in a first sensing value at the first sensor pixel. The third proportion of visible light and the fourth proportion of infrared light result in a second sensing value at the second sensor pixel. The system further includes a processor that is used to analyze the first sensing value and the second sensing value to determine the depth of the surface. The filter is disposed between the sensor and the surface.

[0020] In one or more embodiments, analyzing the first sensed value and the second sensed value includes generating an infrared light image of the surface. Analyzing the first sensed value and the second sensed value may include triangulating POIs in the infrared light image of the surface.

[0021] Additional and other objects, features, and advantages of the present invention are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings illustrate the design and utility of various embodiments of the present invention. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by the same reference numerals throughout the drawings. In order to better understand how the above and other advantages and objects of the various embodiments of the present invention are obtained, a more detailed description of the invention briefly described above will be provided by reference to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0023] Figure 1 shows a user view of a user device through wearable AR according to one embodiment;

[0024] Figures 2 to 4 are detailed views of line segments and patterns projected by various prior art depth sensing systems;

[0025] Figure 3 and Figure 4 Also shown are the scanning areas of various prior art depth sensing systems;

[0026] Figures 5 to 9 is a detailed view of line segments and patterns projected by a depth sensing system according to various embodiments;

[0027] Fig.10 is a detailed schematic diagram of a depth sensing system according to one embodiment;

[0028] Fig.11 is in use Fig.10 A detailed perspective view of the depth sensing system shown;

[0029] Fig.12 is a detailed schematic diagram of a depth sensing system according to another embodiment;

[0030] Fig.13 is a detailed schematic diagram of a prior art depth sensing system;

[0031] Fig.14 and Fig.15are detailed schematic diagrams of depth sensing systems according to two embodiments;

[0032] Figures 16 to 18 is a detailed schematic diagram of a light sensor for a depth sensing system according to various embodiments;

[0033] Fig.19 and Fig.21 A flow chart of an image processing method according to two embodiments is shown;

[0034] Fig. 20 is a detailed schematic diagram of a light sensor for a depth sensing system according to another embodiment. DETAILED DESCRIPTION

[0035] Various embodiments of the present invention are directed, in a single embodiment or in multiple embodiments, to systems, methods and articles of manufacture for a depth sensing system. Other objects, features and advantages of the present invention are described in the detailed description, drawings and claims.

[0036] Various embodiments will now be described in detail with reference to the accompanying drawings, which are provided as illustrative examples of the present invention so that those skilled in the art can practice the present invention. It is noteworthy that the following figures and examples are not meant to limit the scope of the present invention. In the case where known components (or methods or processes) can be used to partially or completely implement certain elements of the present invention, only those parts necessary for understanding the present invention will be described in such known components (or methods or processes), and detailed descriptions of other parts in such known components (or methods or processes) will be omitted to avoid blurring the present invention. In addition, various embodiments include equivalents known to the present and future of the components mentioned in the figures herein by way of example.

[0037] The depth sensing system can be implemented independently of the mixed reality system, but many of the embodiments below are described with respect to an AR system for illustrative purposes only.

[0038] Overview of Problems and Solutions

[0039] There are various methods for optically estimating or sensing the depth of a surface (i.e., the distance between a known point in 3D space and a point on the surface). Depth sensing methods can be divided into two main modes: passive and active. Passive systems detect ambient light from a light source external to the system (e.g., an overhead light or the sun) that reflects off the surface. Active systems project light onto the surface and detect the projected light that reflects off the surface.

[0040] Passive systems typically use two images captured at different locations and separated by a known distance to determine depth. Some passive systems use multiple cameras (such as in a binocular configuration) to capture the two images. Other passive systems use the same sensor to capture two images at different times and locations. After the two images have been captured at different locations, the system processes the images to match POIs in one image with corresponding POIs in the other image. The system then triangulates the angle between the optical axes of the two images and a single POI on the surface, as well as the distance between the two image capture locations, to determine the POI location in 3D space relative to the two image capture locations known to the system.

[0041] Passive systems can determine the location of POIs in 3D space, but have multiple failure modes, including a lack of suitable multiple POIs that can be identified in one image ("identification problem") and matched to their counterparts in other images ("correspondence problem"). An example of such a failure mode is imaging a blank wall, where it is very difficult to (1) identify points on the wall in one image and (2) identify corresponding points in another image. Similarly, in the case of a dark room, there is simply not enough ambient light to clearly see the objects, so identifying and matching them is also very difficult.

[0042] Some active systems address both of these problems (i.e., lack of light and lack of discernible features) by projecting light with patterns and textures (e.g., light from a kaleidoscope). Such systems illuminate a surface and project a pattern on a uniform surface (e.g., a white wall). When two images are captured using such an active system with a static illumination pattern (by simultaneously capturing or two cameras with static projection), it is much simpler to match the pattern (or portion thereof) on the surface from one image to the other. Therefore, correspondingly, it is simpler to triangulate the positions of multiple POIs in 3D space. In fact, using advanced systems with tight mechanical tolerances, a single image captured by a single camera can be used to determine the positions of multiple POIs, because by triangulating the angles at which the reflections are observed, as well as the positions of the camera and light projector, the POI positions in 3D space can be calculated.

[0043] However, even in active systems, the use of visible light is often sub-preferred because it can be distracting or disorienting to the user or others near the surface. Some active systems address this problem by projecting infrared ("IR") light, which modern primitive camera sensors can detect (e.g., near-infrared photons) but is invisible to the human eye.

[0044] Even with an IR-active system, other patterns in the scene may still interfere with depth sensing or cause other depth sensing issues. Other systems include wavelength filters placed on the sensor (which can inherently sense both visible and IR light) so that visible light does not reach the sensor. Adding the filter results in detecting an IR-only image, which is typically illuminated only by the system (IR) projector, the sun, and a few other IR light sources (e.g., warm lights and remote controls).

[0045] In order to obtain the most accurate (i.e., closest to the true value) and most precise (i.e., reproducible) depth information, a very high resolution sensor is required. A greater number of sensor pixels over the viewing area results in a lower angular resolution per sensor pixel, effectively providing a higher precision angular input to the triangulation math. As used in this application, a "sensor pixel" includes, but is not limited to, an identifiable point on a sensor used for light intensity measurement.

[0046] In addition to sensors / sensor pixels that detect visible light in systems that include outward-facing cameras, infrared-active systems require sensors / sensor pixels that detect infrared light. Problems with this approach include: (1) reduced visible light resolution in systems that include hybrid red / green / blue / infrared sensors, and (2) coordinate registration in systems that include separate visible light sensors and infrared sensors. As described below, embodiments disclosed herein address these and other sensor issues by using an improved image sensor with enhanced depth sensing.

[0047] Without additional processing, the maximum angular accuracy of the POI is achieved with the angular resolution of a single sensor pixel. However, image processing algorithms can provide "sub-pixel resolution". For example, the system may observe some features (e.g., the line that makes up the edge of a table corner) at the processing of several sensor pixels and reconstruct the equation of the line with higher accuracy than a single sensor pixel.

[0048] Depth sensing methods are limited by the Nyquist limit of the system because they are signal processing methods. Therefore, reconstructing a signal requires a minimum number of sampling points (and point density) based on the frequency of the signal. Therefore, noisier signals are more difficult to sample and reconstruct than "simpler" (lower bandwidth) signals. As described below, the embodiments disclosed herein address Nyquist limit related issues and other projector related issues by using a non-aliasing pattern projector / generator to reduce the high frequency noise of the projected light.

[0049] Dynamic non-aliasing pattern projector / generator

[0050] A fiber scanning projector ("FSP") projects light by selectively vibrating the tip of an optical fiber at a high frequency. FSPs are also referred to as scanning fiber displays ("FSDs"), fiber scanning displays ("FSDs"). Multiple FSPs can project dynamic images as displays for human observation. Their very small size and low potential power consumption are ideal for certain applications such as mixed reality systems. Exemplary multiple FSPs are described in U.S. utility patent application serial number 14 / 738,877 (attorney docket number ML.20019.00) and U.S. utility patent application serial number 14 / 738,877 (attorney docket number ML.20011.00), the contents of which have been previously incorporated herein by reference.

[0051] In contrast to typical "panel" type displays (e.g., LCOS or DLP), the embodiments disclosed herein describe the use of FSP to project light with texture, pattern, or structure for depth reconstruction. The vast majority of existing active depth sensing systems use photon sources such as LCOS and DLP that project light with pixelated patterns (e.g., squares or rectangles). These systems illuminate the entire frame at the same time, and the projected image consists of multiple straight lines (or fixed shape) of display pixels projected at once. As a result, these images consist of multiple repeating (generally straight / square) shapes rather than pure lines.

[0052] Figure 2 A "line" 200 is depicted projected by a panel-type display at a magnification sufficient to show defects in the projected line 200. At this magnification, the projected line 200 appears as a set of steps formed by a series of right angles. However, at lower magnifications, the projected line 200 appears linear to the viewer. Each right-angle arm is formed by a series of linear display pixels.

[0053] Due to the irregularity of projection line 200, a sensor of comparable (or higher) resolution that detects projection line 200 will have sufficient sensitivity to observe high frequency noise in the projection pattern, complicating the image reconstruction effort. Figure 3 Depicted is a square scan area 202 with overlap from a sensor having a low resolution of the projector. Figure 2 The projection line is 200. Figure 3 As shown, the scan area 202 along the projection lines 200 does not contain the same number of projection lines 200. Therefore, the signal from the sensor is very noisy, resulting in aliasing, i.e., broad grey lines instead of narrow black lines.

[0054] This noise signal problem complicates the task of locating the intersection 204 of the two projection lines 200a, 200b, as shown in FIG. Figure 4As shown, this intersection 204 can be a POI for a depth sensing system. However, high frequency noise in the projected lines 200a, 200b causes the intersection 204 to be a line segment rather than a point. In addition, the scan area 202 cannot resolve the lines 200a, 200b in sufficient detail to identify the intersection 204 with high accuracy.

[0055] Existing systems reconstruct this noisy signal into a sub-pixel (or even at-pixel) accurate map either by using a projector with a much higher resolution than the corresponding sensor, or by using additional signal processing algorithms.

[0056] Embodiments disclosed herein describe the use of FSPs to project light to produce higher quality dynamic patterns to facilitate image reconstruction. When multiple FSPs are used, light is projected by a single traveling beam (e.g., a laser). The beam is mechanically scanned across the scene at a relatively high frequency. Instead of projecting light onto essentially every pixel of a scene (as with a panel-type display), multiple FSPs create optically distinct patterns by projecting a beam pattern with a narrow dispersion angle onto a surface, so that the projected light minimizes aliasing (e.g., image distortion or aliasing) and high-frequency noise that interfere with reconstruction efforts.

[0057] However, panel type displays have a fixed illumination pattern, multiple FSPs are dynamic. While the effective scanning frequency of the FSP may only allow (for example) the X and Y travel paths to be 200 apart, the phase offset, illumination pattern and scanning frequency can be varied, allowing for the provision of clear non-aliased edges, intersections, and therefore easy identification of dynamic patterns of multiple POIs without the need for an extremely high resolution projector.

[0058] For example, Figure 5 A pattern 300 projected by a FSP is depicted. The pattern is made by modulating the FSP (e.g., using an actuator) to "draw" two instances / channels of sine waves 302a, 302b that are phase offset. Thus, the first sine wave 302a and the second sine wave 302b periodically intersect to form a crossover 304. The FSP may project light from a tight beam light source (e.g., a laser) to form Figure 5 The depicted pattern 300. Thus, the signal from the light has minimal high frequency noise and negligible aliasing. The crossovers 304 form a particularly desirable plurality of POIs because crossovers are more accurately and precisely identified by image analysis than points, which increase in diameter as the light propagates, requiring an estimate of the center of the point. Another FSP pattern that forms discrete crossovers is the Lissajous pattern.

[0059] Figure 6 Another pattern 300' projected by a FSP is depicted. Figure 6The depicted pattern 300' and Figure 5 The pattern 300 depicted and described above is similar. In fact, by modulating the FSP to form Figure 6 The pattern 300' is depicted to form Figure 5 The first sine wave 302a and the second sine wave 302b are identically shown. However, the FSP is further modulated by activating the light sources therein only when the first sine wave 302a and the second sine wave 302b intersect to form an intersection 304. Projecting this pattern 300' with the FSP results in the following: (1) more distinct and recognizable intersections 304 (multiple POIs), and (2) reduced system energy usage by deactivating the light sources.

[0060] Figure 7 Another pattern 300" projected by the FSP is depicted. Figure 7 The depicted pattern 300" and Figure 6 The pattern 300' depicted and described above is almost identical to Figure 6 Compared with the first sine wave 302a and the second sine wave 302b, Figure 7 The first sine wave 302a' and the second sine wave 302b' are shown to be phase shifted (by the same amount). Figure 6 Compared to the intersection 304 formed by the first sine wave 302a and the second sine wave 302b, Figure 7 The intersection 304' shown formed by the first sine wave 302a' and the second sine wave 302b' is also phase shifted.

[0061] Figure 8 Depicted is the displacement of crossover 304' compared to crossover 304, which is Figure 7 and the first and second sine waves 302a, 302b shown in shaded form and the crossover 304 (from Figure 6 ). Figure 6 The depicted pattern 300'( Figure 8 The shaded Figure 7 The pattern depicted is 300" ( Figure 8 The time-sequential switching between the intersections 304 / 304' (shown in solid lines) makes the intersections 304 / 304' appear to move. This movement results in a dynamic pattern with more distinct and identifiable intersections 304 / 304' (multiple POIs).

[0062] Fig. 9 Another pattern 300" projected by the FSP is depicted. Fig. 9 The depicted pattern 300"' and Figure 6 Similar to the pattern 300 depicted and described above. Figure 6 Compared with the first sine wave 302a and the second sine wave 302b, Fig. 9The first sine wave 302a" and the second sine wave 302b" shown are modified. Therefore, Figure 6 Compared to the intersection 304 formed by the first sine wave 302a and the second sine wave 302b, Fig. 9 The intersection 304" formed by the first sine wave 302a" and the second sine wave 302b" is shown to have a modified shape. Fig. 9 The shape of the cross 304 is "+", and Figure 6 The shape of the intersection 304 in is "×". However, Figure 6 and Fig. 9 The positions of the crossovers 304, 304" are the same. Therefore, Figure 6 The depicted pattern 300' and Fig. 9 The time-sequential switching between the depicted patterns 300'" causes the intersections 304 / 304" to appear to change shape (between "x" and "+"). This shape change results in a dynamic pattern with more distinct and identifiable intersections 304 / 304" (multiple POIs).

[0063] Figures 5 to 9 The patterns 300, 300', 300", 300"' depicted in the figure depict the same change for each intersection 304, 304', 304" (POI) in the pattern 300, 300', 300", 300"'. In other embodiments, subsets of the intersections 304, 304', 304" may be changed (e.g., position, shape, wavelength, etc.). In still other embodiments, various subsets of the intersections 304, 304', 304" may have different changes. For example, only one or more intersections 304, 304', 304" used for depth sensing may be changed. In other embodiments, the number of intersections 304, 304', 304" may dynamically change from dense to sparse. In still other embodiments, the light source (e.g., a laser) may be pulsed to dynamically change the pattern 300, 300', 300", 300'".

[0064] Fig.10 An active depth sensing system 400 is depicted that is capable of projecting the patterns 300, 300', 300" described above according to one embodiment. The system 400 includes a spatially modulated light projection device 402 (e.g., FSP), two light sensors 404a, 404b (e.g., cameras), and a processor 406 that is operably coupled to the other components 402, 404a, 404b. The spatially modulated light projection device 402 (e.g., FSP), the light sensors 404a, 404b (e.g., cameras), and the processor 406 may be coupled by a bus (not shown) in the system 400. Alternatively, some or all of these components 402, 404a, 404b, 406 may be coupled to each other by a network (e.g., a wireless network).

[0065] Fig.11 Depicts the use of Fig.10 Active depth sensing system 400 is depicted. A spatially modulated light projection device 402 is modulated to project a pattern 408 (e.g., +) onto a substantially uniform surface 410 (e.g., a white wall). Pattern 408 can be used as a POI to determine a distance Y between the light projection device 402 and the surface 410. The light projection device 402 projects light 412 onto the surface 410 to form the pattern 408. Reflected light 412', 412" is detected from a first light sensor 404a and a second light sensor 404b.

[0066] The system 400 measures the angles α, β by which the pattern 408 moves from the respective optical axes 414a, 414b of the first and second optical sensors 404a, 404b. Using one of the measured angle α and the distance X1 separating the light projection device 402 from the respective first optical sensor 404a, and the measured angle β and the known distance X2 separating the light projection device 402 from the respective second optical sensor 404b, the system 400 can calculate the distance Y between the light projection device 402 and the surface 410. With the measured angles α, β, the system 400 can provide a more accurate and precisely calculated distance Y.

[0067] Fig.12 An active depth sensing system 400 is depicted according to another embodiment. Fig.12 The depicted system 400 is similar to Fig.10 and Fig.11 The system depicted is similar. In addition to the spatially modulated light projection device 402, two light sensors 404a, 404b and a processor 406, Fig.12 The depicted system 400 also includes an actuator 416 that is used to modulate the light projection device 402 . Fig.12 The processor 406 of the depicted system 400 includes a pattern designer 418 and a pattern detector 420 running thereon. The pattern designer 418 generates a pattern and sends data specifying the generated pattern to the actuator 416, which modulates the light projection device 402 to display the generated pattern. The pattern detector 420 receives optical data from the first light sensor 404a and the second light sensor 404b and extracts information about the pattern from the received optical data.

[0068] Furthermore, because the FSP merely acts as a spatially modulated conduit for the light beams, light beams with different wavelengths can be passed through the FSP simultaneously. This allows for the transmission of not only invisible infrared light, but also visible light of various colors. Using multiple beams allows the FSP to enhance the infrared pattern with a visible pattern, allowing correspondence between camera sensors that might not normally be seen at similar wavelengths. This can help align the camera sensors to a common coordinate system, provide additional depth reconstruction information (allowing features from one sensor to help provide complementary information to another sensor). Such a system can also perform other functions such as auxiliary lighting to indicate targets, areas for focus, warnings, etc.

[0069] Active depth sensing systems including FSPs have the following characteristics. Multiple FSPs project distinct and dynamic (i.e., time-varying) patterns, providing supplemental or improved information for depth determination over time. In addition, multiple FSPs can adjust energy emission to a focal area, thereby reducing power, and increase energy delivered to a predetermined area to overcome high-frequency noise. Multiple FSPs also have a minimal amount of high-frequency image noise, thereby simplifying depth determination calculations. In addition, multiple FSPs are capable of simultaneously projecting light from two light sources (e.g., infrared and invisible light sources).

[0070] In addition, although the active depth sensing system described herein includes multiple FSPs, other spatially modulated light projection devices may also be used for the active depth sensing system while maintaining desired system characteristics. For example, the active depth sensing system may include a microelectromechanical system ("MEMS") mirror scanner and a laser source. Like the system including multiple FSPs, the system including the MEMS mirror scanner can project and scan a light beam pattern on a surface. In other embodiments, the system can also project and scan the pattern to form a portion of a computer-generated hologram with variable illumination. All spatially modulated light projection devices (such as multiple FSPs and multiple MEMS mirror scanners) project "beams" or "arcs of light" rather than display pixels, and can cause the path and timing of those light beams to vary.

[0071] Image sensor with enhanced depth sensing

[0072] Both passive and active depth sensing systems include at least one sensor (e.g., a camera) that detects light reflected from a surface. As mentioned above, some detect visible light (e.g., ambient light), while other depth sensing systems detect projected light patterns (e.g., projected infrared light).

[0073] Existing depth sensing systems (passive and active) typically use two camera sensors to detect visible light and projected infrared light. These camera sensors are each associated with a different filter, so that one camera sensor detects visible light (possibly with an RGB color filter) and the other camera sensor detects infrared light (by filtering out the visible light). Fig.13 The sensor arrangement is depicted, Fig.13 A passive depth sensing system 500 is depicted that includes a visible light sensor 502 and an infrared light sensor 504, both of which are operably coupled to a processor 506. The infrared light sensor 504 includes a light cutoff filter 508 that blocks all light except infrared light from reaching the infrared light sensor 504. Before data from the visible light and infrared light sensors 502, 504 can be used together, the coordinate systems of the sensors 502, 504 must be registered. Registration of the coordinate systems is particularly difficult when the depth sensing system 500 is moving relative to a surface.

[0074] The embodiments disclosed herein describe a hybrid visible light / full spectrum light sensor for a depth sensing system. As used in this application, "full spectrum light" includes visible light and infrared light (wavelengths from about 390 nm to about 1 mm). For example, Fig.14 A passive depth sensing system 600 is depicted according to one embodiment. The passive depth sensing system 600 includes a hybrid visible light / full spectrum light sensor 602 that is operably coupled to a processor 604. As described in detail below, the visible light / full spectrum light sensor 602 includes a hybrid filter 606. The hybrid filter 606 allows only visible light to reach some portions (i.e., sensor pixels) of the visible light / full spectrum light sensor 602, but allows full spectrum light to reach other portions (i.e., sensor pixels) of the visible light / full spectrum light sensor 602.

[0075] Fig.15 6 is a passive depth sensing system 600 according to another embodiment. The passive depth sensing system 600 includes a hybrid visible light / full spectrum light sensor 602 operably coupled to a processor 604. As described in detail below, the visible light / full spectrum light sensor 602 includes a hybrid filter 606. The processor 604 includes an image processor 608 running thereon.

[0076] Fig.16 Schematically depicts the Fig.14 and Fig.15 A portion of a hybrid visible light / full spectrum light sensor 602 and its hybrid optical filter 606 of a passive depth sensing system 600 as depicted is a passive depth sensing system. Fig.16 The depicted portion of the hybrid visible light / full spectrum light sensor 602 includes five sensor pixels 610. The underlying sensor pixels 610 are identical in that they can each detect full spectrum light, i.e., visible light (wavelengths from about 390 nm to about 700 nm) and infrared light (wavelengths from about 700 nm to about 1 mm). However, the light reaching some of the sensor pixels 610 is modified by filters 606 disposed between the light source (e.g., a reflective surface) and the sensor 602.

[0077] like Fig.16 As shown, filter 606 includes sub-filters or "caps" F1 and V1-V4 disposed on corresponding sensor pixels 610. The full-spectrum light ("F-type") cap allows full-spectrum light to reach sensor pixels 610 located below the F-type cap. In fact, the F-type cap may not have any filtering capabilities at all. The visible light ("V-type") cap only allows visible light to reach sensor pixels 610 located below the V-type cap. Sensor pixel / cap 610F1 is surrounded by four sensor pixels / caps 610V1-610V4 in a base position, forming a "+" sign with sensor pixel / cap 610F1 located in the middle. Therefore, hybrid visible light / full-spectrum light sensor 602 includes a full-spectrum light ("F-type") sensor pixel surrounded by four visible light ("V-type") sensor pixels in a base position relative to the full-spectrum light sensor pixel. Because the visible and full-spectrum light sensor pixels are combined in one sensor, and because of the arrangement of these sensor pixels, the hybrid visible light / full-spectrum light sensor 602 is different from other sensors.

[0078] Although Fig.16 The depicted sensor 602 and filter 606 include five sensor pixels 610 arranged in a cross configuration, but other embodiments include configurations with different numbers of pixels and / or different shapes.The sensor data analysis methods described herein may be modified to analyze data from sensors and filters in different configurations.

[0079] Fig.17 and Fig.18 Schematically depicted Fig.16 A larger portion of a hybrid visible light / full spectrum light sensor 602 and its hybrid filter 606 are depicted. Fig.17 and 18 As shown, the V-type and F-type sensor pixels are arranged in an alternating pattern such that for each sensor pixel (V-type or F-type), all four substantially adjacent regions are of the complementary type.

[0080] In such a configuration, for a hybrid visible light / full spectrum light sensor 602 having P, e.g., (640x480=307,200) total sensor pixels, half of the sensor pixels will be of type V and half will be of type F. When each type of sensor pixel is considered individually without further image processing, this sensor pixel arrangement results in a reduced effective resolution for the sensor pixel type and application. Increasing the overall resolution sensor to compensate for this problem would increase power consumption, sensor size, and other system costs.

[0081] A depth sensing system 600 including a hybrid visible light / full spectrum light sensor 602 according to various embodiments is provided by image processing (using Fig.15 The image processor 608 depicted compensates for the reduced effective resolution. While these methods may not be able to recover 100% of the resolution of a separate visible or infrared sensor of comparable size, these methods can recover the higher resolution that would be expected from the actual sensor pixel resolution of the hybrid visible / full spectrum light sensor 602 (e.g., 50% visible sensor pixels and 50% full spectrum sensor pixels). Embodiments of image processing methods used with a hybrid visible / full spectrum light sensor can recover the resolution of a separate sensor from approximately 50% to approximately 100%. Other embodiments can recover approximately 66% of the resolution to approximately 90% of the resolution. Still other embodiments can recover approximately 75% of the resolution.

[0082] The light used in depth sensing system 600 includes wavelengths that overlap in the visible spectrum because all sensor pixels 610 detect visible light. F-type sensor pixels 610F also detect infrared light. Figures 16 to 18 In the illustrated embodiment, each sensor pixel 610 has eight adjacent neighboring regions, four of each type. Fig.17 As shown, 610F1 has four adjacent V-type sensor pixels 610V1-610V4 in the cardinal direction. 610F1 also has four adjacent F-type sensor pixels 610F2-610F5 between the cardinal directions.

[0083] The following is a general description of an image processing method according to one embodiment. The F-values ​​of all F-type sensor pixels 610F can initially be approximated as consisting of two values ​​- a calculated value for visible light ("CV") and a calculated value for infrared light ("CI"). In a first pass, the measured value of visible light ("V") for a V-type sensor pixel 610V adjacent to the F-type sensor pixel 610F is used to estimate a first pass CV for the F-type sensor pixel 610F. The first pass CV is then used to estimate a first pass CI for the F-type sensor pixel 610F. In a second pass, the first pass CI is used to (more accurately) estimate a second pass CV for the F-type sensor pixel 610F.

[0084] For surfaces with substantially uniform or well-known multiple CIs, the second round of CVs can be estimated by subtracting the first round of CIs from F for F-type sensor pixel 610F, so that information provided from V-type sensor pixel 610V alone is not available. In an indoor room with no additional infrared light sources, and any infrared light projectors in the system are disabled, surfaces with substantially uniform or well-known multiple CIs can be found. Similarly, in outdoor environments, ambient daylight typically provides primarily diffuse illumination on each surface (based on the infrared reflectance of that surface), and therefore, the infrared illumination for each surface is primarily uniform or predictable.

[0085] In the case of a modulated infrared projector (as described above), information about the projected infrared pattern can be used to calculate an effective resolution of the visible light image that is higher than the V-shaped sensor pixels 610V (e.g., P / 2) in sensor 602. Information about dynamically changing projected infrared patterns in certain areas of a surface (as described above) can also be used to calculate an increased effective visible light image resolution. In addition, when all infrared light sources are eliminated from the environment (e.g., indoors with no infrared sources), sensor 602 can be used as a full-resolution visible light sensor.

[0086] Other methods according to various embodiments may include more than two rounds to increase the accuracy of estimating CV and CI. Still other embodiments use calculated values ​​of other measurements to estimate CV and CI for F-type sensor pixel 610F. For example, using Fig.17 The data acquired by the depicted hybrid visible light / full spectrum light sensor 602, in addition to the V of 610V1-610V4, can be used to estimate the CV and CI of 610F1 using the CV and CI of 610F2-610F5. Fig.18610F1 is shown surrounded by more F-type and V-type sensor pixels, which can provide more optical data for estimating the CV and CI of 610F1. One factor in the estimation method using all F-type and V-type sensor pixels is that the sum of the Fs for all F-type sensor pixels 610F should be at least as bright as the sum of the Vs for all V-type sensor pixels 610V, if not brighter.

[0087] A similar process can be used to calculate the CI for F-type sensor pixel 610F in sensor 602 and generate an image estimating infrared light only, although there are no dedicated infrared-only pixels. Fig.19 An image processing method 700 is depicted for generating separate visible light and infrared images from optical data obtained by a single hybrid visible light / full spectrum light sensor 602 according to one embodiment.

[0088] At step 702, the depth sensing system 600 receives light reflected from a surface. The hybrid visible light / full spectrum light sensor 602 simultaneously receives visible light at each V-type sensor pixel 610V and full spectrum light at each F-type sensor pixel 610F.

[0089] At step 704, the depth sensing system 600 determines a visible light value ("V") for each V-type sensor pixel 610V based on the visible light received by the pixel 610V. At step 704, the depth sensing system 600 also determines a full spectrum light value ("F") for each F-type sensor pixel 610F based on the full spectrum light received by the pixel 610V.

[0090] At step 706, the depth sensing system 600 (e.g., the image processor 608) calculates a calculated value ("CV") of visible light for each F-type sensor pixel 610F. The image processor 608 may calculate the CV using the V for the V-type sensor pixel 610V adjacent to each F-type sensor pixel 610F. In a simple embodiment, the multiple Vs of the four V-type sensor pixels 610V adjacent to the F-type sensor pixel 610F are averaged to generate the CV. For example, Fig.16 In the depicted hybrid visible light / full spectrum light sensor 602, multiple Vs of 610V1-610V4 are averaged to generate CV for 610F1. This embodiment is most accurate for uniform surfaces (eg, a white wall), but the accuracy decreases as the surface becomes more heterogeneous.

[0091] In another embodiment, edge detection and gradient detection can be performed on adjacent regions of the opposite surface to determine a more accurate CV by taking into account non-uniform surfaces including edges. Fig.16In the depicted hybrid visible light / full spectrum light sensor 602, edge detection and gradient detection may be performed on the multiple Vs of 610V1 / 610V3 and 610V2 / 610V4. If a large gradient is found between the multiple Vs of 610V1 / 610V3, those multiple Vs may be given less weight when generating the CV for 610F1. This embodiment is more accurate for heterogeneous surfaces (e.g., corners including edges).

[0092] At step 708, the depth sensing system 600 (e.g., the image processor 608) calculates a calculated value ("CI") of infrared light for each F-type sensor pixel 610F. The image processor 608 may calculate the CI using the CV for the F-type sensor pixel 610F. In a simple embodiment, the CI is calculated by subtracting the CV from the F for the F-type sensor pixel 610F.

[0093] At step 710, the depth sensing system 600 (e.g., image processor 608) generates a visible light image using the plurality of Vs from the V-type sensor pixels 610V and the calculated plurality of CVs for the F-type sensor pixels 610F. At step 712, the depth sensing system 600 (e.g., image processor 608) generates an infrared light image using the calculated plurality of CIs for the F-type sensor pixels 610F. The visible light image, the infrared light image, or both can be used for depth sensing.

[0094] Additionally, a demosaicing and / or edge sharpening algorithm may optionally be applied to the image data to resolve visible light values ​​for the V-type sensor pixels 610V and full spectrum light values ​​for the F-type sensor pixels 610F prior to step 704. Resolving these light values ​​V, F increases the accuracy of method 700.

[0095] The system 600 described herein uses optical data from a single hybrid visible light / full spectrum light sensor 602 to generate visible light images with an effective resolution higher than P / 2, and similarly, infrared light images are generated on the same physical sensor 602 at a similar lower resolution. Therefore, the light used to generate the visible light image and the infrared light image will pass through the same lens stack, so any minor defects are reflected in both the visible light image and the infrared light image. In addition, this arrangement has two advantages over existing systems. First, the visible light image and the infrared light image will be acquired from exactly the same optical vantage point, providing a perfect 6-degree of freedom (6-DOF) correspondence for the focus of the corresponding images. This eliminates the need for registration and calibration, which is a particularly difficult problem for systems including separate cameras operating on different wavelength spectra because the cameras may not be able to detect the same calibration target. In addition, registration and calibration introduce additional error sources that reduce the sub-pixel accuracy of any post-operation performed on the image data. Second, the exact same sensor is used, so the exposure times on the two images are completely synchronized. For images of surfaces in relative motion (of the sensor, the surface, or both), the visible light and infrared images are matched temporally (even on the order of microseconds) and geometrically, allowing for more precise and detailed analysis of the images.

[0096] exist Fig. 20 In another depicted embodiment, the hybrid visible light / full spectrum light sensor 802 includes two types of sensor pixels 810, type A and type B. Type A sensor pixels 810A and type B sensor pixels 810B both detect visible light and infrared light, but in different proportions. In one embodiment, type A sensor pixel 810A can detect 75% of visible light ("pV_A") and 25% of infrared light ("pI_A") that impinges on pixel 810A. In this embodiment, type B sensor pixel 810B can detect 60% of visible light ("pV_B") and 40% of infrared light ("pI_B") that impinges on pixel 810A. Although the pV and pI components in this embodiment add up to 100%, in other embodiments, the pV and pI components may add up to more or less than 100%. For example, Figures 16 to 18 An F-type (full spectrum) sensor pixel 610F in a hybrid visible light / full spectrum light sensor 602 is depicted and described above as having pV=100% and pI=100%.

[0097] In such a sensor 802, each sensor pixel 810 has a sensed value ("SV") corresponding to the visible and infrared light detected. Because the SV for a particular A-type sensor pixel 810A ("SV_A") is composed of two contributors (i.e., a total visible light value "V" and a total infrared light value "I" modified by the proportion of each type of light detected by the sensor pixel 810A), we know that SV_A = (V*pV_A) + (I*pI_A).

[0098] Each A-type sensor pixel 810A also has an estimated value ("EV_A") calculated using optical data from neighboring sensor pixels 810 (e.g., a substantially neighboring area). For example, 810A1 has an EV calculated based on the SVs for 810B1-810B4. In other words, for 810B1-810B4, EV_A=f((V*pV_B)+(I*pI_B)). The function f can be as simple as taking an average. In other embodiments, the function f can include edge detection and gradient detection as described above.

[0099] SV_A is determined from the sensor pixel 810A, and EV_A is estimated. pV_A, pI_A, pV_B, pI_B are known from the design of the sensor 802. Using these determined, estimated, and known values, the two equations SV_A = (V*pV_A) + (I*pI_A) and EV_A = f((V*pV_B) + (I*pI_B)) can be solved for V and I for each A-type sensor pixel 810A. A similar process can be used to determine V and I for each B-type sensor pixel 810A.

[0100] Fig.21 An image processing method 900 is depicted for generating separate visible and infrared images from optical data acquired by a single hybrid visible light / full spectrum light sensor 802 according to one embodiment.

[0101] At step 902, the depth sensing system receives light reflected from a surface. The hybrid visible light / full spectrum light sensor 802 receives full spectrum light simultaneously at each sensor pixel 810 (both A-type and B-type), where the percentage of visible light and infrared light detected depends on the pV and pI of each A-type and B-type sensor pixel 810.

[0102] At step 904 , the depth sensing system determines a sensed light value (“SV”) for each sensor pixel 810 .

[0103] At step 906, the depth sensing system calculates the total visible light value ("V") and the total infrared light value ("I") for each sensor pixel 810. For example, the depth sensing system can calculate V and I for each sensor pixel by simultaneously solving the above pair of equations (i.e., SV_A = (V*pV_A) + (I*pI_A) and EV_A = f((V*pV_B) + (I*pI_B))) using the known pV and pI values ​​and the detected SV value and estimated EV value as described above.

[0104] At step 908, the depth sensing system generates a visible light image using the calculated multiple Vs for sensor pixels 810. At step 910, the depth sensing system generates an infrared light image using the calculated multiple Is for sensor pixels 810. The visible light image, the infrared light image, or both can be used for depth sensing.

[0105] Although Fig.21 The depicted method 900 generates a visible light image first and then an infrared image, but in other embodiments, the depth sensing system may generate an infrared image first and then a visible light image. In still other embodiments, the depth sensing system may generate either a visible light image or an infrared image, but not the other.

[0106] Although in some embodiments, the dynamic non-aliasing pattern projector with enhanced depth sensing and the image sensor are described as part of one system, the projector and the sensor are independent and each can achieve all of the advantages described above without the other.

[0107] Although the above-mentioned sensors are described as depth sensing, the sensors according to the embodiments can be used for other visible light / infrared light systems, such as camera focusing systems. The above-mentioned depth sensing systems are provided as examples of various optical systems that can benefit from hybrid sensors. Therefore, the use of the optical systems described herein is not limited to the disclosed depth sensing systems, but is applicable to any optical system.

[0108] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. These examples are provided to illustrate the more widely applicable aspects of the present invention. Various changes may be made to the described invention and may be replaced with equivalents without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt specific situations, materials, material compositions, processes, multiple process actions or multiple steps to multiple goals, spirits or scopes of the present invention. In addition, as will be appreciated by those skilled in the art, each individual variation described and illustrated herein has discrete components and features that may be easily separated or combined with any features of other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to fall within the scope of the claims associated with the present disclosure.

[0109] The present invention includes methods that can be performed using the subject equipment. The methods may include the action of providing such suitable equipment. This provision may be performed by the end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set, activate, power on or other actions to provide the necessary equipment in the subject method. The methods described herein may be implemented in any order of the events that are logically possible and in the order of the events.

[0110] Exemplary aspects of the present invention and details about material selection and manufacturing have been set forth above. As for other details of the present invention, these other details can be understood in conjunction with the patents and publications cited above and what is generally known or understood by those skilled in the art. For method-based aspects of the present invention, the same applies in terms of additional actions that are generally or logically employed.

[0111] In addition, although the present invention has been described with reference to several examples that optionally include various features, the present invention is not limited to being described or indicated as expected with respect to each variation of the present invention. Various changes may be made to the described invention without departing from the true spirit and scope of the present invention, and equivalents (whether described herein or not included for the sake of brevity) may be substituted. In addition, where a range of values ​​is provided, it should be understood that each intermediate value between the upper and lower limits of the range and any other described values ​​or intermediate values ​​within the range are included within the present invention.

[0112] In addition, it is contemplated that any optional features of the described inventive variations may be set forth and claimed independently or in combination with any one or more features described herein. References to singular items include the possibility of multiple identical items. More specifically, unless otherwise specifically stated, when used in this document and in the claims associated with this document, the singular forms "one", "an", "said", and "the" include plural referents. In other words, the use of the article allows for "at least one" in the subject items in the above description and the claims associated with this disclosure. It should also be noted that such claims may be written to exclude any optional elements. Therefore, this statement is intended to serve as a prior basis for the use of exclusive terms such as "alone", "only", etc. associated with the narration of the claim elements or the use of "negative" limitations.

[0113] Without using such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims, or the addition of features may be considered to change the nature of the elements recited in such claims. Unless specifically defined herein, all technical and scientific terms used herein are given as broad a commonly understood meaning as possible while maintaining the validity of the claims.

[0114] The breadth of the present invention is not limited to the examples provided and / or subject specification, but is limited only by the scope of the claim language associated with this disclosure.

[0115] In the foregoing description, the present invention has been described with reference to specific embodiments of the present invention. However, it will be apparent that various modifications and variations may be made to the present invention without departing from the broader spirit and scope of the present invention. For example, the flow of the above-described process is described with reference to a particular sequence of process actions. However, the sequence of many of the described process actions may be changed without affecting the scope or operation of the present invention. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A depth sensing system, comprising: a spatially modulated light projection device for projecting light onto a surface; a sensor for receiving light reflected from the surface; an actuator for controlling spatial modulation of the light projection device to form a first sine wave and a second sine wave that are phase-shifted from each other to form a plurality of intersecting line segments, and receiving light information from the sensor; as well as a processor for analyzing light reflected from the surface to determine a depth of the surface, wherein the light comprises infrared light having a wavelength from 700 nm to 1 mm, and Wherein the actuator controls the spatial modulation of the light projection device to sequentially change a subset of the plurality of intersecting line segments to form a dynamic pattern.

2. The system according to claim 1, wherein: The light includes visible light having a wavelength from 390 nm to 700 nm.

3. The system according to claim 2, wherein: The actuator controls spatial modulation of the light projection device to form a pattern including the infrared light and the visible light.

4. The system according to claim 1, wherein: The actuator controls spatial modulation of the light projecting device to form two intersecting line segments on the surface.

5. The system according to claim 4, wherein: The actuator controls spatial modulation of the light projecting device to form the dynamic pattern including the two intersecting line segments on the surface.

6. The system according to claim 4, wherein: The actuator controls spatial modulation of the light projecting device to form a pattern comprising a plurality of discrete intersecting line segments on the surface.

7. The system according to claim 1, wherein: Controlling the spatial modulation of the light projection device comprises controlling movement of at least a portion of the light projection device.

8. The system according to claim 7, wherein: Controlling spatial modulation of the light projection device includes controlling projection of the light by the light projection device.

9. The system of claim 1, wherein the spatially modulated light projection device comprises a fiber optic scanning display.

10. The system according to claim 1, wherein: Controlling the spatial modulation of the light projection device includes shifting the phase or varying the scanning frequency of the light projection device.

11. The system according to claim 1, wherein: The actuator controls spatial modulation of the light projection device to sequentially change the shapes of the first and second sine waves to form the dynamic pattern.

12. The system of claim 1, wherein: The actuator controls spatial modulation of the light projection device so that the light is sequentially phase shifted to form the dynamic pattern.

13. The system of claim 1, further comprising a light source for generating the light.

14. The system according to claim 13, wherein: The light source is a laser.

15. The system of claim 13, wherein: The actuator also controls the light source.

16. The system of claim 15, wherein: The actuator activates the light source only when the first sine wave and the second sine wave cross each other.

Images