System and method for presenting image content on multiple depth planes by providing multiple intra-pupil parallax views
Patent Information
- Application Number
- JP2024112280
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2016-10-21
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2037-10-20
AI Technical Summary
Existing augmented reality (AR) and virtual reality (VR) technologies struggle to provide a realistic and comfortable three-dimensional viewing experience due to the complexity of the human visual system, often causing discomfort by mismatching accommodation and convergence-divergence motions.
A head-mounted display system that uses a spatial light modulator and projection optics to inject parallax-different intrapupillary images into the viewer's eyes, approximating a continuous wavefront by temporally or spatially multiplexing images, allowing for a range of accommodative responses without requiring a one-to-one correspondence between optical structures and depth planes.
The system provides a more realistic and comfortable three-dimensional viewing experience by accurately matching accommodation and convergence, reducing system bulk and weight, and enabling a wider range of depth planes without the need for multiple dedicated optical structures.
Abstract
Description
[Technical field]
[0001] (Priority Application) This application claims the benefit of priority to U.S. Provisional Application No. 62 / 411,490, filed October 21, 2016, which is incorporated herein by reference.
[0002] (Incorporated by reference) This application incorporates by reference each of the following patent applications in their entirety: U.S. Application No. 14 / 555,585, filed November 27, 2014; U.S. Application No. 14 / 690,401, filed April 18, 2015; U.S. Application No. 14 / 212,961, filed March 14, 2014; U.S. Application No. 14 / 331,218, filed July 14, 2014; U.S. Application No. 15 / 072,290, filed March 16, 2016; and U.S. Provisional Application No. 62 / 156,809, filed May 4, 2015.
[0003] The present disclosure relates to optical devices, including augmented reality and virtual reality imaging and visualization systems. [Background technology]
[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or may be perceived as real. Virtual reality, or "VR", scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, and augmented reality, or "AR", scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality, or "MR", scenarios are a type of AR scenario that typically involve virtual objects that are integrated into and responsive to the natural world. For example, MR scenarios may include AR image content that appears blocked by or is otherwise perceived to interact with objects in the real world.
[0005] 1, an augmented reality scene 10 is depicted in which a user of the AR technology sees a real-world park-like setting 20 featuring people, trees, buildings in the background, and a concrete platform 30. In addition to these items, the user of the AR technology also perceives that they are "seeing" "virtual content," such as a robotic figure 40 standing on the real-world platform 30 and a flying cartoon-like avatar character 50 that appears to be an anthropomorphic bumblebee, but these elements 50, 40 are "virtual" in that they do not exist in the real world. The human visual perception system is complex, making it difficult to produce AR technology that facilitates a pleasing, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements.
[0006] The systems and methods disclosed herein address various challenges associated with AR or VR technology. Summary of the Invention [Means for solving the problem]
[0007] In some embodiments, a head-mounted display system is provided that includes a frame configured to mount on a viewer, a light source, a spatial light modulator configured to modulate light from the light source, and projection optics mounted on the frame and configured to direct light from the spatial light modulator into an eye of the viewer. The display system is configured to display virtual objects on a depth plane by projecting a set of parallax-distinct intra-pupillary images of the objects into the eye.
[0008] In some other embodiments, a method for displaying image content is provided. The method includes providing a spatial light modulator, providing a light source configured to output light to the spatial light modulator from a plurality of different light output locations, and displaying a virtual object on a depth plane by projecting a set of parallax-distinct intra-pupillary images of the virtual object in temporal succession into a viewer's eye. Each intra-pupillary image is formed by outputting light from a light source to the spatial light modulator, the light being output from one or more associated light output locations of the light source, modulating the light with the spatial light modulator to form an intra-pupillary image corresponding to the one or more associated light output locations, and propagating the modulated light to the eye. The one or more associated light output locations for each intra-pupillary image are distinct from the one or more associated light output locations for the rest of the intra-pupillary images.
[0009] In yet another embodiment, a display system is provided that includes a light source with a plurality of spatially distinct light output locations, a spatial light modulator configured to modulate light from the light source, and projection optics mounted on a frame and configured to direct light from the spatial light modulator into a viewer's eye, the display system configured to display a virtual object on a depth plane by projecting a set of parallax-distinct intrapupillary images of the object into the eye in temporal succession.
[0010] In some other embodiments, a method for displaying image content is provided that includes providing a head mounted display comprising a light source and a spatial light modulator, and further including displaying a virtual object on a depth plane by projecting a set of parallax-distinct intra-pupillary images of the virtual object from the display into a viewer's eyes within a flicker fusion threshold.
[0011] In addition, various innovative aspects of the subject matter described in this disclosure may be implemented in the following embodiments. 1. A method for displaying image content, comprising: Providing a spatial light modulator; providing a light source configured to output light to a spatial light modulator from a plurality of different light output locations; displaying the virtual object on the depth plane by projecting a set of parallax-distinct intrapupillary images of the virtual object into the viewer's eyes in time succession, each intrapupillary image comprising: outputting light from a light source to a spatial light modulator, the light being output from one or more associated light output locations of the light source; modulating light with a spatial light modulator to form intra-pupil images corresponding to one or more associated light output locations; Propagating the modulated light to an eye; wherein the one or more associated light output locations for each of the intra-pupillary images are distinct from the one or more associated light output locations for the others of the intra-pupillary images; A method comprising: 2. The method of embodiment 1, wherein the step of activating one or more associated light-emitting regions includes a step of selecting one or more associated light-emitting regions based on a depth plane, and the physical separation between the light-emitting regions for the intra-pupil image increases with decreasing distance from the depth plane to the viewer. 3. The method of any of embodiments 1-2, wherein the light rays forming each of the parallaxly distinct images are collimated and the depth plane is less than optical infinity. 4. A method as described in any of embodiments 1-3, wherein the step of introducing a set of parallaxically distinct intra-pupillary images is performed within a time frame below the viewer's flicker fusion threshold. 5. The method of embodiment 4, wherein the flicker fusion threshold is 1 / 60 of a second. 6. The method further includes an eye tracking sensor configured to track the line of sight of an eye, and the step of displaying the virtual object includes: determining eye gaze using an eye tracking sensor; A method according to any one of embodiments 1-5, comprising the step of selecting content for the intra-pupillary image based on the determined line of sight of the eye. 7. The method of any of embodiments 1-6, further comprising a projection optical system configured to direct modulated light from the spatial light modulator to an eye. 8. The method of any one of embodiments 1-7, wherein one or more associated luminous regions for an intrapupil image partially overlap. 9. The method of any of embodiments 1-8, further comprising a step of varying the position of one or more associated light-emitting regions during the projection of at least one of the intrapupillary images into the eye. 10. A display system configured to implement the method according to any one of embodiments 1-9. 11. A method for displaying image content, comprising: A light source; A spatial light modulator; providing a head mounted display comprising: displaying a virtual object on a depth plane by projecting a set of parallax-distinct intra-pupillary images of the virtual object from the display into the viewer's eyes within a flicker fusion threshold; A method comprising: 12. The method of embodiment 11, wherein the step of injecting a set of parallaxly distinct intra-pupillary images includes a step of injecting individual ones of the intra-pupillary images in temporal succession into the viewer's eye. 13. The method of embodiment 11, wherein the step of introducing a set of parallaxically distinct intra-pupil images includes the step of introducing multiple ones of the intra-pupil images simultaneously. 14. The method of embodiment 13, wherein the step of introducing a set of parallax-distinct intra-pupil images includes introducing multiple intra-pupil images at once in temporal succession. 15. The method of any of embodiments 11-14, wherein the light beam forming the intrapupillary image is collimated. 16. The method of any of embodiments 11-14, wherein the light beam forming the intra-pupillary image has a diverging wavefront. 17. A method as described in any of embodiments 11-16, wherein the light source comprises a plurality of selectively activated light-emitting regions, and the step of introducing a set of parallax-distinct intra-pupil images includes the step of activating a different light-emitting region for each intra-pupil image. 18. The method of any of embodiments 11-17, wherein the light source is configured to output light from a plurality of distinct light output locations, and further comprising a step of jittering the light output locations during the projection of at least one of the intrapupillary images into the eye. 19. A display system configured to implement the method according to any one of embodiments 11-18. 20. A head mounted display system, comprising: a frame configured to mount on a viewer; and A light source; a spatial light modulator configured to modulate light from the light source; a projection optic mounted on the frame and configured to direct light from the spatial light modulator into an eye of a viewer; and wherein the display system is configured to display a virtual object on a depth plane by projecting a set of parallaxically distinct intra-pupillary images of the object into the eye. 21. A display system as described in embodiment 20, wherein the display system is configured to temporally multiplex the display of individual intrapupillary images. 22. A display system according to any of embodiments 20-21, wherein the display system is configured to spatially multiplex the display of intrapupil images. 23. A display system according to any of embodiments 20-22, wherein the display system is configured to temporally multiplex the display of multiple spatially multiplexed intrapupil images. 24. A display system according to any of embodiments 20-23, wherein the projection optics comprises a waveguide comprising an internal coupling optical element and an external coupling optical element. 25. The display system of embodiment 24, wherein the projection optics comprises a plurality of waveguides, each waveguide configured to output light of a different primary color than other waveguides of the plurality of waveguides. 26. The display system of any of embodiments 20-25, wherein the light source comprises a plurality of selectively activated light emitting regions. 27. The display system of embodiment 26, wherein the light source includes at least one of a light emitting diode array and a spatial light modulator. 28. The display system of embodiment 8, wherein the light emitting diode array comprises an organic light emitting diode array or an inorganic light emitting diode array. 29. The display system of embodiment 27, wherein the spatial light modulator light source comprises a liquid crystal array or a digital light processing (DLP) chip. 30. A display system as described in any of embodiments 20-29, wherein the display system is configured to vary the position of the activated light-emitting region during the projection of at least one of the intrapupillary images into the eye. 31. The light source is A light emitter; A display system described in any of embodiments 20-25, comprising an actuator configured to direct light to the spatial light modulator along different paths. 32. A display system as described in embodiment 31, wherein the actuator is a two-axis galvanometer. 33. The display system of embodiment 31, wherein the light source is a fiber scanner. 34. The display system of any of embodiments 20-33, wherein the spatial light modulator configured to modulate light from the light source comprises an LCOS panel. 35. The display system further comprises an eye tracking sensor configured to track the line of sight of an eye. determining a line of sight of the eye using an eye tracking sensor and selecting content for the intra-pupillary image based on the determined line of sight of the eye; A display system as described in any one of embodiments 20-34, configured as follows. 36. A display system according to any of embodiments 20-35, wherein the display system is configured to synchronize the light output location of the light source with the image content provided by the spatial light modulator. 37. A display system according to any of embodiments 20-36, further comprising an optical mechanism between the spatial light modulator and the projection optics, the optical mechanism being configured to direct light from different locations of the spatial light modulator to the projection optics at different angles. 38. The display system of embodiment 37, wherein the optical mechanism comprises one or more of a prism or lens structure. 39. The display system of embodiment 38, wherein the lens structure is a lenslet array. 40. A display system comprising: a light source having a plurality of spatially distinct light output locations; a spatial light modulator configured to modulate light from the light source; a projection optic mounted on the frame and configured to direct light from the spatial light modulator into an eye of a viewer; and wherein the display system is configured to display a virtual object on a depth plane by temporally successively projecting a set of parallax-distinct intra-pupillary images of the object into the eye. 41. A display system as described in embodiment 40, configured to output light from different light output locations of the light source for different intrapupillary images. 42. A display system as described in embodiment 41, configured to vary the lateral separation between light output locations based on the distance of the depth plane from the viewer's eyes. 43. A display system according to any of embodiments 41-42, configured to increase the lateral separation between light output locations with increasing distance of the depth plane from the viewer's eyes. 44. A display system as described in any of embodiments 41-42, wherein the display system is configured to vary the light output location during the projection of at least one of the intrapupillary images into the eye. The present invention provides, for example, the following: (Item 1) 1. A head mounted display system, comprising: a frame configured to mount on a viewer; A light source; a spatial light modulator configured to modulate light from the light source; projection optics mounted on the frame and configured to direct light from the spatial light modulator into an eye of a viewer; Equipped with The display system is configured to display a virtual object on a depth plane by projecting a set of parallaxly distinct intra-pupillary images of the object into the eye. (Item 2) 2. The display system of claim 1, wherein the display system is configured to temporally multiplex the display of individual intrapupillary images. (Item 3) 2. The display system of claim 1, wherein the display system is configured to spatially multiplex the display of the intrapupillary image. (Item 4) 2. The display system of claim 1, wherein the display system is configured to temporally multiplex the display of multiple spatially multiplexed intrapupillary images. (Item 5) Item 6. The display system of item 5, wherein the projection optics comprises a waveguide comprising an in-coupling optical element and an out-coupling optical element. (Item 6) 6. The display system of claim 5, wherein the projection optics comprises a plurality of waveguides, each waveguide configured to output light of a different primary color than other waveguides of the plurality of waveguides. (Item 7) Item 2. The display system of item 1, wherein the light source comprises a plurality of selectively activated light-emitting areas. (Item 8) 8. The display system of claim 7, wherein the light source comprises at least one of a light emitting diode array and a spatial light modulator. (Item 9) 9. The display system of claim 8, wherein the light emitting diode array comprises an organic light emitting diode array or an inorganic light emitting diode array. (Item 10) 9. The display system of claim 8, wherein the spatial light modulator light source comprises a liquid crystal array or a digital light processing (DLP) chip. (Item 11) 8. The display system of claim 7, wherein the display system is configured to vary the position of an activated light-emitting region during projection of at least one of the intrapupillary images into the eye. (Item 12) The light source is A light emitter; an actuator configured to direct light to the spatial light modulator along different paths; Item 2. The display system of item 1, comprising: (Item 13) Item 14. The display system of item 12, wherein the actuator is a two-axis galvanometer. Item 13. The display system of item 12, wherein the light source comprises a fiber scanner. (Item 15) Item 2. The display system of item 1, wherein the spatial light modulator configured to modulate light from the light source comprises an LCOS panel. (Item 16) The display system further comprises an eye tracking sensor configured to track a line of sight of the eye, determining a line of sight of the eye using the eye tracking sensor; selecting content for the intrapupillary image based on the determined line of sight of the eye; 2. The display system of item 1, configured to: (Item 17) Item 2. The display system of item 1, wherein the display system is configured to synchronize the light output location of the light source with the image content provided by the spatial light modulator. (Item 18) 2. The display system of item 1, further comprising an optical mechanism between the spatial light modulator and the projection optics, the optical mechanism configured to direct light from different locations of the spatial light modulator to the projection optics at different angles. (Item 19) 20. The display system of claim 18, wherein the optical mechanism comprises one or more of a prism or lens structure. (Item 20) 20. The display system of claim 19, wherein the lens structure is a lenslet array. (Item 21) 1. A method for displaying image content, the method comprising: Providing a spatial light modulator; providing a light source configured to output light to the spatial light modulator from a plurality of different light output locations; displaying a virtual object on a depth plane by projecting a set of parallax-distinct intrapupillary images of the virtual object into a viewer's eye in temporal succession, each of the intrapupillary images comprising: outputting light from the light source to the spatial light modulator, the light being output from one or more associated light output locations of the light source; modulating the light with the spatial light modulator to form an intra-pupil image corresponding to the one or more associated light output locations; Propagating the modulated light to the eye. wherein the one or more associated light output locations for each intra-pupillary image are distinct from the one or more associated light output locations for other ones of the intra-pupillary images. A method comprising: (Item 22) 22. The method of claim 21, wherein activating the one or more associated light-emitting regions includes selecting the one or more associated light-emitting regions based on the depth plane, wherein a physical separation between light-emitting regions for the intra-pupil image increases with decreasing distance from the depth plane to the viewer. (Item 23) 22. The method of claim 21, wherein the light rays forming each of the parallaxly distinct images are collimated and the depth plane is less than optical infinity. (Item 24) 22. The method of claim 21, wherein injecting the set of disparately distinct intrapupillary images occurs within a time frame below the viewer's flicker fusion threshold. (Item 25) 25. The method of claim 24, wherein the flicker fusion threshold is 1 / 60 of a second. (Item 26) The display device further includes an eye tracking sensor configured to track a line of sight of the eye, and displaying the virtual object further comprises: determining a line of sight of the eye using the eye tracking sensor; selecting content for the intrapupillary image based on the determined line of sight of the eye; 22. The method according to claim 21, comprising: (Item 27) 22. The method of claim 21, further comprising a projection optical system configured to direct modulated light from the spatial light modulator to the eye. (Item 28) 22. The method of claim 21, wherein the one or more associated light emitting regions for the intrapupil image overlap. (Item 29) 22. The method of claim 21, further comprising varying a position of the one or more associated light-emitting regions during projection of at least one of the intrapupillary images into the eye. (Item 30) 1. A display system comprising: a light source having a plurality of spatially distinct light output locations; a spatial light modulator configured to modulate light from the light source; a projection optic mounted on the frame and configured to direct light from the spatial light modulator into a viewer's eye; Equipped with The display system is configured to display a virtual object on a depth plane by temporally successively projecting a set of parallax-distinct intra-pupillary images of the object into the eye. (Item 31) Item 31. The display system of item 30, configured to output light from different light output locations of the light source for different intrapupillary images. (Item 32) Item 31. The display system of item 30, configured to vary lateral separation between the light output locations based on a distance of the depth plane from the viewer's eye. (Item 33) Item 31. The display system of item 30, configured to increase lateral separation between the light output locations with increasing distance of the depth plane from the viewer's eye. (Item 34) Item 31. The display system of item 30, wherein the display system is configured to vary the light output location during projection of at least one of the intrapupillary images into the eye. (Item 35) 1. A method for displaying image content, the method comprising: Providing a head mounted display; displaying a virtual object on a depth plane by projecting a set of parallax-distinct intrapupillary images of the virtual object from said display into the viewer's eyes within a flicker fusion threshold; A method comprising: (Item 36) Item 36. The method of item 35, further comprising varying a depth plane in which the virtual object is placed by varying the parallax difference of the intrapupillary image. (Item 37) 36. The method of claim 35, wherein injecting a set of parallaxly distinct intra-pupillary images includes injecting individual ones of the intra-pupillary images in temporal succession into the viewer's eye. (Item 38) 36. The method of claim 35, wherein introducing a set of parallaxically distinct intrapupillary images includes introducing multiple ones of the intrapupillary images simultaneously. (Item 39) Item 39. The method of item 38, wherein injecting a set of parallax-distinct intrapupillary images includes injecting a plurality of intrapupillary images at once in temporal succession. (Item 40) Item 36. The method of item 35, wherein the light beam forming the intrapupillary image is collimated. (Item 41) Item 36. The method of item 35, wherein the light beam forming the intrapupillary image has a diverging wavefront. (Item 42) The display comprises: a spatial light modulator configured to modulate light to form the intra-pupillary image; a light source configured to direct light to the spatial light modulator; 36. The method of claim 35, comprising: (Item 43) 43. The method of claim 42, wherein the light source comprises a plurality of selectively activated light-emitting regions, and wherein projecting a set of parallaxically distinct intra-pupil images includes activating a different light-emitting region for each of the intra-pupil images. (Item 44) Item 43. The method of item 42, wherein the light source is configured to output light from a plurality of distinct light output locations, further comprising jittering the light output locations during projection of at least one of the intrapupillary images into the eye. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 illustrates a user's view of an augmented reality (AR) device.
[0013] [Diagram 2] FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user.
[0014] [Diagram 3] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.
[0015] [Figure 4] 4A-4C illustrate the relationship between curvature and focal length.
[0016] [Diagram 5] FIG. 5 illustrates an embodiment of a waveguide stack for outputting image information to a user.
[0017] [Figure 6A] FIG. 6A illustrates the pre-accommodation and post-accommodation states of the eye for a continuous incident wavefront.
[0018] [Figure 6B] FIG. 6B illustrates the pre-accommodation and post-accommodation states of the eye for a piecewise approximation of a continuous incident wavefront.
[0019] [Figure 7A] FIG. 7A illustrates the eye accommodating to a diverging wavefront emanating from a finite focal length virtual image provided by a projection system.
[0020] [Figure 7B] FIG. 7B illustrates a system for forming an approximation of the diverging wavefront of FIG. 7A utilizing wavefront segments formed by infinity-focused virtual images.
[0021] [Figure 8] FIG. 8 illustrates an example of parallax views forming the diverging wavefront approximation of FIG. 7B.
[0022] [Figure 9] FIG. 9 illustrates an example of a display system that includes a projection system for forming the diverging wavefront approximation of FIG. 7B.
[0023] [Figure 10] FIG. 10 illustrates examples of sizes, shapes, and distributions for the light emitting regions.
[0024] [Figure 11A] FIG. 11A illustrates another embodiment of a projection system for forming the diverging wavefront approximation of FIG. 7B.
[0025] [Figure 11B] FIG. 11B illustrates an example of a range of depth planes provided by the projection system of FIG. 11A.
[0026] [Figure 12] FIG. 12 illustrates an example of a light source configuration for a projection system.
[0027] [Figure 13A] FIG. 13A illustrates an example of a projection system for placing virtual objects on a default depth plane less than optical infinity.
[0028] [Figure 13B] FIG. 13B illustrates an example of a range of depth planes provided by the projection system of FIG. 13A.
[0029] [Figure 14] FIG. 14 illustrates an embodiment of a projection system configured for spatially multiplexed display of intrapupillary images.
[0030] [Figure 15] FIG. 15 illustrates an example of a projection system configured for spatially and temporally multiplexed display of intrapupillary images.
[0031] [Figure 16] FIG. 16 illustrates an example projection system that includes a pupil relay combiner eyepiece for superimposing image content onto a user's view of the world.
[0032] [Figure 17] FIG. 17 illustrates an example of a display system that includes an eye tracking system and a combiner eyepiece with a pupil expander.
[0033] [Figure 18] FIG. 18 illustrates an example of a display system comprising an eye tracking system and a pupil relay combiner eyepiece with a pupil expander configured to produce a non-infinite depth plane.
[0034] [Figure 19] FIG. 19 illustrates a light source with mirrors for directing light propagation to different light output locations.
[0035] [Figure 20] FIG. 20 illustrates a light source that includes a fiber scanner.
[0036] [Figure 21]FIG. 21 illustrates an example of an eyepiece with stacked waveguide assemblies for outputting different wavelengths of light corresponding to different primary colors.
[0037] [Figure 22] FIG. 22 illustrates an example of a wearable display system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0038] The human visual system can be made to perceive an image presented by a display as being "three-dimensional" by providing a slightly different presentation of the image to each of the viewer's left and right eyes. Depending on the image presented to each eye, the viewer perceives the "virtual" object in the image as being at a selected distance (e.g., a "depth plane") from the viewer. However, simply providing different presentations of the image to the left and right eyes may cause discomfort to the viewer. As discussed further herein, viewing comfort can be increased by having the eyes accommodate to the image, similar to the accommodation that would occur if the viewer were to view a real object at the depth plane in which the virtual object is located.
[0039] Appropriate accommodation for a virtual object on a given depth plane can be induced by presenting an image to the eye with light having a wavefront divergence that matches the wavefront divergence of light resulting from a real object on that depth plane. Some display systems use distinct structures with distinct refractive powers to provide the appropriate wavefront divergence. For example, one structure may provide a specific amount of wavefront divergence (for placing a virtual object on one depth plane) and another structure may provide a different amount of wavefront divergence (for placing a virtual object on a different depth plane). Thus, there may be a one-to-one correspondence between physical structures and depth planes in these display systems. Due to the need for a separate structure for each depth plane, such display systems may be bulky and / or heavy, which may not be desirable for some applications, such as portable head-mounted displays. In addition, such display systems may be limited in the number of different accommodative responses they can evoke from the eye due to practical limitations on the number of structures of different refractive powers that may be utilized.
[0040] It has been found that a continuous wavefront, e.g., a continuous diverging wavefront, can be approximated by injecting parallax-disparate intra-pupillary images directed into the eye. In some embodiments, a display system may provide a range of accommodative responses without requiring a one-to-one correspondence between the optical structures in the display and the accommodative responses. For example, the same optical projection system may be utilized to output light with a selected amount of perceived wavefront divergence corresponding to a desired depth plane by injecting a set of parallax-disparate intra-pupillary images into the eye. These images may be referred to as "parallax-disparate" intra-pupillary images because each image may be considered to be a different parallax view of the same virtual object or scene on a given depth plane. These are "intra-pupillary" images because a set of images carrying parallax disparity are projected into the pupil of one eye, e.g., the viewer's right eye. Although some overlap may occur, the light beams forming these images will have at least some areas with no overlap and will impinge on the pupil from slightly different angles. In some embodiments, the viewer's other eye, e.g., the left eye, may be provided with its own set of parallaxically distinct intra-pupillary images. The set of parallaxically distinct intra-pupillary images projected into each eye may be slightly different, e.g., the images may show slightly different views of the same scene due to the slightly different perspectives provided by each eye.
[0041] The wavefronts of light forming each of the intra-pupillary images projected into the pupil of the eye for a view may, in the aggregate, approximate a continuous diverging wavefront. The amount of perceived divergence of this approximated wavefront may be varied by varying the amount of parallax difference between the intra-pupillary images, which varies the angular range spanned by the wavefronts of light forming the intra-pupillary images. Preferably, this angular range mimics the angular range spanned by the continuous wavefront being approximated. In some embodiments, the wavefronts of light forming the intra-pupillary images are collimated or quasi-collimated.
[0042] In some embodiments, the display system utilizes a light source configured to output light from multiple distinct light output locations. For example, the light source may comprise multiple selectively activated light emitting regions, each region being a discrete light output location. The amount of parallax difference between intra-pupillary images may be varied by varying the light output location for each image. It should be understood that light from a given light output location may propagate along one path through the display system to the eye, and light from different light output locations on the light source may propagate along different paths through the display system to the eye. As a result, spatial differences in light output locations may be translated into differences in the paths that the light follows to the eye. Different paths may correspond to different amounts of parallax difference. Advantageously, in some embodiments, the amount of parallax difference may be selected by selecting the amount of spatial displacement or separation between the light output locations of the light source.
[0043] In some embodiments, as mentioned above, the light source may comprise a plurality of selectively activated light emitting regions, each corresponding to a distinct light output location. The light emitting regions may be arranged on a plane to form a 2D light emitter array. In some other embodiments, the light source may comprise a linear transfer lens, such as an F-theta (F-θ or F-tan θ) lens, a common or shared light emitter, and an actuator for directing the light emitted by the light emitter through the F-theta lens along different paths. Light exits from the light source at different locations through the F-theta lens, which focuses the exiting light onto an image plane. The light exiting the F-theta lens at different locations is also located at different locations on the image plane, which may be considered to provide a virtual 2D light emitter array. As a result, both the individual regions of the light emitter array and the locations through which the light from the linear transfer lens passes through the image plane may be referred to herein as light output locations of the light source.
[0044] In some embodiments, the actuator may be part of a two-axis galvanometer comprising multiple (e.g., pairs) mirrors that are independently actuated on different axes to direct light from the light emitter along a desired propagation path. In some other embodiments, the light source may comprise a fiber scanner, and the actuator may be an actuator configured to move a fiber of the fiber scanner. The light source may also comprise or be in communication with a processing module that synchronizes the output of light by the light source with the location of the mirror or fiber and the intra-pupillary image to be displayed. For example, the mirror or fiber may be moved along a known path, and the light emitter may be controlled by the processing module to emit light when the mirror or fiber is in a position that corresponds to a desired light output location for a particular intra-pupillary image (and the parallax difference associated with that image), as discussed further herein.
[0045] The display system may also include a spatial light modulator between the light source and the projection optics to inject light into the eye. The spatial light modulator may be configured to modulate light from the light source and encode image information in the light stream to form an intra-pupillary image. Preferably, the image is injected into the eye through projection optics that simultaneously provide an image of the spatial light modulator plane at or near optical infinity or some other selected "home plane" and an image of the light source at or near the viewer's pupil. Thus, both image content and precise parallax difference amounts can be provided to the eye.
[0046] In some embodiments, the same spatial light modulator may be used to modulate light and form various intra-pupillary images to be provided to the eye. In some such embodiments, the active light output locations (light output locations from which light actively propagates at a given time) may be synchronized with modulation by the spatial light modulator. For example, activation of a light output location corresponding to one intra-pupillary image may be synchronized, i.e., simultaneous with, activation of a display element in the spatial light modulator, which is configured to form an intra-pupillary image corresponding to a particular light-emitting region. Once another light output location corresponding to a second intra-pupillary image is activated, an appropriate, possibly different, display element in the spatial light modulator may be activated to form that second intra-pupillary image. Additional intra-pupillary images may be formed by synchronizing activation of the light output locations with the image content provided by the spatial light modulator. This time-based sequential presentation of intra-pupillary images to the eye may be referred to as temporal multiplexing or a temporally multiplexed display of intra-pupillary images. It should also be understood that an active or activated light output location is a location from which light actively propagates from a light source to a spatial light modulator that is used to form an intra-pupillary image.
[0047] In some other embodiments, spatial multiplexing may be utilized. In such embodiments, different areas (e.g., different pixels) of the spatial light modulator may be dedicated to forming different intra-pupillary images. Optical mechanisms may be provided between the spatial light modulator and the projection optics to direct light from different regions so that the light propagates through the projection optics in different directions. Examples of suitable optical mechanisms include lenslet arrays. As a result, different intra-pupillary images may be formed and provided to the eye simultaneously with the parallax difference determined by the location of the pixels forming the image and the optical mechanisms directing the propagation of light from those pixels. In some embodiments, a light source without selectively activated light-emitting regions (e.g., point light sources) may be utilized to generate light for the display system, since the parallax difference may be set using a combination of the spatial light modulator and the optical mechanisms.
[0048] In some other embodiments, both spatial and temporal multiplexing may be utilized. In such embodiments, the display system may include a light source with selectively activated light output locations in addition to the optical features and formation of different intra-pupillary images in different areas of the spatial light modulator described above. Parallax difference may be provided using a combination of selective activation of the light output locations and simultaneous formation of different intra-pupillary images in different locations of both the optical features and the spatial light modulator.
[0049] In embodiments involving temporal multiplexing, the set of intra-pupillary images for approximating a particular continuous wavefront are preferably launched into the eye so rapidly that the human visual system cannot detect that the images were presented at different times. Without being limited by theory, the visual system may perceive the images formed on the retina as being simultaneously present within the flicker fusion threshold. In some embodiments, approximating a continuous wavefront may include launching a beam of light for each set of intra-pupillary images into the eye sequentially, with the total duration for launching all of the beams of light being less than the flicker fusion threshold above which the human visual system would perceive the images as being launched separately into the eye. As an example, the flicker fusion threshold may be approximately 1 / 60 of a second. It should be understood that each set of images may consist of a certain number of parallax views, e.g., two or more views, three or more views, four or more views, etc., all of which are presented within the flicker fusion threshold.
[0050] Preferably, the display system has a small enough exit pupil such that the depth of field provided by the light forming the individual intrapupillary images is substantially infinite and the visual system operates in an "open loop" mode in which the eye cannot accommodate to the individual intrapupillary images. In some embodiments, the light beams forming the individual images occupy an area having a width or diameter of less than about 0.5 mm when incident on the eye. However, it should be understood that the light beams forming the set of intrapupillary images are at least partially non-overlapping and preferably define an area greater than 0.5 mm to provide sufficient information to the eye's crystalline lens to elicit a desired accommodation response based on the wavefront approximation formed by the wavefront of the light forming the intrapupillary images.
[0051] Without being limited by theory, the area defined by the set of beams of light may be considered to mimic a synthetic aperture through which the eye views the scene. It should be understood that viewing the scene through a sufficiently small pinhole in front of the pupil provides a nearly infinite depth of field. Given the small aperture of the pinhole, the eye's lens is not provided with adequate scene sampling to discern distinct depths of focus. As the pinhole expands, additional information is provided to the eye's lens, and natural optical phenomena allow a limited depth of focus to be perceived. Advantageously, the area defined by the set of beams of light and the corresponding set of parallaxically distinct intra-pupillary images may be made larger than a pinhole that would produce an infinite depth of field, and the multiple intra-pupillary images may produce an approximation of the effect provided by the enlarged pinhole described above.
[0052] As discussed herein, in some embodiments, different angles at which light beams propagate toward the pupil may be provided using a light source having multiple selectively activated light output locations that output light to a spatial light modulator, which modulates the light and forms an image. It should be understood that light from different light output locations of the light source will follow different paths to the spatial light modulator, which in turn will follow different paths from the spatial light modulator to the output pupil of the projection optics and thus the viewer's eye. As a result, lateral displacement of the active light output locations translates into angular displacement of the light exiting the spatial light modulator and ultimately propagating through the projection optics toward the viewer's pupil. In some embodiments, an increase in lateral displacement between activated light-emitting regions may be understood to translate into an increase in angular displacement as measured relative to the spatial light modulator plane. In some embodiments, each intra-pupil image may be formed by outputting light from different light output locations, thereby providing an angular displacement between the beams of light that form each of the images.
[0053] In some embodiments, the light sources and / or light output locations of the light sources may vary in position or jitter within a single parallax image (intra-pupillary image) display episode. For example, the light sources and / or light emitting regions may be physically moved and / or different light output locations (e.g., different light emitters of an array of light emitters) may be activated to display the intra-pupillary image while providing the desired positional variation. The rate of displacement or jitter may be higher than the update rate of the parallax image on the spatial light modulator. The jitter displacement may be in any direction, including rotation, depending on the desired perceptual effect.
[0054] In some embodiments, the display system may include a combiner eyepiece, which allows the virtual image content to be overlaid with the viewer's view of the world or surrounding environment. For example, the combiner eyepiece may be an optically transparent waveguide that allows the viewer to see the world. In addition, the waveguide may be utilized to receive, guide, and ultimately output light that forms an intrapupillary image to the viewer's eye. Because the waveguide may be positioned between the viewer and the world, the light output by the waveguide may be perceived to form a virtual image that is located on a depth plane within the world. In essence, the combiner eyepiece allows the viewer to receive a combination of light from the display system and light from the world.
[0055] In some embodiments, the display system may also include an eye-tracking system to detect the viewer's gaze direction. Such an eye-tracking system allows appropriate content to be selected based on where the viewer is looking.
[0056] Advantageously, by shifting the mechanism for providing a diverging wavefront from multiple discrete light output structures, which create a wavefront with a specific associated divergence, to a single structure that can create an arbitrary amount of divergence, the physical size and complexity of the system can be reduced. That is, some of the output structures can be eliminated. In addition, it may be possible to place virtual content on a larger number of depth planes, which would be practical if each depth plane required a dedicated structure to create a given wavefront divergence. This increase in the number of depth planes may provide a more realistic and comfortable viewing experience for the viewer. In addition, in some embodiments, the light from each spatial light modulator pixel may remain nominally collimated, thereby facilitating the integration of a projection system having the spatial light modulator with a combiner eyepiece that utilizes collimated pixel light.
[0057] Reference is now made to the drawings, in which like reference numbers refer to like parts throughout.
[0058] As discussed herein, the perception of an image as "three-dimensional" or "3-D" may be achieved by providing a slightly different presentation of the image to each eye of a viewer. FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user. Two distinct images 190, 200, one for each eye 210, 220, are output to the user. The images 190, 200 are spaced apart from the eyes 210, 220 by a distance 230 along an optical or z-axis parallel to the viewer's line of sight. The images 190, 200 are flat, and the eyes 210, 220 may focus on the images by assuming a single accommodated state. Such a 3-D display system relies on the human visual system to combine the images 190, 200 and provide a perception of depth and / or scale of the combined image.
[0059] However, it should be appreciated that the human visual system is more complex and more difficult to provide a realistic perception of depth. For example, many viewers of conventional "3-D" display systems may find such systems uncomfortable or may not perceive any sensation of depth at all. Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of vergence and accommodation. The vergence movement of the two eyes relative to each other (e.g., eye rotation, in which the pupils move toward or away from each other to converge the gaze of the eyes and fixate on an object) is closely linked to the focusing (or "accommodation") of the eye's lenses and pupils. Under normal conditions, changing the focus of the eye's lens or accommodating the eye from one object to another at different distances will automatically produce a matching change in vergence to the same distance, under a relationship known as the "accommodation-vergence-divergence reflex" and pupil dilation or constriction. Similarly, a change in vergence will induce a matching change in accommodation of lens shape and pupil size under normal conditions. As described herein, many stereoscopic or "3-D" display systems display a scene using a slightly different presentation (and therefore a slightly different image) to each eye such that a three-dimensional perspective is perceived by the human visual system. However, such systems, among other things, simply provide a different presentation of the scene, but are uncomfortable for many viewers because they work against the "accommodation-vergence-divergence reflex" when the eyes view the entire image information in a single accommodated state. Display systems that offer better matching between accommodation and convergence-divergence movements may result in a more realistic and comfortable simulation of three-dimensional images and may contribute to increased wearing duration.
[0060] FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3, objects at various distances from the eyes 210, 220 on the z-axis are accommodated by the eyes 210, 220 such that the objects are in focus. That is, the eyes 210, 220 assume a particular accommodated state and focus on objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 240 having an associated focal length such that an object or part of an object at a particular depth plane is in focus when the eye is in an accommodated state for that depth plane. In some embodiments, a three-dimensional image may be simulated by providing different presentations of images for each eye 210, 220, the presentation of the images also being different for different depth planes. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 210, 220 may overlap, for example, as the distance along the z-axis increases. Additionally, while shown as flat for ease of illustration, it should be understood that the contour of the depth plane may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular accommodated state.
[0061] The distance between an object and the eye 210 or 220 may also change the amount of divergence of light from that object as viewed by that eye. Figures 4A-4C illustrate the relationship between distance and divergence of light rays. The distance between the object and the eye 210 is represented in the order of decreasing distances R1, R2, and R3. As shown in Figures 4A-4C, the light rays become more divergent as the distance to the object decreases. As the distance increases, the light rays become more collimated. In other words, the light field generated by a point (an object or a part of an object) may be said to have a spherical wavefront curvature that is a function of the distance the point is away from the user's eye. The curvature increases with decreasing distance between the object and the eye 210. As a result, at different depth planes, the divergence of the light rays is also different, and the divergence increases with decreasing distance between the depth plane and the viewer's eye 210. Although only a monocular 210 is illustrated in FIGS. 4A-4C and various other figures herein for clarity of illustration, it should be understood that the discussion regarding eye 210 may apply to both eyes 210 and 220 of a viewer.
[0062] Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with different presentations of images corresponding to each of these limited number of depth planes. The different presentations may be focused separately by the viewer's eye, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features for a scene located on the different depth planes and / or based on the observation of different image features on the different depth planes that are out of focus.
[0063] Since each depth plane has an associated wavefront divergence, some displays may utilize a waveguide with an optical power to output light with a divergence corresponding to that depth plane to display image content that appears to be at a particular depth plane. Multiple similar, but different optical power, waveguides may be utilized to display image content on multiple depth planes. For example, such a system may utilize multiple such waveguides formed in a stack. FIG. 5 illustrates an example of a waveguide stack for outputting image information to a user. The display system 250 includes a stack of waveguides 260 that may be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310 for outputting image information. Image injection devices 360, 370, 380, 390, 400 may be utilized to inject light containing image information into the waveguides 270, 280, 290, 300, 310. Each waveguide 270, 280, 290, 300, 310 may include structures (e.g., optical gratings and / or lenses 570, 580, 590, 600, 610, respectively) that provide refractive power such that each waveguide outputs light with a pre-set amount of wavefront divergence that corresponds to a particular depth plane. Thus, each waveguide 270, 280, 290, 300, 310 places image content on an associated depth plane determined by the amount of wavefront divergence provided by that waveguide.
[0064] However, it should be understood that a one-to-one correspondence between waveguides and depth planes may lead to bulky and heavy devices in systems where multiple depth planes are desired. In such an embodiment, multiple depth planes would require multiple waveguides. Additionally, if a color image is desired, an even larger number of waveguides may be required, since each depth plane may have multiple corresponding waveguides, one waveguide per primary color may be required to form the color image.
[0065] Advantageously, various embodiments may provide a simpler display system that approximates the desired continuous wavefront by using discrete light beams that form intra-pupillary images that present different parallax views of an object or scene.
[0066] 6A, a pre-accommodation state of the eye 210 and a post-accommodation state in response to receiving a continuous input wavefront 1000 are illustrated. Illustration a) illustrates the pre-accommodation state before the visual system focuses the wavefront 1000 onto the retina 211. It is to be noted that the focal point 212 is not on the retina 211. For example, the focal point 212 may be in front of the retina 211 as shown. Illustration b) illustrates the post-accommodation state after the human visual system flexes the pupil musculature of the viewer's eye 210 to focus the wavefront 1000 onto the retina 211. As shown, the focal point 212 may be on the retina 211.
[0067] It has been found that a continuous wavefront, such as the wavefront 1000 of FIG. 6A, can be approximated using multiple wavefronts. FIG. 6B illustrates a pre-accommodation state of the eye 210 and a post-accommodation state in response to receiving a piecewise approximation of the continuous wavefront 1000 of FIG. 6A. Illustration a) of FIG. 6B illustrates the pre-accommodation state of the eye 210, and illustration b) illustrates the post-accommodation state. The approximation may be formed using multiple constituent wavefronts 1010a, 1010b, and 1010c, each associated with a separate beam of light. As used herein, reference numerals 1010a, 1010b, and 1010c may refer to both a beam of light and the associated wavefronts of the beam of light. In some embodiments, the constituent wavefronts 1010a and 1010b may be planar wavefronts, such as those formed by a collimated beam of light. As shown in illustration b), a wavefront approximation 1010 formed by constituent wavefronts 1010a and 1010b is focused by eye 210 onto retina 211, with focal point 212 on retina 211. Advantageously, the pre- and post-accommodation conditions are similar to those produced by the continuous wavefront 1000 shown in FIG.
[0068] It should be understood that a continuous diverging wavefront may be formed using an optical projection system. FIG. 7A illustrates an eye accommodating to a diverging wavefront emanating from a finite focal length virtual image provided by a projection system. The system includes a spatial light modulator 1018 and a projection optics 1020 with a focal length "F" and an external stop. An image may be formed by the spatial light modulator 1018, and light from the spatial light modulator 1018 containing image information may be directed to the eye 210 through the projection optics 1020. As shown in FIG. 7A, the spacing (less than F) between the spatial light modulator 1018 and the projection optics 1020 may be selected such that a diverging wavefront 1000 is output toward the eye 210. As described above with respect to FIG. 6A, the eye 210 may then focus the wavefront 1000 onto the retina 211.
[0069] FIG. 7B illustrates a system for forming an approximation of the diverging wavefront of FIG. 7A utilizing a wavefront segment formed by an infinity-focused virtual image. As described above, the system includes a spatial light modulator 1018 and a projection optics 1020. The spatial light modulator 1018 forms two images that are shifted relative to each other. The spatial light modulator 1018 is placed at a distance F from a back focal plane of the projection optics 1020, which has a back focal length of F. A light beam 1010a, containing image information for a first image, propagates through the projection optics 1020 into the eye 210. A light beam 1010b, containing image information for a second image, follows a different path through the projection optics 1020 into the eye 210. As discussed herein, the light beams 1010a and 1010b may be emitted from different regions of a light source (not shown), causing them to illuminate the spatial light modulator 1018 from different angles, thus causing the images formed by the light beams 1010a and 1010b to be spatially shifted relative to one another. The light beams 1010a and 1010b propagate along paths from the spatial light modulator through the projection optics 1020 into the eye 210 such that they define an angular range for each light beam that matches the angular range of the diverging wavefront 1000 (FIG. 7A). It should be understood that the angular separation between the light beams 1010a and 1010b increases with increasing approximated wavefront divergence. In some embodiments, the projection optics 1020 and the spacing between the spatial light modulator 1018 and the projection optics 1020 are configured such that the light beams 1010a and 1010b are collimated, respectively.
[0070] Referring now to FIG. 8, an example of parallax views forming the diverging wavefront approximation of FIG. 7B is illustrated. It should be understood that the light beams 1010a, 1010b, and 1010c each form distinctly different images of one view of the same object or scene from slightly different viewpoints corresponding to different locations of the image in space. As illustrated, the images may be injected into the eye 210 sequentially at different times. Alternatively, the images may be injected simultaneously if the optical system allows, or the images can be injected as a group, as discussed herein. In some embodiments, the total duration during which the light forming all of the images is injected into the eye 210 is less than the viewer's flicker fusion threshold. For example, the flicker fusion threshold may be 1 / 60 of a second, and the light beams 1010a, 1010b, and 1010c are all injected into the eye 210 for a duration less than the flicker fusion threshold. The human visual system therefore integrates all of these images, which appear to the eye 210 as if light beams 1010a, 1010b, and 1010c were launched simultaneously into the eye 210. Light beams 1010a, 1010b, and 1010c thus form a wavefront approximation 1010.
[0071] 9, an example of a display system 1001 is illustrated that includes a projection system 1003 for forming the diverging wavefront approximation 1010 of FIG. 7B. The projection system 1003 includes a light source 1026 configured to modulate the light and output light 1010a′ and 1010b′ to a spatial light modulator 1018, which forms images showing slightly different parallax views of the same object or scene. The modulated light with the image information then propagates through and is output by the relay / projection optics 1020 as light beams 1010a and 1010b into the eye 210. The projection system 1003 may also include a lens structure 1014, which may be configured to convert the spatial difference in the emission of light 1010a′ and 1010b′ into an angular difference in the propagation of that light to the spatial light modulator 1018. The projection system 1003 may further include a polarizing beam splitter 1016 configured to 1) direct light from the light source 1026 to the spatial light modulator 1018 and 2) allow modulated light from the spatial light modulator 1018 to propagate back through the beam splitter 1016 to the relay / projection optics 1020. In some embodiments, the display system 1001 may include an eye tracking device 1022, e.g., a camera, configured to monitor the gaze of the eye. Such monitoring may be used to determine the direction in which the viewer is looking, which may be used to select appropriate image content with respect to that direction. Preferably, the eye tracking device 1022 tracks both of the viewer's eyes, or each eye includes its own associated eye tracking device. As a result, the vergence and divergence movements of the viewer's eyes may be tracked and the convergence points of the eyes may be determined to determine the direction and distance at which the eyes are pointed.
[0072] It should be understood that the light 1010a' and 1010b' may be output at different times by the light source 1026, as discussed herein, the spatial light modulator 1018 may form different parallax views at different times using the light 1010a' and 1010b', and the resulting light beams 1010a and 1010b may be injected into the eye 210 at different times.
[0073] Continuing with reference to FIG. 9, the light source 1026 may be a 2D light source having a plurality of selectively activated light output locations disposed substantially on a plane. In some embodiments, the selectively activated light output locations may be selectively activated light emitting regions. For example, the light source 1026 may be a light emitting diode (LED) array or a spatial light modulator (e.g., a digital micromirror device such as a digital light processing (DLP) device, an LCOS device, etc.) that contains an array of discrete units or light emitters that output light. Examples of LED arrays include organic light emitting diode (OLED) arrays and inorganic light emitting diode (ILED) arrays. In some embodiments, individual light emitting diodes and / or light modulators within the light source 1026 may constitute a light emitting region. In some other embodiments, a group of light emitting diodes and / or light modulators may form a light emitting region. In such embodiments, there may be some overlap between the light emitting diodes and / or light modulators of different light emitting regions, but the regions may be considered distinctly different because the overlap is not complete.
[0074] In some other embodiments, the light source 1026 may be configured to focus light onto an image plane, in effect providing a virtual 2D light source at that image plane. Different locations on the image plane may be considered to be different light output locations, which may be activated by steering the light from the light emitter using actuated mirrors or fiber scanners to direct the light through those locations on the image plane. Further details regarding such virtual 2D light sources are provided in the discussion of Figures 19 and 20 below.
[0075] In some embodiments, an example of the spatial light modulator 1018 includes a liquid crystal on silicon (LCOS) panel. As another example, in some other embodiments, the spatial light modulator 1018 may comprise a transmissive liquid crystal panel, such as a DLP, or a MEMS device.
[0076] Continuing with reference to FIG. 9, the display system 1001 may also include a control system 1024 for determining the timing and type of image content provided by the display system. In some embodiments, the control system 1024 comprises one or more hardware processors with a memory that stores programs for controlling the display system 1001. For example, the system 1024 may be configured to control the activation of light emitting regions of the light source 1026, the actuation of individual pixel elements of the spatial light modulator 1018, and / or the interpretation and reaction of the display system 1001 to data received from the eye tracking device 1022. Preferably, the system 1024 includes a calculation module 1024a configured to receive input regarding a desired depth plane or wavefront divergence and calculate appropriate light emitting regions to activate to form a parallax view with an appropriate amount of difference for the desired depth plane or wavefront divergence. Additionally, the calculation module 1024a may be configured to determine appropriate actuation of pixels of the spatial light modulator 1018 to form an image of the desired parallax view. The system 1024 may also include a synchronization module 1024b configured to synchronize activation of certain light-emitting regions of the light source 1026 with modulation of light by the spatial light modulator 1018 to form images to provide parallax views corresponding to those activated light-emitting regions. In addition, the system 1024 may include an eye tracking module 1024c that receives input from the eye tracking device 1022. For example, the eye tracking device 1022 may be a camera configured to image the eye 210. Based on the images captured by the eye tracking device 1022, the eye tracking module 1024c may be configured to determine the orientation of the pupil and extrapolate the line of sight of the eye 210. This information may be electronically communicated to the calculation module 1024a. The calculation module 1024a may be configured to select image content based on the line of sight or line of sight of the eye 210 (and preferably also based on the line of sight or line of sight of the viewer's other eye).
[0077] Since the light source 1026 may include an array of discrete light emitters, the size and shape of the light emitting area formed by the light emitters may be varied as desired by activating selected ones of the light emitters. FIG. 10 illustrates examples of sizes, shapes, and distributions for the light emitting areas. It should be understood that the light and dark areas in the figure indicate different emitting areas that are activated for different parallax views. Example a) illustrates elongated light emitting areas that are horizontally spaced apart, which may be desirable for horizontal parallax-only driven accommodation. Example b) illustrates a circular light emitting area with both horizontal and vertical displacement. Example c) illustrates a light emitting area with a luminance falloff. Example d) illustrates overlapping light emitting areas. Example e) illustrates light emitting areas that form an array. As illustrated by the illustrated examples, the light source 1026 (FIG. 9) may include light emitters that incorporate binary (simply on and off) and / or grayscale (selectively emitting light of variable intensity). In some embodiments, light source 1026 may include elements that switch at a very high rate, including a rate that exceeds the disparity switching rate for system 1001. For example, light source 1026 may have light output elements that switch their light output on and off at a rate higher than the rate at which the disparity (intrapupillary) images are switched, in embodiments in which different intrapupillary images are displayed at different times.
[0078] Referring again to FIG. 9, in some embodiments, the control system 1024 may include two parts: 1) bright field generation; and 2) resolved field optimization. To approximate a wavefront as discussed herein, an appropriate image is displayed on the spatial light modulator 1018 for each activated light emitting area of the light source 1026. It should be appreciated that these images are created during the bright field generation step, where the 3D scene is rendered from multiple slightly offset viewpoints corresponding to slight shifts in the activated light emitting areas. For example, to display a 5×5 bright field, the 3D scene would be rendered 25 times from 25 different viewpoints arranged in a grid pattern. The locations of the viewpoints in the grid pattern correspond to the locations of the activated light source areas, and the rendered images would correspond to the images formed by the spatial light modulator.
[0079] It may be desirable to increase the brightness of the image formed by the spatial light modulator 1018. Advantageously, utilizing a light source 1026 with an array of light emitters allows for the formation of light-emitting regions having various shapes and sizes, which may be utilized to increase the brightness. In some embodiments, the brightness may be increased by increasing the size of the activated light-emitting regions without significantly changing the image formed by the spatial light modulator 1018. The calculation module 1024a may be configured to determine the size and shape of the activated light-emitting regions using a resolved field of view optimization. The module 1024a may be configured to take an input focal stack and create a series of patterns to be displayed on the spatial light modulator 1018 as well as on the light source 1026, the patterns being configured to create a desired approximation to the focal stack in a least squares sense. The optimization utilizes the fact that slight shifts in viewpoint do not significantly change the perceived image, and it is possible to display the same image on the spatial light modulator 1018 while generating a light-emitting region pattern using illumination from a larger area on the light source 1026.
[0080] The optimization problem can be formulated as a non-convex optimization problem, given below: [ka] where the projection operator ρ performs a linear transformation from a 4D bright field to a 3D focal stack (using a shift and add algorithm). The problem is a non-negative matrix decomposition embedded in a deconvolution problem. The algorithm for solving the problem uses the Alternating Directional Multiplier Method (ADMM). Additional details regarding an exemplary method for solving the problem are discussed in Appendix I. It should be appreciated that the module 1024a is configured to actively calculate in real time the appropriate size and shape of the light emitting region based on the parallax view to be formed by the spatial light modulator 1018.
[0081] In some other embodiments, the optimization problem may be formulated as a slightly different non-convex optimization problem, as given below. [ka] where A and B represent the pattern displayed on the spatial light modulator (e.g., light source 1026 and spatial light modulator 1018 to form an image), y is the desired output of the algorithm, a target 4D bright field, and AB' is an operator to combine the spatial light modulator patterns and simulate the 4D bright field emitted by a physical display when A and B are shown on the modulator. The problem is a non-negative matrix decomposition. The algorithm to solve this problem uses iterative optimization techniques to refine A and B from a random initial guess.
[0082] With continued reference to FIG. 9, it should be appreciated that the flicker fusion threshold of the human visual system imposes a time constraint on the number of images that can be thrown into the eye 210 while still being perceived as being thrown simultaneously. For example, the processing bandwidth of the control system 1024 and the ability to switch between the light emitting areas of the light source 1026 and the light modulators of the spatial light modulator 1018 may limit the number of images that can be thrown into the eye 210 within the duration allowed by the flicker fusion threshold. Given this finite number of images, the control system 1024 may be configured to make a selection regarding the image to be displayed. For example, within the flicker fusion threshold, the display system may be required to throw a set of parallaxically distinct intrapupillary images into the eye, and thus each parallax view may require images of different primary colors to form a full color image. In some embodiments, the formation of a full color image using primary color images is diverged from the elucidation of the desired accommodation response. For example, without being limited by theory, it may be considered possible to induce a desired accommodation response with monochromatic light. In such a case, the parallaxically distinct intrapupillary images used to elicit an accommodation response would be in only one color, so that it would not be required to form the parallaxically distinct intrapupillary images using light of other colors, thereby freeing up time within the flicker fusion threshold for other types of images to be displayed. For example, a set of larger parallaxically distinct intrapupillary images may be generated to better approximate the wavefront.
[0083] In some other embodiments, the control system 1024 may be configured to spend less time within the flicker fusion threshold to display images with colors of light to which the human visual system is less sensitive. For example, the human visual system is less sensitive to blue light than to green light. As a result, the display system may be configured to generate images formed with more green light than images formed with blue light.
[0084] Referring now to FIG. 11A, another embodiment of a projection system 1003 for forming the diverging wavefront approximation of FIG. 7B is illustrated. Preferably, the projection system produces a relatively long depth of field, which may be controlled by a limiting aperture in the system. Without being limited by theory, it is believed that the projection system providing an image to the eye with an effective pupil diameter of about 0.5 mm forces the human visual system to operate in an "open loop" mode, since the eye cannot accommodate to such an image. By providing an image with such an effective pupil diameter, the display system reduces the spot size on the retina for an image focused at infinity.
[0085] 11A, the projection system 1003 forms an image with parallax difference as discussed herein. The images can be rapidly alternated to the viewer's eyes at a rate (e.g., >60Hz) higher than the perceptual persistence of the human visual system. As discussed herein, the illustrated projection system 1003 simultaneously produces an image of an illumination source at a finite conjugate plane and an image of a pixel (image) source at infinity. In addition, the selectively activated light-emitting regions 1026a and 1026b are spaced apart to produce a displacement of the optical system pupil to align the parallax images relative to each other within the viewer's pupil.
[0086] 11B, an example of a range of depth planes provided by the projection system of FIG. 11A is illustrated. The range extends from a far plane at optical infinity to a near plane closer to the eye 210. The far plane at optical infinity may be provided by collimated light beams 1010a and 1010b. The near plane may be provided using spatially displaced activated light emitting regions as disclosed herein and may be understood to be the nearest depth plane provided by the display system. In some embodiments, the perceived proximity of the near plane to the eye 210 may be determined by the maximum parallax difference between light beams 1010a and 1010b, which may be determined by the maximum distance at which the display system allows selectively activated light emitting regions 1026a and 1026b to be separated while still forming a sharp image of the light source 1026 at or near the viewer's pupil.
[0087] Advantageously, the use of light source 1026, comprising multiple discrete, selectively activated light emitters as discussed herein, provides the ability to produce a wide range of pupil or perceived image shapes, brightness profiles, and arrays (through manipulation of illumination source size, shape, and position) to achieve various depth-of-field effects. Light source 1026 also advantageously provides the ability to flexibly and bidirectionally vary pupil shape as desired to accommodate horizontal parallax only, full parallax, or other combinations of parallax, in order to provide high luminous efficiency while driving accommodation.
[0088] 12, an example of a light source configuration for a projection system 1003 is illustrated. The light source 1026 includes a single fixed illuminator 1028 and a spatial light modulator 1032 for adjusting the output of light from the illuminator 1028 to the spatial light modulator 1018 for forming an image. The light source 1026 may also include a condenser / collimator lens to direct light from the illuminator 1028 to the spatial light modulator 1032. The spatial light modulator 1032 may include pixels and / or shutters that allow or block light to pass, as desired. It should be understood that the pixels and / or shutters may be actuated to allow light to pass, and that the areas through which light passes are considered to be light-emitting regions (e.g., light-emitting regions 1026a and 1026b).
[0089] 13A, an example of a projection system 1003 for placing virtual objects on a default depth plane less than optical infinity is illustrated. As shown, the projection optics 1020 may have a focal length "F" and the spatial light modulator may be positioned less than F, which biases the system 1003 to have a far depth plane less than optical infinity by diverging the light beams 1010a and 1010b. The amount that the light beams 1010a and 1010b diverge may be determined by the position of the spatial light modulator 1018 relative to the projection optics 1020; the closer the spacing, the more divergent. Because a certain amount of divergence is expected as a default due to the spacing, in some embodiments, the size of the light emitting regions 1026a and 1026b per intra-pupil image may be scaled up (e.g., by increasing the number of LEDs, increasing the number of light source illumination pixels activated to illuminate the spatial light modulator when forming the intra-pupil image, etc.) and the exit pupil associated with each intra-pupil image may be larger than 0.5 mm. As a result, the visual system may not function in an open-loop mode. In some embodiments, the size of the light emitting regions 1026a and 1026b may be set by a control system 1024 (FIG. 9), which may be programmed to vary the size of the light emitting regions 1026a and 1026b based on a desired default depth plane for the system 1003. In addition, the width of the cross section of the light beam for forming each intra-pupil image is preferably large enough for the optical structure of the projection optics 1020 to act on the light to provide the desired divergence.
[0090] 13B, an example of a range of depth planes provided by the projection system of FIG. 13A is illustrated. The illustrated range extends from a far plane at a distance D less than optical infinity to a near plane relative to the eye 210. The far plane may be set by appropriate selection of the position of the spatial light modulator relative to the projection optics 1020. The near plane may be provided as disclosed above with respect to FIG. 11B.
[0091] 14, an example of a projection system 1003 configured for spatially multiplexed display of intra-pupillary images is illustrated. Rather than relying on spatial displacement between light-emitting regions to provide the desired parallax difference, the parallax difference may be provided by utilizing different areas of a spatial light modulator 1018 to form different intra-pupillary images. An optical mechanism 1019 is configured to direct light from each of these different areas at different angles toward a projection optics 1020 that outputs light beams 1010a and 1010b toward a viewer's eye (not shown). In some embodiments, the areas of the spatial light modulator 1018 for forming the different intra-pupillary images may be interleaved. For example, pixels that provide image information for different intra-pupillary images may be interleaved with each other. The optical mechanism 1019 may be configured to convert different locations (e.g., different pixels) where the optical mechanism 1019 receives light to different angles at which the light from the pixels enters the projection optics 1020. In some embodiments, the optical mechanism 1019 may include a prism and / or lens structure, such as a lenslet array.
[0092] It should be understood that different non-overlapping regions of the spatial light modulator 1018 may be dedicated to providing image information for different intra-pupillary images. Because these regions are distinct from one another, in some embodiments, they may be activated simultaneously. As a result, multiple intra-pupillary images may be presented to the eye simultaneously. This may advantageously reduce the speed requirements that the spatial light modulator 1018 is required to refresh the images. As discussed above, to provide the perception that all images of a set of intra-pupillary images exist simultaneously to approximate a continuous wavefront, these images must all be presented within the flicker fusion threshold. In some embodiments, all or multiple images of a set of intra-pupillary images are presented simultaneously in different regions of the spatial light modulator such that a rapid sequential display of these simultaneously presented images is not required for the human visual system to perceive the images as existing simultaneously.
[0093] As shown, light source 1028 provides light through lens structure 1014 to illuminate spatial light modulator 1018. In some embodiments, light source 1028 may be a single fixed illuminator without any selectively activated light emitting regions.
[0094] In some other embodiments, the light source may include selectively activated light emitting regions, which may advantageously provide additional control of parallax difference. Thus, the projection system may utilize both spatial and temporal multiplexing. Now referring to FIG. 15, an example of a projection system 1003 configured for spatial and temporal multiplexing of intrapupil images is illustrated. The projection system 1003 may include a light source 1026, which may include selectively activated light emitting regions, e.g., regions 1026a and 1026b. As discussed herein, the spatial displacement between the light emitting regions may be utilized to provide parallax difference to the alternately output light beams 1010a, 1010b, 1010c, and 1010d. In addition, as discussed above with respect to FIG. 14, the projection system 1000 may include an optical mechanism 1019 between the spatial light modulator 1018 and the projection optics 1020. The spatial light modulator 1018 and the optical mechanism 1019 may cooperate to provide spatial multiplexing. Thus, illumination of the spatial light modulator 1018 with a single light-emitting region can produce multiple intra-pupillary images. For example, activation of light-emitting region 1026a illuminates the spatial light modulator 1018, which simultaneously generates image information for two intra-pupillary images, with the light beams for each image being directed in different directions by optical mechanism 1019. The light enters the projection optics 1020 and exits as light beams 1010b and 1010d to form two distinctly different intra-pupillary images. Similarly, subsequent activation of light-emitting region 1026b results in light beams 1010a and 1010d to form two other intra-pupillary images.
[0095] Referring to both Figures 14 and 15, in some embodiments, as discussed above with respect to Figures 11A-11B and 13A-13B, the location of the spatial light modulator 1018 and optical mechanism 1019 relative to the projection optics 1020 may be selected to provide a desired default "home" depth plane, which may be less than optical infinity.
[0096] The projection system 1003 shown in various figures herein may be part of a hybrid system that employs a single finite focal length eyepiece that places objects on a non-infinity depth plane as a default while employing parallax-driven accommodation to place virtual objects on other depth planes. For example, the projection system 1003 may be configured to have a default depth plane at 0.3 dpt or 0.5 dpt, which may be close enough to optical infinity to be within the tolerance of the human visual system for accommodation-vergence-divergence mismatch. For example, without being limited by theory, it is believed that the human visual system can comfortably tolerate displaying content from optical infinity on a 0.3 dpt depth plane. In such a system, the beams of light 1010a and 1010b would have a wavefront divergence that corresponds to the default depth plane. Advantageously, such a configuration may reduce the computational load on a processor (e.g., a graphics processing unit) in the display system, which may provide the advantages of lower power consumption, reduced latency, and increased processor options, among other benefits.
[0097] 16, an example of a projection system 1003 is illustrated that includes a pupil relay combiner eyepiece 1030 for superimposing image content onto a user's view of the world. Preferably, the eyepiece 1030 is optically transmissive, allowing light from the world to propagate through the eyepiece and into the viewer's eye 210. In some embodiments, the eyepiece 1030 includes one or more waveguides having an in-coupling optical element 770 and an out-coupling optical element 800. The in-coupling optical element 770 receives light from the projection optics 1020 and redirects it through the eyepiece 1030 by total internal reflection to propagate to the out-coupling optical element 800. The out-coupling optical element 800 outputs the light to the viewer's eye 210. Advantageously, the eyepiece 1030 preserves all of the image attributes provided by the projection system 1003 so that rapid switching of parallax views is accurately depicted through the eyepiece 1030.
[0098] The incoupling optical element 770 and the outcoupling optical element 800 may be refractive or reflective structures. Preferably, the incoupling optical element 770 and the outcoupling optical element 800 are diffractive optical elements. Examples of diffractive optical elements include surface relief features, volume phase features, metamaterials, or liquid crystal polarization gratings.
[0099] It should be appreciated that the outcoupling optical element 800 or other optical elements forming part of the eyepiece 1030 may be configured to have a refractive power. In some embodiments, the refractive power may be selected to correct refractive errors of the eye 210, including refractive errors such as myopia, hyperopia, presbyopia, and astigmatism.
[0100] 17, an example of a projection system 1003 is illustrated that includes an eye tracking system 1022 and a combiner eyepiece 1030 with a pupil expander 1034. The pupil expander replicates the projection system pupil across the eyepiece 1030. Because the pupil expander 1034 replicates the projection system pupil across a large area that may be traversed by the viewer's pupil through eye movement, the image formed by the spatial light modulator 1018 and the location of the light emitting region of the light source 1026 can be updated in real time based on input from the eye tracking system 1022. Advantageously, this configuration allows for a larger eyebox for more comfortable viewing and reduces limitations on the relative positioning of the eye and combiner and variations in interpupillary distance.
[0101] 18, an example of a projection system 1003 is illustrated that includes an eye tracking system 1022 and a combiner eyepiece 1030 with a pupil expander 1035 configured to produce a non-infinity depth plane. In some embodiments, the non-infinity depth plane may be at 3 meters, which results in a budgeted accommodation of about 2.5 meters to infinity. For example, given the tolerance of the human visual system for accommodation-vergence-divergence mismatch, virtual content at a distance of about 2.5 meters from the viewer to infinity may be placed on the 3 meter depth plane with little discomfort. In such a system, parallaxically distinct intrapupillary images may be used to drive accommodation for a narrower range of depth planes, possibly closer to the viewer than an all-fixed "default" focal plane. In some embodiments, the system may also incorporate an eye tracking system 1022 to determine the distance of the viewer's fixation, for example, based on the viewer's binocular vergence-divergence angle.
[0102] In some embodiments, the light source 1026 may be replaced with a virtual light source formed on an image plane of an optical projection system. The optical projection system may include an actuator capable of scanning a beam of light across an area on the image plane corresponding to the virtual light source. To mimic the ability to activate discrete light-emitting areas of the light source 1026, the output of light by the projection system is synchronized with the movement of the actuator to cause the light to be output at a particular time to a desired location on the image plane. Preferably, the rate at which the actuator can scan the beam of light across the image plane is high enough so that all desired light output locations on the image plane can be accessed during the time frame in which any given intra-pupillary image is displayed. For example, during the amount of time in which a particular intra-pupillary image is displayed, the actuator is preferably capable of scanning the beam of light at least once, and preferably multiple times, across the area of the image plane corresponding to the virtual 2D light source.
[0103] FIG. 19 illustrates a light source 2026 with mirrors for directing light propagation to different light output locations. The light source 2026 includes a light emitter 2028 and mirrors 2030 and 2032, which are moved by actuators 2031 and 2033, respectively. Examples of light emitters 2028 include LEDs and lasers. In some embodiments, fiber optic cables may transmit light from remotely located light emitters. As shown, light 1010a', 1010b' propagates from emitter 2028 to mirror 2032, which reflects the light to mirror 2030, which then reflects the light and propagates through lens 2034 to focus on intermediate image plane 1026'. The mirrors 2030 and 2032 may be part of a two-axis galvanometer, and the actuators 2031 and 2033 rotate the mirrors along different axes, e.g., orthogonal axes, thereby allowing the light to be directed to an area defined along the two axes of the image plane 1026'. In some embodiments, the actuators 2031, 2033 may be motors. The lens 2034 may be a linear transfer lens, such as an F-theta (F-θ or F-tan θ) lens, and may be configured to focus the light onto the flat image plane 1026'. In some embodiments, the light rays 1010a', 1010b' propagate from the image plane 1026' in a similar manner that the light would propagate from the light source 1026 (see, e.g., FIG. 9). In some embodiments, the light source 2026 may also include a collimating lens 2036 to collimate the light emitted by the light emitter 2028 before it reaches the mirror 2032.
[0104] The light source 2026 also preferably includes or communicates with a processing module 2038 that controls the output of light from the light emitter 2028 and synchronizes it with the movement of the actuators 2031, 2033 and the intra-pupillary image to be formed. For example, the processing module 2038 may coordinate the movement of the mirrors 2032, 2030 with the emission of light from the light emitter 2028. In some embodiments, the mirrors 2032, 2030 are continuously rotated or pivoted back and forth by the actuators 2031, 2033 on the axis about which the mirrors are designed to move. The emission of light (e.g., pulses of light) by the light emitter 2028 is timed with this movement such that the light is directed at a given moment to a desired location on the intermediate image plane 1026', the location and time also being determined based on the intra-pupillary image to be displayed (e.g., activation of a particular light output location coincides in time with the display of an intra-pupillary image having a parallax difference associated with that particular light output location). In some embodiments, the emission of light from the light emitter 2028 is controlled by switching the light emitter 2028 between an on state and an off state (e.g., by supplying or not supplying power to the light emitter, respectively). In some other embodiments, the emission of light from the light emitter 2028 may be mechanically controlled using a physical switch that selectively allows or blocks light from reaching the image plane 1026'.
[0105] 20, the light source 2026 may include a fiber scanner 2027. The fiber scanner 2027 may include a light emitter 2028 and an actuator 2040 that moves the fiber 2042. The light 1010a', 1010b' propagates from the end of the fiber 2042 through the lens 2034 and is focused onto the image plane 2026'. It should be appreciated that the actuator 2040 may move the fiber 2042 at a known speed along a predetermined path (e.g., a circular path). As a result, the processing module 2038 may be configured to synchronize the propagation of light from the end of the fiber 2042 with the movement of the fiber 2042 such that the light propagates from the fiber 2042 at a desired light output location and thus is synchronized with the intra-pupillary image to be displayed.
[0106] As noted above, the light source 2026 may replace the light source 1026 in any of the display systems discussed. For example, the light source 2026 may be a substitute for the light source 1026 in the projection system 1003 or display system 1001 of any of Figures 9, 11A, 12, 13A, and 15-18.
[0107] 21, an example of eyepiece 660 (which may correspond to eyepiece 1030 of FIGS. 14-16) is illustrated that includes a stacked waveguide assembly for outputting different wavelengths of light corresponding to different primary colors. In some embodiments, the waveguide assembly includes waveguides 670, 680, and 690. Each waveguide includes an associated internal coupling optical element (which may also be referred to as the light input area on the waveguide), for example, internal coupling optical element 700 is disposed on a major surface (e.g., upper major surface) of waveguide 670, internal coupling optical element 710 is disposed on a major surface (e.g., upper major surface) of waveguide 680, and internal coupling optical element 720 is disposed on a major surface (e.g., upper major surface) of waveguide 690. In some embodiments, one or more of the incoupling optical elements 700, 710, 720 may be disposed on a bottom major surface of the respective waveguide 670, 680, 690 (particularly, one or more of the incoupling optical elements are reflective polarizing optical elements). As shown, the incoupling optical elements 700, 710, 720 may be disposed on an upper major surface of that respective waveguide 670, 680, 690 (or on top of the next lower waveguide), and particularly, the incoupling optical elements are transmissive polarizing optical elements. In some embodiments, the incoupling optical elements 700, 710, 720 may be disposed within the body of the respective waveguide 670, 680, 690. In some embodiments, as discussed herein, the incoupling optical elements 700, 710, 720 are wavelength selective such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of the respective waveguides 670, 680, 690, it should be understood that the internal coupling optical elements 700, 710, 720 may be located within other areas of the respective waveguides 670, 680, 690 in some embodiments.
[0108] As shown, the in-coupling optical elements 700, 710, 720 may be laterally offset from one another. In some embodiments, each in-coupling optical element may be offset to receive light without that light passing through another in-coupling optical element. For example, each in-coupling optical element 700, 710, 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400, as shown in FIG. 6, and may be separated (e.g., laterally spaced) from the other in-coupling optical elements 700, 710, 720 so as to substantially not receive light from others of the in-coupling optical elements 700, 710, 720. In some embodiments, the in-coupling optical elements 700, 710, 720 are vertically aligned and not laterally offset.
[0109] Each waveguide also includes an associated optically dispersive element, for example, optically dispersive element 730 is disposed on a major surface (e.g., a top major surface) of waveguide 670, optically dispersive element 740 is disposed on a major surface (e.g., a top major surface) of waveguide 680, and optically dispersive element 750 is disposed on a major surface (e.g., a top major surface) of waveguide 690. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on the bottom major surfaces of associated waveguides 670, 680, 690, respectively. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on both the top and bottom major surfaces of associated waveguides 670, 680, 690, respectively, or optically dispersive elements 730, 740, 750 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 670, 680, 690, respectively.
[0110] The waveguides 670, 680, 690 may be spaced apart and separated by, for example, gas, liquid, and / or solid layers of material. For example, as shown, layer 760a may separate the waveguides 670 and 680, and layer 760b may separate the waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low index material (i.e., a material having a lower index of refraction than the material forming the immediate neighbors of the waveguides 670, 680, 690). Preferably, the index of refraction of the material forming layers 760a, 760b is 0.05 or more, or 0.10 or less than the index of refraction of the material forming the waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may act as cladding layers to promote TIR of light through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layers 760a, 760b are formed from air. Although not shown, it should be understood that the top and bottom of the illustrated set of waveguides 660 may include immediate cladding layers.
[0111] With continued reference to FIG. 21, light rays 770, 780, 790 are incident on and launched into waveguides 670, 680, 690 by projection system 1003 (FIGS. 9 and 11-16).
[0112] In some embodiments, the light beams 770, 780, 790 have different properties, e.g., different wavelengths or different wavelength ranges, that may correspond to different colors. The internal coupling optical elements 700, 710, 720 each deflect the incident light such that the light propagates through a respective one of the waveguides 670, 680, 690 by TIR.
[0113] For example, incoupling optical element 700 may be configured to deflect light beam 770 having a first wavelength or range of wavelengths. Similarly, transmitted light beam 780 impinges on and is deflected by incoupling optical element 710, which is configured to selectively deflect light of a second wavelength or range of wavelengths. Similarly, light beam 790 is deflected by incoupling optical element 720, which is configured to selectively deflect light of a third wavelength or range of wavelengths.
[0114] 21 , the incoupling light rays 770, 780, 790 are deflected by the incoupling optical elements 700, 710, 720, respectively, and then propagate by TIR in the waveguides 670, 680, 690, respectively. The light rays 770, 780, 790 then impinge on the optically dispersive elements 730, 740, 750, respectively. The optically dispersive elements 730, 740, 750 deflect the light rays 770, 780, 790 to propagate towards the outcoupling optical elements 800, 810, 820, respectively.
[0115] In some embodiments, the optically dispersive elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or disperse the light to the out-coupling optical elements 800, 810, 820 and may increase the beam or spot size of this light as it propagates to the out-coupling optical elements. In some embodiments, for example, if the beam size is already the desired size, the optically dispersive elements 730, 740, 750 may be omitted and the in-coupling optical elements 700, 710, 720 may be configured to deflect the light directly to the out-coupling optical elements 800, 810, 820. In some embodiments, the out-coupling optical elements 800, 810, 820 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct the light to the viewer's eye 210 (FIGS. 15-16). It should be understood that the OPE may be configured to increase a dimension of the eyebox in at least one axis, and the EPE may increase the eyebox in an axis that intersects, e.g., is perpendicular to, the axis of the OPE.
[0116] Thus, in some embodiments, the eyepiece 660 includes a waveguide 670, 680, 690, an in-coupling optical element 700, 710, 720, a light dispersive element (e.g., OPE) 730, 740, 750, and an out-coupling optical element (e.g., EP) 800, 810, 820 for each primary color. The waveguides 670, 680, 690 may be stacked with an air gap / cladding layer between each one. The in-coupling optical element 700, 710, 720 redirects or deflects the incoming light into its waveguide (with different in-coupling optical elements receiving different wavelengths of light). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the illustrated embodiment, light ray 770 (e.g., blue light) is deflected by the first in-coupling optical element 700 in the manner described above, then continues to bounce back down the waveguide, interacting with the optically dispersive element (e.g., OPE) 730 and then the out-coupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red light, respectively) pass through the waveguide 670, with light ray 780 incident on the in-coupling optical element 710 and being deflected thereby. Light ray 780 will then bounce back down the waveguide 680, via TIR, to its optically dispersive element (e.g., OPE) 740 and then the out-coupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through the waveguide 690 and impinges on the optically in-coupling optical element 720 of the waveguide 690. The light in-coupling optical element 720 deflects the light beam 790 such that it propagates by TIR to a light dispersive element (e.g., OPE) 750 and then by TIR to an out-coupling optical element (e.g., EP) 820. The out-coupling optical element 820 then finally out-couples the light beam 790 to a viewer, who also receives the out-coupled light from the other waveguides 670, 680.
[0117] 22, there is illustrated an example of a wearable display system 60. Display system 60 may correspond to display system 1001 of FIG. 9, with a projection system 1003 for each eye of a viewer or user 90.
[0118] The display system 60 includes a display 70 and various mechanical and electronic modules and systems to support the functionality of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and configured to position the display 70 in front of the user's 90 eye. The display 70 may be considered eyewear in some embodiments. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned adjacent to the ear canal of the user 90 (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / adjustable sound control). In some embodiments, the display system may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphones may be configured to allow the user to provide input or commands (e.g., voice menu command selections, natural language questions, etc.) to the system 60 and / or enable audio communication with other persons (e.g., other users of similar display systems). The microphone may further be configured as an ambient sensor to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system may also include an ambient sensor 120a, which may be separate from the frame 80 and attached to the body of the user 90 (e.g., on the head, torso, limbs, etc. of the user 90). The ambient sensor 120a, in some embodiments, may be configured to obtain data characterizing a physiological state of the user 90. For example, the sensor 120a may be an electrode.
[0119] 22, the display 70 is operably coupled by a communication link 130, such as wired or wireless connectivity, to a local data processing module 140, which may be mounted in a variety of configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 90 (e.g., in a backpack configuration, in a belt-coupled configuration). Similarly, the sensor 120a may be operably coupled to the local processor and data module 140 by a communication link 120b, such as wired or wireless connectivity. The local processing and data module 140 may comprise a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or hard disk drive), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) captured from sensors (e.g., which may be operatively coupled to the frame 80 or otherwise attached to the user 90), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein, and / or b) data obtained and / or processed using a remote processing module 150 and / or a remote data repository 160 (including data related to virtual content), possibly for processing or retrieval and then passing to the display 70. The local processing and data module 140 may be operatively coupled to the remote processing module 150 and the remote data repository 160 by communications links 170, 180, such as via wired or wireless communication links, such that these remote modules 150, 160 are operatively coupled to each other and available as resources to the local processing and data module 140.In some embodiments, local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be mounted to frame 80 or may be a stand-alone structure that communicates with local processing and data module 140 by a wired or wireless communication path. In some embodiments, local processing and data module 140 may include one or more graphics processors and may correspond to control system 1024 (FIG. 9).
[0120] 22, in some embodiments, the remote processing module 150 may comprise one or more processors configured to analyze and process the data and / or image information. In some embodiments, the remote data repository 160 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers, which provide information, e.g., information for generating augmented reality content, to the local processing and data module 140 and / or the remote processing module 150. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from the remote modules. EXAMPLES
[0121] Brightfield and focal stack decomposition Bright field and focal stack decompositions can be utilized to determine the light output of the display system 1001, including the output of the light sources 1026, 2026 and the spatial light modulator 1018. More details regarding the decompositions are discussed below.
[0122] 1. Focal Stack Decomposition The focal stack y is decomposed into a series of time-multiplexed patterns to be displayed on two spatial light modulators A and B, located at the pupil and image planes, respectively. In some embodiments, spatial light modulators A and B may correspond to light sources 1026, 2026 and spatial light modulator 1018, respectively. All quantities are represented as a focal stack with a vertical resolution of m pixels, a horizontal resolution of n pixels, and s focal slices. [ka] But a single vector [ka] The tensor representation is vectorized so that it would be represented as: Bold symbols are used below for discrete vectors. Unless otherwise specified, different color channels are ignored and assumed to be independent. Table 1 provides an overview of the tensor notations and operators employed herein. [ka]
[0123] The spatial light modulator at image plane B also has a resolution of m×n pixels, but in addition, k time-multiplexed patterns can be displayed in rapid succession such that they will be perceptually averaged by the viewer. These spatiotemporal patterns are vectorized into a matrix such that every spatial pixel is a vector forming the row index of this matrix, and the k time steps are the column indexes of the matrix. [ka] Similarly, the pupil plane SLMA is expressed as the matrix [ka] where o is the total number of addressable SLM pixels in the pupil plane and the column index is again the time step.
[0124] Hence, the goal of decomposing the focal stack y into a set of time-multiplexed patterns can be written as a non-convex optimization problem. [ka] In the formula, the projection operator [ka] performs a linear transformation (using a shift-and-add algorithm) from a 4D bright field to a 3D focal stack. The problem is a non-negative matrix decomposition embedded in the deconvolution problem. The Alternating Direction Multipliers Method (ADMM) (Boyd et al, 2001, “Distributed optimization and statistical learning via the alternating direction method of multipliers”, Foundations and Trends in Machine Learning 3, 1, 1-122) may be used to solve it.
[0125] Equation 1 can be rewritten as an equivalence problem to put it into standard ADMM form. [ka] In the formula, the matrix [ka] is the operator [ka] In the matrix form of [ka] simply vectorizes the matrix into a single 1D vector (eg, using column major order as done by software MATLAB available from The MathWorks, Natick, Massachusetts).
[0126] The augmented Lagrangian method for this system is then formulated as follows: [ka] In scaled form, the present augmented Lagrangian method can be written as follows: [ka] During the ceremony, [ka] It is.
[0127] The ADMM algorithm then consists of three separate updates (or proximity operators), which are performed iteratively as follows: [ka] Here, the operator [ka] reshapes the vector into a matrix, and the operator [ka] did to vectorize the matrix. Equations 5-7 can be solved iteratively, each time using the most recent output from the previous step.
[0128] 1.1 Efficient z-updates Equation 5 is an unconstrained linear problem that can be rewritten as the following single linear equation system: [ka] Although the system is large, all operations can be expressed as matrix-free function handles, so matrices are not explicitly formed. A variety of different solvers can be used to solve the system for z. For example, MATLAB's very simple simultaneous algebraic reconstruction method (SART) may be utilized.
[0129] To increase computational efficiency, it would be desirable to derive a closed-form solution for the z-update, which could facilitate real-time implementation of the entire algorithm. One approach to deriving a closed-form solution starts with the normal equations for Equation 8. [ka] To find a closed-form solution for this, [ka] The inverse of is derived. Since P transforms the bright field into a focal stack, and the Fourier slice theorem dictates that refocusing in the primary domain is slicing in the Fourier domain, a closed form solution in the frequency domain can be derived. Using this insight, it can be written as follows: [ka] where s is the number of slices in the focal stack, and i is a focal slice in the 4D frequency domain i is a diagonal matrix representing the slicing operator on F 4D and [ka] represent the discrete 4D Fourier transform and its inverse, respectively.
[0130] The expected algebraic expression for the inverse matrix is: [ka] It should be appreciated that such closed-form solutions may provide a solution more quickly than iterative algorithms, since no iterations are required, which may also be preferred if sufficient computational resources are available.
[0131] 1.2 Efficient A,B-updates The A,B-update (Equation 6) is a Nonnegative Matrix Factorization (NMF) problem. In this case, it is the simplest possible NMF problem. Standard solutions for this and more advanced NMF approaches are detailed in Section 2 below.
[0132] 1.3 Handling Color Channels In the above derivations, a grayscale separation was assumed, or each color channel was assumed to be processed independently. In some cases, this may not provide a satisfactory approximation. For example, a two-color SLM may introduce color crosstalk, which is not modeled above. In addition, in some embodiments, the display system may use a combination of a grayscale LCoS at the image plane and a color LED or OLED array at the pupil plane. In this case, all color channels are linked.
[0133] When considering linked color channels, neither the z-updates nor the u-updates change; that is, they can be calculated independently per color channel per ADMM iteration. However, the matrix decomposition routines A,B-updates change. R,G,B / B R,G,B Instead of independently decomposing A, a single decomposition is performed simultaneously for all color channels of A (B contains no color channels in this case) as follows: [ka]
[0134] 2. Matrix decomposition transformation Nonnegative matrix decomposition is one approach to decomposing a matrix into a sum of nonnegative rank-1 matrices. The decomposition problem is nonconvex and therefore the solution is not simple. The problem and possible solutions will be discussed here.
[0135] The problem can be stated as decomposing the matrix X into a sum of rank-1 matrices. [ka] During the ceremony, [ka] The sum of rank-1 matrices yields a rank-K approximation of the original matrix.
[0136] A minimum squared error solution to this problem can be found by optimizing the following objective function: [ka] where the Frobenius norm of a matrix is [ka] is given as:
[0137] 2.1 Alternating least squares approach Cost function [ka] is both nonlinear and nonconvex, with a certain number of local minima. Fixing either A or B makes the solution for the other matrix convex. An alternating least-squares approach, which is expected to converge if the nonnegativity constraint is not taken into account, can be employed to solve this decomposition problem. To this end, each decomposition matrix is updated while fixing the other in an alternating fashion. The individual updates are computed using gradient descent. [ka] During the ceremony, [ka] is the derivative of the cost function for each decomposition matrix, and α A,B is its individual step length. As shown in the following section, one approach to choosing the step length is to choose one such that the update rule is multiplicative. Before discussing the step length, the gradient can be considered and given as follows: [ka] In matrix form, the gradient can be written as: [ka]
[0138] 2.2 Multiplicative Update Rules As mentioned above, the key to choosing the step length is that the additive update rule (Equation 15) can be written in simple multiplication by combining it with the steepest descent direction. ij ≧0 and A, B are initialized with positive values, the multiplicative update rule provides that the decomposition matrix remains positive throughout the iterative update process. The following step lengths result in the multiplicative update rule: [ka] Combining equations 15, 17 and 18 gives: [ka] The following multiplicative update rule is a simplified version of Equation 19: [ka] Starting from an initial guess containing only positive values (usually random noise) and assuming the data matrix X is non-negative, these update rules are expected to keep A and B positive throughout the iteration process. In practice, a small value is added to the divisor to avoid division by zero. [ka] 2.3 Weighted non-negative matrix factorization The multiplicative update rule is that matrix element x ij It may be modified to include weighting for each. [ka] where W is a weighting matrix of the same size as X. 2.4 Projected NMF Projective NMF adds an additional projection matrix P to the objective function that remains fixed throughout the optimization procedure. [ka] Here, A and B remain invariant in their dimensions to the previous surjection, but [ka] teeth, [ka] The gradient with respect to this formula is: [ka] This can be written in matrix form as follows: [ka] If the step length is selected as follows, [ka] This leads to the following multiplicative update rule: [ka]
[0139] 2.5 Projection-weighted NMF For projective NMF, the weights may be added to the light field, resulting in the following update rule: [ka]
[0140] It should be understood that each of the processes, methods, and algorithms described herein and / or depicted in the figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and thereby be fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuits, etc. Code modules may be written in a programming language that may be compiled and linked into an executable program, installed in a dynamic link library, or interpreted. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.
[0141] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, a video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.
[0142] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. In some embodiments, the non-transitory computer readable medium may be part of one or more of the local processing and data module (140), the remote processing module (150), and the remote data repository (160). The methods and modules (or data) may also be transmitted as a data signal generated (e.g., as part of a carrier wave or other analog or digital propagating signal) over a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device or communicated via a computer readable transmission medium.
[0143] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith may be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the program components, methods, and systems described may generally be integrated together in a single computer product or packaged into multiple computer products.
[0144] In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the invention. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.
[0145] Indeed, it should be understood that the systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure.
[0146] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may even be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or variation of the subcombination. No single feature or group of features is required or essential to every embodiment.
[0147] In particular, it should be understood that conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to suggest that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether or not those features, elements, and / or steps should be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprise," "include," "have," and the like, are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the appended claims, should be interpreted to mean "one or more" or "at least one," unless otherwise specified. Similarly, although operations may be depicted in the figures in a particular order, it should be recognized that such operations need not be performed in the particular order depicted, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Additionally, the figures may diagrammatically depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may also be incorporated within the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the depicted operations.In addition, operations may be rearranged or reordered in other embodiments. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.
[0148] Thus, the claims are not intended to be limited to the implementations shown in the specification but are to be accorded the widest scope consistent with the disclosure, the principles and novel features disclosed herein.
Claims
1. A head-mountable display system, comprising: a light source configured to emit light from different locations; a spatial light modulator configured to receive the light emitted by the light source and modulate the received light to form virtual content, the light emitted by the light source from the different locations being received and modulated in different respective areas of the spatial light modulator; a projection optics element configured to direct the modulated light from the spatial light modulator toward an eye of a wearer of the head mountable display system; A head-mountable display system comprising:
2. A head-mountable display system as described in claim 1, wherein the light source is configured to emit the light from the different locations at different times.
3. A head-mountable display system as described in claim 1, wherein the light source includes at least two light emitters that emit light of different colors.
4. A head-mountable display system as described in claim 1, wherein the light source comprises a fiber and an actuator configured to move the fiber so that ends of the fiber are at the different locations at different times, and the light is emitted from the ends of the fiber.
5. A head-mountable display system as described in claim 1, wherein the light source comprises a plurality of selectively activated light-emitting regions, each of the plurality of selectively activated light-emitting regions including the different locations, each of the light-emitting regions comprising at least one light emitter, and the light from the different light-emitting regions of the light source is received and modulated by the different areas of the spatial light modulator.
6. A head-mountable display system as described in claim 5, wherein at least two of the light-emitting areas overlap on the light source.
7. A head-mountable display system as described in claim 5, wherein the light-emitting areas are arranged separately on the light source without overlapping.
8. A head-mountable display system as described in claim 5, wherein the at least one light emitter includes at least one light-emitting diode.
9. A head-mountable display system as described in claim 5, wherein at least one of the light-emitting areas is substantially rectangular in shape or substantially circular in shape.
10. A head-mountable display system as described in claim 5, wherein at least two of the light-emitting areas are configured to emit light of different intensities.
11. A head-mountable display system as described in claim 5, wherein one or more of the light emitters are configured to emit light of selectively variable intensity.
12. A head-mountable display system as described in claim 5, wherein the light from different light-emitting regions of the light source is output from the spatial light modulator at different angles.
13. A head-mountable display system as described in claim 1, further comprising a lens structure between the light source and the spatial light modulator.
14. A head-mountable display system as described in claim 13, wherein the lens structure is configured to convert a spatial difference in the light emitted from the light source into an angular difference in the propagation of the light to the spatial light modulator.
15. A head-mountable display system as described in claim 1, further comprising a polarizing beam splitter arranged to direct at least a portion of the light from the light source toward the spatial light modulator.
16. A head-mountable display system as described in claim 1, wherein the spatial light modulator includes one or more of a liquid crystal panel on silicon, a transmissive liquid crystal panel, a digital light processing device, or a MEMS device.
17. A head-mountable display system as described in claim 1, wherein the projection optical element is further configured to collimate the light output from the spatial light modulator at different angles.
18. A head-mountable display system as described in claim 1, further comprising a lenslet array between the spatial light modulator and the projection optical element, the lenslet array configured to direct the light output from the different areas of the spatial light modulator at different angles to the projection optical element.
19. A head-mountable display system as described in claim 1, further comprising a prism between the spatial light modulator and the projection optical element, the prism configured to direct the light output from the different areas of the spatial light modulator at different angles to the projection optical element.
