Systems and methods for presenting image content on multiple depth planes by providing multiple intra-pupillary suggested views

Abstract

To provide optical devices, including augmented reality and virtual reality imaging and visualization systems.SOLUTION: An augmented reality display system is configured to direct a plurality of parallactically-disparate intra-pupil images into a viewer's eye. The parallactically-disparate intra-pupil images provide different parallax views of a virtual object, and impinge on the pupil from different angles. In the aggregate, the wavefronts of light forming the images approximate a continuous divergent wavefront and provide selectable accommodation cues for the user, depending on the amount of parallax disparity between the intra-pupil images. The amount of parallax disparity is selected using a light source that outputs light for different images from different locations, with spatial differences in the locations of the light output providing differences in the paths that the light takes to the eye, which in turn provide different amounts of parallax disparity. Advantageously, the wavefront divergence and the accommodation cue that is provided to the eye of the user may be varied by appropriate selection of parallax disparity, which may be set by selecting the amount of spatial separation between the locations of light output.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (Priority application) This application claims the benefit of priority to U.S. Provisional Application No. 62 / 411,490, filed October 21, 2016, which is incorporated herein by reference.

[0002] (Incorporated by reference) This application incorporates by reference the entirety of each of the following patent applications: U.S. Application No. 14 / 555,585, filed November 27, 2014; U.S. Application No. 14 / 690,401, filed April 18, 2015; U.S. Application No. 14 / 212,961, filed March 14, 2014; U.S. Application No. 14 / 331,218, filed July 14, 2014; U.S. Application No. 15 / 072,290, filed March 16, 2016; and U.S. Provisional Application No. 62 / 156,809, filed May 4, 2015.

[0003] The present disclosure relates to optical devices, including augmented reality and virtual reality imaging and visualization systems. [Background technology]

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality, or "VR," scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input, while augmented reality, or "AR," scenarios typically involve the presentation of digital or virtual image information as an augmentation to the user's visualization of the real world around them. Mixed reality, or "MR," scenarios are a type of AR scenario that typically involve virtual objects integrated into and responsive to the natural world. For example, MR scenarios may include AR image content that appears blocked by or is perceived to otherwise interact with objects in the real world.

[0005] Referring to Figure 1, an augmented reality scene 10 is depicted in which a user of the AR technology sees a real-world park-like setting 20 featuring people, trees, a building in the background, and a concrete platform 30. In addition to these items, the user of the AR technology also perceives as "seeing" "virtual content," such as a robotic figure 40 standing on the real-world platform 30 and a flying, cartoon-like avatar character 50 that appears to be an anthropomorphic bumblebee; however, these elements 50, 40 are "virtual" in that they do not exist in the real world. The human visual perception system is complex, making it difficult to produce AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0006] The systems and methods disclosed herein address various challenges associated with AR or VR technology. Summary of the Invention [Means for solving the problem]

[0007] In some embodiments, a head-mounted display system is provided, the display system comprising: a frame configured to mount on a viewer; a light source; a spatial light modulator configured to modulate light from the light source; and projection optics mounted on the frame and configured to direct light from the spatial light modulator into the viewer's eyes. The display system is configured to display virtual objects on a depth plane by projecting a set of parallaxly distinct intra-pupillary images of the objects into the eyes.

[0008] In some other embodiments, a method for displaying image content is provided. The method includes providing a spatial light modulator, providing a light source configured to output light to the spatial light modulator from a plurality of different light output locations, and displaying a virtual object on a depth plane by temporally sequentially projecting a set of parallaxly distinct intra-pupillary images of the virtual object into a viewer's eye. Each intra-pupillary image is formed by outputting light from a light source to the spatial light modulator, the light being output from one or more associated light output locations of the light source, modulating the light using the spatial light modulator to form an intra-pupillary image corresponding to the one or more associated light output locations, and propagating the modulated light into the eye. The one or more associated light output locations for each intra-pupillary image are distinct from one or more associated light output locations for the rest of the intra-pupillary images.

[0009] In yet another embodiment, a display system is provided that includes a light source with a plurality of spatially distinct light output locations, a spatial light modulator configured to modulate light from the light source, and projection optics mounted on a frame and configured to direct light from the spatial light modulator into a viewer's eye, wherein the display system is configured to display a virtual object on a depth plane by projecting a set of parallaxly distinct intrapupillary images of the object into the eye in temporal succession.

[0010] In some other embodiments, a method for displaying image content is provided, the method including providing a head-mounted display comprising a light source and a spatial light modulator, the method further including displaying a virtual object on a depth plane by projecting a set of parallaxly distinct intra-pupillary images of the virtual object from the display into the viewer's eyes within a flicker fusion threshold.

[0011] Additionally, various innovative aspects of the subject matter described in this disclosure may be implemented in the following embodiments. 1. A method for displaying image content, comprising: providing a spatial light modulator; providing a light source configured to output light to a spatial light modulator from a plurality of different light output locations; displaying the virtual object on the depth plane by temporally successively projecting a set of parallaxly distinct intrapupillary images of the virtual object into the viewer's eyes, each of the intrapupillary images comprising: outputting light from a light source to a spatial light modulator, the light being output from one or more associated light output locations of the light source; modulating light with a spatial light modulator to form intra-pupil images corresponding to one or more associated light output locations; Propagating the modulated light to the eye; wherein the one or more associated light output locations for each of the intra-pupil images are distinctly different from the one or more associated light output locations for the others of the intra-pupil images; A method comprising: 2. The method of embodiment 1, wherein the step of activating one or more associated light-emitting regions includes selecting one or more associated light-emitting regions based on a depth plane, and the physical separation between the light-emitting regions for the intra-pupil image increases with decreasing distance from the depth plane to the viewer. 3. A method according to any of embodiments 1-2, wherein the light rays forming each of the parallaxly distinct images are collimated and the depth plane is less than optical infinity. 4. A method as described in any of embodiments 1-3, wherein the step of introducing a set of parallaxly distinct intra-pupillary images is performed within a time frame below the viewer's flicker fusion threshold. 5. The method of embodiment 4, wherein the flicker fusion threshold is 1 / 60 of a second. 6. The method further includes an eye tracking sensor configured to track the line of sight of an eye, and the step of displaying the virtual object includes: determining eye gaze using an eye tracking sensor; A method according to any one of embodiments 1-5, comprising selecting content for the intra-pupillary image based on the determined line of sight of the eye. 7. The method of any of embodiments 1-6, further comprising a projection optics configured to direct modulated light from the spatial light modulator to an eye. 8. The method of any one of embodiments 1-7, wherein one or more associated luminous regions for the intrapupil image partially overlap. 9. A method as described in any of embodiments 1-8, further comprising the step of varying the position of one or more associated light-emitting regions during the projection of at least one of the intra-pupillary images into the eye. 10. A display system configured to implement the method of any one of embodiments 1-9. 11. A method for displaying image content, comprising: A light source and a spatial light modulator; providing a head mounted display comprising: displaying a virtual object on a depth plane by projecting a set of parallaxly distinct intrapupillary images of the virtual object from a display into the viewer's eyes within a flicker fusion threshold; A method comprising: 12. The method of embodiment 11, wherein the step of injecting a set of parallaxly distinct intra-pupillary images includes the step of injecting individual ones of the intra-pupillary images into the viewer's eye in temporal succession. 13. The method of embodiment 11, wherein the step of introducing a set of parallaxly distinct intra-pupil images includes the step of introducing multiple ones of the intra-pupil images simultaneously. 14. The method of embodiment 13, wherein the step of introducing a set of parallaxly distinct intra-pupil images includes introducing multiple intra-pupil images at once in temporal succession. 15. The method of any of embodiments 11-14, wherein the light beam forming the intrapupillary image is collimated. 16. The method of any of embodiments 11-14, wherein the light beam forming the intra-pupil image has a diverging wavefront. 17. A method according to any of embodiments 11-16, wherein the light source comprises a plurality of selectively activated light-emitting regions, and wherein the step of projecting a set of parallax-different intra-pupil images includes the step of activating a different light-emitting region for each intra-pupil image. 18. A method as described in any of embodiments 11-17, wherein the light source is configured to output light from a plurality of distinct light output locations, and further comprising a step of jittering the light output locations during the projection of at least one of the intra-pupillary images into the eye. 19. A display system configured to implement the method of any of embodiments 11-18. 20. A head-mounted display system, comprising: a frame configured to mount on a viewer; A light source and a spatial light modulator configured to modulate light from the light source; projection optics mounted on the frame and configured to direct light from the spatial light modulator into the eye of a viewer; wherein the display system is configured to display a virtual object on a depth plane by projecting a set of parallaxly distinct intra-pupillary images of the object into the eye. 21. A display system as described in embodiment 20, wherein the display system is configured to temporally multiplex the display of individual intrapupillary images. 22. A display system according to any of embodiments 20-21, wherein the display system is configured to spatially multiplex the display of intrapupillary images. 23. A display system according to any of embodiments 20-22, wherein the display system is configured to temporally multiplex the display of multiple spatially multiplexed intrapupillary images. 24. A display system according to any one of embodiments 20-23, wherein the projection optics comprises a waveguide comprising an internal coupling optical element and an external coupling optical element. 25. The display system of any of embodiments 24-25, wherein the projection optics comprises a plurality of waveguides, each waveguide configured to output light of a different primary color than the other waveguides of the plurality of waveguides. 26. The display system of any of embodiments 20-25, wherein the light source comprises a plurality of selectively activated light-emitting regions. 27. The display system of embodiment 26, wherein the light source includes at least one of a light emitting diode array and a spatial light modulator. 28. The display system of embodiment 8, wherein the light-emitting diode array comprises an organic light-emitting diode array or an inorganic light-emitting diode array. 29. The display system of embodiment 27, wherein the spatial light modulator light source comprises a liquid crystal array or a digital light processing (DLP) chip. 30. A display system described in any of embodiments 20-29, wherein the display system is configured to vary the position of the activated light-emitting region during the projection of at least one of the intrapupillary images into the eye. 31. The light source is a light emitter; 26. A display system as described in any one of embodiments 20-25, comprising an actuator configured to direct light to the spatial light modulator along different paths. 32. A display system as described in embodiment 31, wherein the actuator is a two-axis galvanometer. 33. The display system of embodiment 31, wherein the light source is a fiber scanner. 34. The display system of any of embodiments 20-33, wherein the spatial light modulator configured to modulate light from the light source comprises an LCOS panel. 35. The display system further comprises an eye tracking sensor configured to track the line of sight of an eye. determining a line of sight of the eye using an eye tracking sensor and selecting content for the intra-pupillary image based on the determined line of sight of the eye; A display system according to any one of embodiments 20-34, configured as follows: 36. A display system according to any of embodiments 20-35, wherein the display system is configured to synchronize the light output location of the light source with the image content provided by the spatial light modulator. 37. A display system described in any of embodiments 20-36, further comprising an optical mechanism between the spatial light modulator and the projection optics, the optical mechanism being configured to direct light from different locations on the spatial light modulator to the projection optics at different angles. 38. The display system of embodiment 37, wherein the optical mechanism comprises one or more of a prism or lens structure. 39. The display system of embodiment 38, wherein the lens structure is a lenslet array. 40. A display system comprising: a light source having a plurality of spatially distinct light output locations; a spatial light modulator configured to modulate light from the light source; projection optics mounted on the frame and configured to direct light from the spatial light modulator into the eye of a viewer; wherein the display system is configured to display a virtual object on a depth plane by temporally successively projecting a set of parallaxly distinct intrapupillary images of the object into the eye. 41. A display system as described in embodiment 40, configured to output light from different light output locations of the light source for different intrapupillary images. 42. A display system as described in embodiment 41, configured to vary the lateral separation between light output locations based on the distance of the depth plane from the viewer's eyes. 43. A display system according to any of embodiments 41-42, configured to increase the lateral separation between light output locations with increasing distance of the depth plane from the viewer's eyes. 44. A display system described in any of embodiments 41-42, wherein the display system is configured to change the light output location during the projection of at least one of the intrapupillary images into the eye. The present invention provides, for example, the following. (Item 1) 1. A head-mounted display system, comprising: a frame configured to mount on a viewer; A light source and a spatial light modulator configured to modulate light from the light source; projection optics mounted on the frame and configured to direct light from the spatial light modulator into a viewer's eye; Equipped with The display system is configured to display a virtual object on a depth plane by projecting a set of parallaxly distinct intra-pupillary images of the object into the eye. (Item 2) Item 10. The display system of item 1, wherein the display system is configured to temporally multiplex the display of individual intrapupillary images. (Item 3) Item 10. The display system of item 1, wherein the display system is configured to spatially multiplex the display of the intrapupillary image. (Item 4) Item 10. The display system of item 1, wherein the display system is configured to temporally multiplex the display of multiple spatially multiplexed intrapupillary images. (Item 5) Item 6. The display system of item 5, wherein the projection optics comprises a waveguide comprising an internal coupling optical element and an external coupling optical element. (Item 6) Item 6. The display system of item 5, wherein the projection optics comprises a plurality of waveguides, each waveguide configured to output light of a different primary color than the other waveguides of the plurality of waveguides. (Item 7) Item 10. The display system of item 1, wherein the light source comprises a plurality of selectively activated light-emitting areas. (Item 8) Item 8. The display system of item 7, wherein the light source comprises at least one of a light emitting diode array and a spatial light modulator. (Item 9) Item 9. The display system of item 8, wherein the light emitting diode array comprises an organic light emitting diode array or an inorganic light emitting diode array. (Item 10) Item 9. The display system of item 8, wherein the spatial light modulator light source comprises a liquid crystal array or a digital light processing (DLP) chip. (Item 11) 8. The display system of claim 7, wherein the display system is configured to vary the position of an activated light-emitting region during the projection of at least one of the intrapupillary images into the eye. (Item 12) The light source is a light emitter; an actuator configured to direct light along different paths to the spatial light modulator; Item 1. A display system according to item 1, comprising: (Item 13) Item 14. The display system of item 12, wherein the actuator is a two-axis galvanometer. Item 13. The display system of item 12, wherein the light source comprises a fiber scanner. (Item 15) Item 10. The display system of item 1, wherein the spatial light modulator configured to modulate light from the light source comprises an LCOS panel. (Item 16) and an eye tracking sensor configured to track the line of sight of the eye, the display system further comprising: determining a line of sight of the eye using the eye tracking sensor; selecting content for the intrapupillary image based on the determined line of sight of the eye; Item 1. The display system of item 1, configured to perform the following: (Item 17) Item 10. The display system of item 1, wherein the display system is configured to synchronize the light output location of the light source with the image content provided by the spatial light modulator. (Item 18) Item 1. The display system of item 1, further comprising an optical mechanism between the spatial light modulator and the projection optics, the optical mechanism configured to direct light from different locations on the spatial light modulator to the projection optics at different angles. (Item 19) Item 19. The display system of item 18, wherein the optical mechanism comprises one or more of a prism or lens structure. (Item 20) 20. The display system of claim 19, wherein the lens structure is a lenslet array. (Item 21) 1. A method for displaying image content, the method comprising: providing a spatial light modulator; providing a light source configured to output light to the spatial light modulator from a plurality of different light output locations; displaying a virtual object on a depth plane by temporally successively projecting a set of parallaxly distinct intrapupillary images of the virtual object into a viewer's eye, each of the intrapupillary images comprising: outputting light from the light source to the spatial light modulator, the light being output from one or more associated light output locations of the light source; modulating the light with the spatial light modulator to form an intra-pupillary image corresponding to the one or more associated light output locations; transmitting the modulated light to the eye; and wherein the one or more associated light output locations for each intra-pupil image are distinct from the one or more associated light output locations for other ones of the intra-pupil images; A method comprising: (Item 22) 22. The method of claim 21, wherein activating the one or more associated light-emitting regions includes selecting the one or more associated light-emitting regions based on the depth plane, wherein a physical separation between light-emitting regions for the intra-pupil image increases with decreasing distance from the depth plane to the viewer. (Item 23) 22. The method of claim 21, wherein the light rays forming each of the parallaxly distinct images are collimated and the depth plane is less than optical infinity. (Item 24) Item 22. The method of item 21, wherein injecting the set of parallaxly distinct intrapupillary images occurs within a time frame below the viewer's flicker fusion threshold. (Item 25) 25. The method of claim 24, wherein the flicker fusion threshold is 1 / 60 of a second. (Item 26) The displaying of the virtual object further comprises: determining a line of sight of the eye using the eye tracking sensor; selecting content for the intrapupillary image based on the determined line of sight of the eye; 22. The method according to item 21, comprising: (Item 27) Item 22. The method of item 21, further comprising a projection optics configured to direct modulated light from the spatial light modulator to the eye. (Item 28) 22. The method of claim 21, wherein the one or more associated light-emitting regions for the intrapupil image partially overlap. (Item 29) 22. The method of claim 21, further comprising varying a position of the one or more associated light-emitting regions during the projection of at least one of the intrapupillary images into the eye. (Item 30) 1. A display system comprising: a light source having a plurality of spatially distinct light output locations; a spatial light modulator configured to modulate light from the light source; projection optics mounted on the frame and configured to direct light from the spatial light modulator into the eye of a viewer; Equipped with The display system is configured to display a virtual object on a depth plane by temporally successively projecting a set of parallaxly distinct intra-pupillary images of the object into the eye. (Item 31) Item 31. The display system of item 30, configured to output light from different light output locations of the light source for different intrapupillary images. (Item 32) Item 31. The display system of item 30, configured to vary lateral separation between the light output locations based on the distance of the depth plane from the viewer's eyes. (Item 33) Item 31. The display system of item 30, configured to increase lateral separation between the light output locations with increasing distance of the depth plane from the viewer's eyes. (Item 34) Item 31. The display system of item 30, wherein the display system is configured to vary the light output location during the projection of at least one of the intrapupillary images into the eye. (Item 35) 1. A method for displaying image content, the method comprising: Providing a head mounted display; displaying a virtual object on a depth plane by projecting a set of parallaxly distinct intrapupillary images of the virtual object from said display into the viewer's eyes within a flicker fusion threshold; A method comprising: (Item 36) Item 36. The method of item 35, further comprising changing the depth plane in which the virtual object is placed by changing the parallax difference of the intrapupillary image. (Item 37) Item 36. The method of item 35, wherein injecting a set of parallaxly distinct intrapupillary images includes injecting individual ones of the intrapupillary images into the viewer's eye in temporal succession. (Item 38) 36. The method of claim 35, wherein introducing a set of parallaxly distinct intrapupillary images includes simultaneously introducing a plurality of said intrapupillary images. (Item 39) Item 39. The method of item 38, wherein injecting a set of parallaxly distinct intrapupillary images includes injecting multiple intrapupillary images at once in temporal succession. (Item 40) Item 36. The method of item 35, wherein the light beam forming the intrapupillary image is collimated. (Item 41) Item 36. The method of item 35, wherein the light beam forming the intrapupillary image has a diverging wavefront. (Item 42) The display a spatial light modulator configured to modulate light to form the intra-pupillary image; a light source configured to direct light to the spatial light modulator; Item 36. The method of item 35, comprising: (Item 43) 43. The method of claim 42, wherein the light source comprises a plurality of selectively activated light-emitting regions, and wherein projecting the set of parallaxly distinct intra-pupil images includes activating a different light-emitting region for each of the intra-pupil images. (Item 44) Item 43. The method of item 42, wherein the light source is configured to output light from a plurality of distinct light output locations, further comprising jittering the light output locations during the projection of at least one of the intrapupillary images into the eye. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a user's view of an augmented reality (AR) device.

[0013] [Figure 2] FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user.

[0014] [Figure 3] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0015] [Figure 4] 4A-4C illustrate the relationship between curvature and focal length.

[0016] [Figure 5] FIG. 5 illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0017] [Figure 6A] FIG. 6A illustrates the pre-accommodation and post-accommodation states of the eye for a continuous incident wavefront.

[0018] [Figure 6B] FIG. 6B illustrates the pre-accommodation and post-accommodation states of the eye for a piecewise approximation of a continuous incident wavefront.

[0019] [Figure 7A] FIG. 7A illustrates the eye accommodating a diverging wavefront emanating from a finite focal length virtual image provided by a projection system.

[0020] [Figure 7B] FIG. 7B illustrates a system for forming an approximation of the diverging wavefront of FIG. 7A utilizing wavefront segments formed by infinity-focused virtual images.

[0021] [Figure 8] FIG. 8 illustrates an example of parallax views forming the diverging wavefront approximation of FIG. 7B.

[0022] [Figure 9] FIG. 9 illustrates an example of a display system that includes a projection system for forming the diverging wavefront approximation of FIG. 7B.

[0023] [Figure 10] FIG. 10 illustrates examples of sizes, shapes, and distributions for the light-emitting regions.

[0024] [Figure 11A] FIG. 11A illustrates another embodiment of a projection system for forming the diverging wavefront approximation of FIG. 7B.

[0025] [Figure 11B] FIG. 11B illustrates an example of a range of depth planes provided by the projection system of FIG. 11A.

[0026] [Figure 12] FIG. 12 illustrates an example of a light source configuration for a projection system.

[0027] [Figure 13A] FIG. 13A illustrates an example of a projection system for placing virtual objects on a default depth plane below optical infinity.

[0028] [Figure 13B] FIG. 13B illustrates an example of a range of depth planes provided by the projection system of FIG. 13A.

[0029] [Figure 14] FIG. 14 illustrates an example of a projection system configured for spatially multiplexed display of intrapupillary images.

[0030] [Figure 15] FIG. 15 illustrates an example of a projection system configured for spatially and temporally multiplexed display of intrapupillary images.

[0031] [Figure 16] FIG. 16 illustrates an example projection system with a pupil relay combiner eyepiece for superimposing image content onto a user's view of the world.

[0032] [Figure 17] FIG. 17 illustrates an example of a display system that includes an eye tracking system and a combiner eyepiece with a pupil expander.

[0033] [Figure 18] FIG. 18 illustrates an example display system comprising an eye tracking system and a pupil relay combiner eyepiece with a pupil expander configured to produce a non-infinite depth plane.

[0034] [Figure 19] FIG. 19 illustrates a light source with mirrors for directing light propagation to different light output locations.

[0035] [Figure 20] FIG. 20 illustrates a light source that includes a fiber scanner.

[0036] [Figure 21]FIG. 21 illustrates an example of an eyepiece with stacked waveguide assemblies for outputting light of different wavelengths corresponding to different primary colors.

[0037] [Figure 22] FIG. 22 illustrates an example of a wearable display system. DETAILED DESCRIPTION OF THE INVENTION

[0038] The human visual system can be made to perceive an image presented by a display as being "three-dimensional" by providing a slightly different presentation of the image to each of the viewer's left and right eyes. Depending on the image presented to each eye, the viewer perceives "virtual" objects in the image as being at a selected distance (e.g., a certain "depth plane") from the viewer. However, simply providing different presentations of the image to the left and right eyes may cause discomfort to the viewer. As discussed further herein, viewing comfort can be increased by forcing the eyes to accommodate to the image, similar to the accommodation that would occur if the viewer were viewing a real object at the depth plane in which the virtual object is located.

[0039] Appropriate accommodation for a virtual object at a given depth plane can be induced by presenting an image to the eye using light with a wavefront divergence that matches the wavefront divergence of light originating from a real object at that depth plane. Some display systems provide the appropriate wavefront divergence using distinct structures with distinct refractive powers. For example, one structure may provide a specific amount of wavefront divergence (to place a virtual object at one depth plane), while another structure may provide a different amount of wavefront divergence (to place a virtual object at a different depth plane). Thus, there may be a one-to-one correspondence between physical structures and depth planes in these display systems. Due to the need for a separate structure for each depth plane, such display systems may be bulky and / or heavy, which may be undesirable for some applications, such as portable head-mounted displays. Additionally, such display systems may be limited in the number of different accommodative responses they can evoke from the eye due to practical limitations on the number of structures with different refractive powers that can be utilized.

[0040] It has been discovered that a continuous wavefront, e.g., a continuous diverging wavefront, can be approximated by projecting parallax-disparate intrapupillary images directed into the eye. In some embodiments, a display system may provide a range of accommodative responses without requiring a one-to-one correspondence between the optical structures in the display and the accommodative response. For example, the same optical projection system may be utilized to output light with a selected amount of perceived wavefront divergence corresponding to a desired depth plane by projecting a set of parallax-disparate intrapupillary images into the eye. These images may be referred to as “parallax-disparate” intrapupillary images because each image may be considered a different parallax view of the same virtual object or scene at a given depth plane. These are “intrapupillary” images because a set of images bearing parallax disparity are projected into the pupil of one eye, e.g., the viewer's right eye. While some overlap may occur, the light beams forming these images will have at least some areas without overlap and will impinge on the pupil from slightly different angles. In some embodiments, the viewer's other eye, e.g., the left eye, may be provided with its own set of parallaxly distinct intra-pupillary images. The set of parallaxly distinct intra-pupillary images projected into each eye may be slightly different, e.g., the images may show slightly different views of the same scene due to the slightly different perspectives provided by each eye.

[0041] The wavefronts of light forming each of the intra-pupillary images projected into the pupil of an eye for a given view may collectively approximate a continuous diverging wavefront. The amount of perceived divergence of this approximated wavefront may be varied by varying the amount of parallax difference between the intra-pupillary images, which varies the angular range spanned by the wavefronts of light forming the intra-pupillary images. Preferably, this angular range mimics the angular range spanned by the continuous wavefront being approximated. In some embodiments, the wavefronts of light forming the intra-pupillary images are collimated or quasi-collimated.

[0042] In some embodiments, the display system utilizes a light source configured to output light from multiple distinct light output locations. For example, the light source may comprise multiple selectively activated light-emitting regions, each region being a discrete light output location. The amount of parallax difference between intrapupillary images can be varied by varying the light output location for each image. It should be understood that light from a given light output location may propagate along one path through the display system to the eye, while light from different light output locations on the light source may propagate along different paths through the display system to the eye. As a result, spatial differences in light output locations can be translated into differences in the paths light takes to the eye. Different paths may correspond to different amounts of parallax difference. Advantageously, in some embodiments, the amount of parallax difference can be selected by selecting the amount of spatial displacement or separation between the light output locations of the light source.

[0043] In some embodiments, as described above, the light source may include multiple selectively activated light-emitting regions, each corresponding to a distinct light output location. The light-emitting regions may be arranged on a plane to form a 2D light emitter array. In some other embodiments, the light source may include a linear transfer lens, such as an F-theta (F-θ or F-tan θ) lens, a common or shared light emitter, and an actuator for directing light emitted by the light emitter along different paths through the F-theta lens. Light exits the light source at different locations through the F-theta lens, which focuses the exiting light onto an image plane. The light exiting the F-theta lens at different locations is also located at different locations on the image plane, which may be considered to provide a virtual 2D light emitter array. As a result, both the individual regions of the light emitter array and the locations through which light from the linear transfer lens passes through the image plane may be referred to herein as light output locations of the light source.

[0044] In some embodiments, the actuator may be part of a two-axis galvanometer comprising multiple (e.g., pairs) mirrors independently actuated on different axes to direct light from the light emitter along a desired propagation path. In some other embodiments, the light source may comprise a fiber scanner, and the actuator may be an actuator configured to move the fiber of the fiber scanner. The light source may also comprise or be in communication with a processing module that synchronizes the output of light by the light source with the location of the mirror or fiber and the intra-pupillary image to be displayed. For example, the mirror or fiber may move along a known path, and the light emitter may be controlled by the processing module to emit light when the mirror or fiber is in a position corresponding to a desired light output location for a particular intra-pupillary image (and the parallax difference associated with that image), as discussed further herein.

[0045] The display system may also include a spatial light modulator between the light source and the projection optics to direct light into the eye. The spatial light modulator may be configured to modulate light from the light source and encode image information in the light stream to form an intra-pupillary image. Preferably, the image is directed into the eye through projection optics that simultaneously present an image of the spatial light modulator plane at or near optical infinity or some other selected "home plane" and an image of the light source at or near the viewer's pupil. Thus, both the image content and precise parallax difference amount can be provided to the eye.

[0046] In some embodiments, the same spatial light modulator may be used to modulate light and form various intrapupillary images to be provided to the eye. In some such embodiments, active light output locations (light output locations from which light actively propagates at a given time) may be synchronized with modulation by the spatial light modulator. For example, activation of a light output location corresponding to one intrapupillary image may be synchronized, i.e., simultaneous, with activation of a display element within the spatial light modulator, where the display element is configured to form an intrapupillary image corresponding to a particular light-emitting region. Once another light output location corresponding to a second intrapupillary image is activated, an appropriate, possibly different, display element within the spatial light modulator may be activated to form that second intrapupillary image. Additional intrapupillary images may be formed by synchronizing activation of the light output locations with the image content provided by the spatial light modulator. This time-based sequential presentation of intrapupillary images to the eye may be referred to as temporal multiplexing or temporally multiplexed display of intrapupillary images. It should also be understood that an active or activated light output location is a location from which light actively propagates from a light source to a spatial light modulator used to form an intra-pupillary image.

[0047] In some other embodiments, spatial multiplexing may be utilized. In such embodiments, different areas (e.g., different pixels) of the spatial light modulator may be dedicated to forming different intra-pupillary images. Optical mechanisms may be provided between the spatial light modulator and the projection optics to direct light from different regions so that the light propagates through the projection optics in different directions. Examples of suitable optical mechanisms include lenslet arrays. As a result, different intra-pupillary images may be formed and presented to the eye simultaneously, with parallax difference determined by the location of the pixels forming the images and the optical mechanisms directing the propagation of light from those pixels. In some embodiments, because the parallax difference may be set using a combination of the spatial light modulator and the optical mechanisms, a light source without selectively activated light-emitting regions (e.g., point light sources) may be utilized to generate light for the display system.

[0048] In some other embodiments, both spatial and temporal multiplexing may be utilized. In such embodiments, the display system may include a light source with selectively activated light output locations in addition to the optical arrangement and formation of different intra-pupillary images in different areas of the spatial light modulator described above. Parallax difference may be provided using a combination of selective activation of light output locations and both the optical arrangement and simultaneous formation of different intra-pupillary images in different locations of the spatial light modulator.

[0049] In embodiments involving temporal multiplexing, the set of intrapupillary images for approximating a particular continuous wavefront are preferably injected into the eye so rapidly that the human visual system cannot detect that the images were presented at different times. Without being limited by theory, the visual system may perceive the images formed on the retina as simultaneously present within the flicker fusion threshold. In some embodiments, approximating a continuous wavefront may include sequentially injecting beams of light for each set of intrapupillary images into the eye, with the total duration for injecting all of the beams of light being less than the flicker fusion threshold, above which the human visual system would perceive the images as being separately injected into the eye. By way of example, the flicker fusion threshold may be approximately 1 / 60 of a second. It should be understood that each set of images may consist of a certain number of parallax views, e.g., two or more views, three or more views, four or more views, etc., all of which are presented within the flicker fusion threshold.

[0050] Preferably, the display system has a sufficiently small exit pupil that the depth of field provided by the light forming the individual intrapupillary images is substantially infinite and the visual system operates in an "open loop" mode, in which the eye cannot accommodate to the individual intrapupillary images. In some embodiments, the light beams forming the individual images occupy an area having a width or diameter of less than about 0.5 mm when incident on the eye. However, it should be understood that the light beams forming the set of intrapupillary images are, at least in part, non-overlapping and preferably define an area greater than 0.5 mm to provide sufficient information to the eye's lens to elicit a desired accommodation response based on the wavefront approximation formed by the wavefront of the light forming the intrapupillary images.

[0051] Without being limited by theory, the area defined by the set of beams of light can be considered to mimic a synthetic aperture through which the eye views a scene. It should be understood that viewing a scene through a sufficiently small pinhole in front of the pupil provides a nearly infinite depth of field. Given the small aperture of the pinhole, the eye's lens is not provided with adequate scene sampling to distinguish distinct depths of focus. As the pinhole enlarges, additional information is provided to the eye's lens, and natural optical phenomena allow a limited depth of focus to be perceived. Advantageously, the area defined by the set of beams of light and the corresponding set of parallaxly distinct intra-pupillary images can be made larger than a pinhole to produce an infinite depth of field, and the multiple intra-pupillary images can produce an approximation of the effect provided by the enlarged pinhole.

[0052] As discussed herein, in some embodiments, different angles at which light beams propagate toward the pupil may be provided using a light source having multiple selectively activated light output locations that output light to a spatial light modulator, which modulates the light and forms an image. It should be understood that light from different light output locations of the light source will follow different paths to the spatial light modulator, which in turn will follow different paths from the spatial light modulator to the output pupil of the projection optics and, therefore, to the viewer's eye. As a result, lateral displacement of the active light output locations translates into angular displacement of light exiting the spatial light modulator and ultimately propagating through the projection optics toward the viewer's pupil. In some embodiments, an increase in lateral displacement between activated light-emitting regions may be understood to translate into an increase in angular displacement as measured relative to the plane of the spatial light modulator. In some embodiments, each intra-pupil image may be formed by outputting light from a different light output location, thereby providing an angular displacement between the beams of light that form each of the images.

[0053] In some embodiments, the light sources and / or light output locations of the light sources may vary in position or jitter within a single parallax image (intra-pupillary image) display episode. For example, the light sources and / or light-emitting regions may be physically moved, and / or different light output locations (e.g., different light emitters of an array of light emitters) may be activated to display the intra-pupillary image while providing the desired positional variation. The rate of displacement or jitter may be higher than the update rate of the parallax image on the spatial light modulator. The jitter displacement may be in any direction, including rotation, depending on the desired perceptual effect.

[0054] In some embodiments, the display system may include a combiner eyepiece, which allows virtual image content to be overlaid with the viewer's view of the world or surrounding environment. For example, the combiner eyepiece may be an optically transparent waveguide that allows the viewer to see the world. Additionally, the waveguide may be utilized to receive, guide, and ultimately output light that forms an intrapupillary image to the viewer's eye. Because the waveguide may be positioned between the viewer and the world, the light output by the waveguide may be perceived to form a virtual image located on a depth plane within the world. Essentially, the combiner eyepiece allows the viewer to receive a combination of light from the display system and light from the world.

[0055] In some embodiments, the display system may also include an eye-tracking system to detect the viewer's gaze direction, allowing appropriate content to be selected based on where the viewer is looking.

[0056] Advantageously, by shifting the mechanism for providing a diverging wavefront from multiple discrete optical output structures, which create a wavefront with a specific associated divergence, to a single structure that can create an arbitrary amount of divergence, the physical size and complexity of the system can be reduced. That is, some of the output structures can be eliminated. In addition, placing virtual content on a larger number of depth planes is also possible, which would be practical if each depth plane required a dedicated structure to create a given wavefront divergence. This increase in the number of depth planes can provide a more realistic and comfortable viewing experience for the viewer. Additionally, in some embodiments, the light from each spatial light modulator pixel remains nominally collimated, thereby facilitating the integration of a projection system having the spatial light modulator with a combiner eyepiece that utilizes collimated pixel light.

[0057] Reference is now made to the drawings, wherein like reference numerals refer to like parts throughout.

[0058] As discussed herein, the perception of an image as “three-dimensional” or “3-D” can be achieved by providing a slightly different presentation of the image to each eye of a viewer. FIG. 2 illustrates a conventional display system for simulating a three-dimensional image for a user. Two distinct images 190, 200, one for each eye 210, 220, are output to the user. The images 190, 200 are spaced from the eyes 210, 220 by a distance 230 along the optical or z-axis parallel to the viewer's line of sight. The images 190, 200 are flat, and the eyes 210, 220 can focus on the images by assuming a single accommodative state. Such a 3-D display system relies on the human visual system to combine the images 190, 200 and provide the perception of depth and / or scale of the combined image.

[0059] However, it should be understood that the human visual system is more complex and providing a realistic perception of depth is more difficult. For example, many viewers of conventional “3-D” display systems find such systems uncomfortable or may not perceive any sense of depth at all. Without being limited by theory, it is believed that a viewer of an object may perceive the object as “three-dimensional” due to a combination of vergence and accommodation. Vergence movement of two eyes relative to each other (e.g., eye rotation, in which the pupils move toward or away from each other to converge the lines of sight of the eyes and fixate on an object) is closely linked to the focusing (or “accommodation”) of the eye's lens and pupil. Under normal conditions, changing the focus of the eye's lens or accommodating the eye to change focus from one object to another at different distances will automatically produce a matching change in convergence to the same distance, a relationship known as the "accommodation-vergence reflex" and pupil dilation or constriction. Similarly, a change in convergence will, under normal conditions, induce a matching change in accommodation of lens shape and pupil size. As described herein, many stereoscopic or "3-D" display systems display a scene using slightly different presentations (and therefore slightly different images) to each eye so that a three-dimensional perspective is perceived by the human visual system. However, such systems, among other things, simply provide different presentations of a scene, but are uncomfortable for many viewers because they work against the "accommodation-vergence reflex" when the eyes view all image information in a single accommodated state. Display systems that offer better matching between accommodation and convergence-divergence movements may create a more realistic and comfortable simulation of three-dimensional images and contribute to increased wearing duration.

[0060] FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. Referring to FIG. 3 , objects at various distances from the eyes 210, 220 on the z-axis are accommodated by the eyes 210, 220 so that the objects are in focus. That is, the eyes 210, 220 assume a particular accommodated state and focus on objects at different distances along the z-axis. As a result, a particular accommodated state can be said to be associated with a particular one of the depth planes 240 having an associated focal length such that an object or portion of an object at a particular depth plane is in focus when the eye is in the accommodated state for that depth plane. In some embodiments, a three-dimensional image can be simulated by providing different presentations of images for each eye 210, 220, the presentations of the images also being different for different depth planes. While shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 210, 220 may overlap, for example, as the distance along the z-axis increases. Additionally, although shown as flat for ease of illustration, it should be understood that the contour of the depth plane may be curved in physical space so that all features within the depth plane are in focus with the eye in a particular accommodative state.

[0061] The distance between an object and the eye 210 or 220 can also change the amount of divergence of light from the object as viewed by that eye. Figures 4A-4C illustrate the relationship between distance and divergence of light rays. The distance between the object and the eye 210 is represented by decreasing distances R1, R2, and R3. As shown in Figures 4A-4C, light rays become more divergent as the distance to the object decreases. As the distance increases, the light rays become more collimated. In other words, the light field generated by a point (an object or part of an object) can be said to have a spherical wavefront curvature that is a function of the distance the point is from the user's eye. The curvature increases as the distance between the object and the eye 210 decreases. As a result, the divergence of light rays at different depth planes also differs, and the divergence increases as the distance between the depth plane and the viewer's eye 210 decreases. While only a single eye 210 is illustrated in FIGS. 4A-4C and various other figures herein for clarity of illustration, it should be understood that the discussion regarding eye 210 may apply to both eyes 210 and 220 of a viewer.

[0062] Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth can be achieved by providing the eye with different representations of images corresponding to each of these limited number of depth planes. The different representations may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features for a scene located on the different depth planes and / or based on the observation of different image features on different depth planes that are out of focus.

[0063] Because each depth plane has an associated wavefront divergence, some displays may utilize a waveguide with an optical power to output light with a divergence corresponding to that depth plane to display image content that appears to be at that depth plane. Multiple similar but differently optically powered waveguides may be utilized to display image content at multiple depth planes. For example, such a system may utilize multiple such waveguides formed in a stack. FIG. 5 illustrates an example of a waveguide stack for outputting image information to a user. Display system 250 includes a stack of waveguides 260 that may be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310 for outputting image information. Image injection devices 360, 370, 380, 390, 400 may be utilized to inject light containing image information into the waveguides 270, 280, 290, 300, 310. Each waveguide 270, 280, 290, 300, 310 may include structures (e.g., optical gratings and / or lenses 570, 580, 590, 600, 610, respectively) that provide refractive power such that each waveguide outputs light with a predetermined amount of wavefront divergence corresponding to a particular depth plane. Thus, each waveguide 270, 280, 290, 300, 310 places image content on an associated depth plane determined by the amount of wavefront divergence provided by that waveguide.

[0064] However, it should be understood that a one-to-one correspondence between waveguides and depth planes can lead to bulky and heavy devices in systems where multiple depth planes are desired. In such embodiments, multiple depth planes would require multiple waveguides. Additionally, if a color image is desired, an even larger number of waveguides may be required, as each depth plane may have multiple corresponding waveguides, one waveguide per primary color may be required to form the color image.

[0065] Advantageously, various embodiments may provide a simpler display system that approximates the desired continuous wavefront by using discrete light beams that form intra-pupillary images that present different parallax views of an object or scene.

[0066] 6A , a pre-accommodation state of an eye 210 and a post-accommodation state in response to receiving a continuous input wavefront 1000 are illustrated. Illustration a) shows the pre-accommodation state before the visual system focuses the wavefront 1000 onto the retina 211. Notably, the focal point 212 is not on the retina 211. For example, the focal point 212 may be in front of the retina 211, as shown. Illustration b) shows the post-accommodation state after the human visual system flexes the pupil musculature of the viewer's eye 210 to focus the wavefront 1000 onto the retina 211. As shown, the focal point 212 may be on the retina 211.

[0067] It has been discovered that a continuous wavefront, such as wavefront 1000 in FIG. 6A, can be approximated using multiple wavefronts. FIG. 6B illustrates a pre-accommodation state of eye 210 and a post-accommodation state in response to receiving a piecewise approximation of continuous wavefront 1000 in FIG. 6A. Illustration a) of FIG. 6B shows the pre-accommodation state of eye 210, and illustration b) shows the post-accommodation state. The approximation may be formed using multiple constituent wavefronts 1010a, 1010b, and 1010c, each associated with a separate beam of light. As used herein, reference numerals 1010a, 1010b, and 1010c may refer to both a light beam and the associated wavefronts of that light beam. In some embodiments, constituent wavefronts 1010a and 1010b may be planar wavefronts, such as those formed by a collimated beam of light. As shown in illustration b), wavefront approximation 1010 formed by constituent wavefronts 1010a and 1010b is focused by eye 210 onto retina 211, with focal point 212 on retina 211. Advantageously, the pre- and post-accommodation conditions are similar to those produced by continuous wavefront 1000 shown in FIG.

[0068] It should be understood that a continuous diverging wavefront may also be formed using an optical projection system. FIG. 7A illustrates the eye accommodating a diverging wavefront emanating from a finite focal length virtual image provided by a projection system. The system includes a spatial light modulator 1018 and projection optics 1020 with a focal length “F” and an external stop. An image may be formed by the spatial light modulator 1018, and light from the spatial light modulator 1018 containing image information may be directed to the eye 210 through the projection optics 1020. As shown in FIG. 7A, the spacing (less than F) between the spatial light modulator 1018 and the projection optics 1020 may be selected so that a diverging wavefront 1000 is output toward the eye 210. As discussed above with respect to FIG. 6A, the eye 210 may then focus the wavefront 1000 onto the retina 211.

[0069] Figure 7B illustrates a system for forming an approximation of the diverging wavefront of Figure 7A using wavefront segments formed by an infinity-focused virtual image. As described above, the system includes a spatial light modulator 1018 and projection optics 1020. The spatial light modulator 1018 forms two images that are shifted relative to one another. The spatial light modulator 1018 is located a distance F from the back focal plane of the projection optics 1020, which has a back focal length of F. A light beam 1010a, containing image information for a first image, propagates through the projection optics 1020 into the eye 210. A light beam 1010b, containing image information for a second image, follows a different path through the projection optics 1020 into the eye 210. As discussed herein, light beams 1010a and 1010b may be emitted from different regions of a light source (not shown), causing them to illuminate spatial light modulator 1018 from different angles, thus causing the images formed by light beams 1010a and 1010b to be spatially shifted relative to one another. Light beams 1010a and 1010b propagate from the spatial light modulator along paths through projection optics 1020 and into eye 210 such that they define angular ranges for each light beam that match the angular range of diverging wavefront 1000 (FIG. 7A). It should be understood that the angular separation between light beams 1010a and 1010b increases with increasing approximated wavefront divergence. In some embodiments, projection optics 1020 and the spacing between spatial light modulator 1018 and projection optics 1020 are configured such that light beams 1010a and 1010b are collimated, respectively.

[0070] Referring now to FIG. 8, an example of parallax views forming the diverging wavefront approximation of FIG. 7B is illustrated. It should be understood that light beams 1010a, 1010b, and 1010c each form distinctly different images of a view of the same object or scene from slightly different viewpoints corresponding to different locations of the image in space. As shown, the images may be injected into the eye 210 sequentially at different times. Alternatively, the images may be injected simultaneously if the optical system allows, or the images can be injected as a group, as discussed herein. In some embodiments, the light forming all of the images is injected into the eye 210 for a total duration that is less than the viewer's flicker fusion threshold. For example, the flicker fusion threshold may be 1 / 60 of a second, and light beams 1010a, 1010b, and 1010c are all injected into the eye 210 for a duration that is less than the flicker fusion threshold. The human visual system therefore integrates all of these images, which appear to the eye 210 as if light beams 1010a, 1010b, and 1010c were simultaneously injected into the eye 210. Light beams 1010a, 1010b, and 1010c therefore form a wavefront approximation 1010.

[0071] 9, an example of a display system 1001 is illustrated that includes a projection system 1003 for forming the diverging wavefront approximation 1010 of FIG. 7B. The projection system 1003 includes a light source 1026 configured to modulate light and output light 1010a′ and 1010b′ to a spatial light modulator 1018, which modulates the light and forms images showing slightly different parallax views of the same object or scene. The modulated light with the image information then propagates through relay / projection optics 1020 and is output by the relay / projection optics 1020 as light beams 1010a and 1010b into the eye 210. The projection system 1003 may also include a lens structure 1014, which may be configured to convert the spatial difference in the emission of light 1010a′ and 1010b′ into an angular difference in the propagation of that light to the spatial light modulator 1018. The projection system 1003 may further include a polarizing beam splitter 1016 configured to 1) direct light from the light source 1026 to the spatial light modulator 1018 and 2) allow modulated light from the spatial light modulator 1018 to propagate back through the beam splitter 1016 to the relay / projection optics 1020. In some embodiments, the display system 1001 may include an eye tracking device 1022, e.g., a camera, configured to monitor the gaze of the eyes. Such monitoring may be used to determine the direction in which the viewer is looking, which may be used to select appropriate image content with respect to that direction. Preferably, the eye tracking device 1022 tracks both of the viewer's eyes, or each eye includes its own associated eye tracking device. As a result, the vergence and divergence movements of the viewer's eyes may be tracked, the convergence point of the eyes may be determined, and the direction and distance at which the eyes are pointed may be determined.

[0072] It should be understood that the light 1010a' and 1010b' may be output at different times by the light source 1026, as discussed herein, the spatial light modulator 1018 may form different parallax views at different times using the light 1010a' and 1010b', and the resulting light beams 1010a and 1010b may be injected into the eye 210 at different times.

[0073] Continuing with reference to FIG. 9 , light source 1026 may be a 2D light source having a plurality of selectively activated light output locations arranged substantially on a plane. In some embodiments, the selectively activated light output locations may be selectively activated light-emitting regions. For example, light source 1026 may be a light-emitting diode (LED) array or a spatial light modulator (e.g., a digital micromirror device such as a digital light processing (DLP) device, an LCOS device, or the like) containing an array of discrete units or light emitters that output light. Examples of LED arrays include organic light-emitting diode (OLED) arrays and inorganic light-emitting diode (ILED) arrays. In some embodiments, individual light-emitting diodes and / or light modulators within light source 1026 may constitute a light-emitting region. In some other embodiments, a group of light-emitting diodes and / or light modulators may form a light-emitting region. In such embodiments, there may be some overlap between the light-emitting diodes and / or light modulators of different light-emitting regions, but the regions may be considered distinct because the overlap is not complete.

[0074] In some other embodiments, the light source 1026 may be configured to focus light onto an image plane, effectively providing a virtual 2D light source at that image plane. Different locations on the image plane may be considered to be different light output locations, which may be activated by steering light from the light emitter using actuated mirrors or fiber scanners to direct light through those locations on the image plane. Further details regarding such virtual 2D light sources are provided in the discussion of Figures 19 and 20 below.

[0075] In some embodiments, an example of spatial light modulator 1018 includes a liquid crystal on silicon (LCOS) panel. As another example, in some other embodiments, spatial light modulator 1018 may comprise a transmissive liquid crystal panel such as a DLP or a MEMS device.

[0076] Continuing with reference to FIG. 9 , the display system 1001 may also include a control system 1024 for determining the timing and type of image content provided by the display system. In some embodiments, the control system 1024 comprises one or more hardware processors with memory that stores programs for controlling the display system 1001. For example, the system 1024 may be configured to control the activation of light-emitting regions of the light source 1026, the actuation of individual pixel elements of the spatial light modulator 1018, and / or the display system 1001's interpretation and response to data received from the eye tracking device 1022. Preferably, the system 1024 includes a calculation module 1024a configured to receive input related to a desired depth plane or wavefront divergence and calculate appropriate light-emitting regions to activate to form a parallax view with an appropriate amount of difference for the desired depth plane or wavefront divergence. Additionally, the calculation module 1024a may be configured to determine appropriate actuation of pixels of the spatial light modulator 1018 to form an image of the desired parallax view. The system 1024 may also include a synchronization module 1024b configured to synchronize the activation of specific light-emitting regions of the light source 1026 with the modulation of light by the spatial light modulator 1018 to form images to provide parallax views corresponding to those activated light-emitting regions. Additionally, the system 1024 may include an eye tracking module 1024c that receives input from the eye tracking device 1022. For example, the eye tracking device 1022 may be a camera configured to image the eye 210. Based on images captured by the eye tracking device 1022, the eye tracking module 1024c may be configured to determine the orientation of the pupil and extrapolate the line of sight of the eye 210. This information may be electronically communicated to the calculation module 1024a. The calculation module 1024a may be configured to select image content based on the line of sight or line of sight of the eye 210 (and preferably also based on the line of sight or line of sight of the viewer's other eye).

[0077] Because the light source 1026 may include an array of discrete light emitters, the size and shape of the light-emitting regions formed by the light emitters can be varied as desired by activating selected ones of the light emitters. FIG. 10 illustrates examples of sizes, shapes, and distributions for the light-emitting regions. It should be understood that the light and dark areas in the figure indicate different emitting regions activated for different parallax views. Example a) illustrates elongated light-emitting regions that are horizontally spaced apart, which may be desirable for horizontal parallax-only driven accommodation. Example b) illustrates a circular light-emitting region with both horizontal and vertical displacement. Example c) illustrates a light-emitting region with a luminance decrease. Example d) illustrates overlapping light-emitting regions. Example e) illustrates light-emitting regions that form an array. As illustrated by the illustrated examples, the light source 1026 ( FIG. 9 ) may include binary (simply on and off) light emitters and / or light emitters incorporating grayscale (selectively emitting light of varying intensities). In some embodiments, light source 1026 may include elements that switch at very high rates, including rates that exceed the disparity switching rate for system 1001. For example, light source 1026 may have light output elements that switch their light output on and off at a rate higher than the rate at which disparity (endo-pupillary) images are switched, in embodiments in which different intra-pupillary images are displayed at different times.

[0078] Referring again to FIG. 9 , in some embodiments, the control system 1024 may include two sections: 1) brightfield generation; and 2) resolution field optimization. To approximate a wavefront, as discussed herein, an appropriate image is displayed on the spatial light modulator 1018 for each activated light-emitting region of the light source 1026. It should be understood that these images are created during the brightfield generation step, in which the 3D scene is rendered from multiple slightly offset viewpoints corresponding to slight shifts in the activated light-emitting regions. For example, to display a 5×5 brightfield, the 3D scene would be rendered 25 times from 25 different viewpoints arranged in a grid pattern. The locations of the viewpoints within the grid pattern correspond to the locations of the activated light source regions, and the rendered images would correspond to the image formed by the spatial light modulator.

[0079] It may be desirable to increase the brightness of the image formed by the spatial light modulator 1018. Advantageously, utilizing a light source 1026 comprising an array of light emitters allows for the formation of light-emitting regions having various shapes and sizes, which may be utilized to increase brightness. In some embodiments, brightness may be increased by increasing the size of the activated light-emitting regions without significantly changing the image formed by the spatial light modulator 1018. The calculation module 1024a may be configured to determine the size and shape of the activated light-emitting regions using a resolved field-of-view optimization. The module 1024a may be configured to take an input focal stack and create a series of patterns to be displayed on the spatial light modulator 1018 as well as on the light source 1026, where the patterns are configured to create a desired approximation to the focal stack in a least-squares sense. The optimization takes advantage of the fact that slight shifts in viewpoint do not significantly change the perceived image, allowing the same image to be displayed on the spatial light modulator 1018 while generating a light-emitting region pattern using illumination from a larger area on the light source 1026.

[0080] The optimization problem can be formulated as a non-convex optimization problem, given below: [ka] where the projection operator ρ performs a linear transformation (using a shift-and-add algorithm) from a 4D bright field to a 3D focal stack. The problem is a non-negative matrix decomposition embedded in a deconvolution problem. An algorithm for solving the problem uses the Alternating Direction Method of Multipliers (ADMM). Additional details regarding an exemplary method for solving the problem are discussed in Appendix I. It should be understood that module 1024a is configured to actively calculate the appropriate size and shape of the luminous region in real time based on the parallax views to be formed by the spatial light modulator 1018.

[0081] In some other embodiments, the optimization problem may be formulated as a slightly different non-convex optimization problem, as given below. [ka] where A and B represent the pattern displayed on the spatial light modulator (e.g., light source 1026 and spatial light modulator 1018 to form an image), y is the desired output of the algorithm, a target 4D bright field, and AB' is an operator for combining the spatial light modulator patterns to simulate the 4D bright field emitted by a physical display when A and B are shown on the modulator. The problem is a non-negative matrix decomposition. The algorithm for solving this problem uses iterative optimization techniques to refine A and B from a random initial guess.

[0082] Continuing with reference to FIG. 9 , it should be understood that the flicker fusion threshold of the human visual system imposes a time constraint on the number of images that can be injected into the eye 210 while still being perceived as being injected simultaneously. For example, the processing bandwidth of the control system 1024 and the ability to switch between the light emitting regions of the light source 1026 and the light modulators of the spatial light modulator 1018 may limit the number of images that can be injected into the eye 210 within the duration allowed by the flicker fusion threshold. Given this finite number of images, the control system 1024 may be configured to make a selection regarding the images to be displayed. For example, within the flicker fusion threshold, the display system may be required to inject a set of parallaxly distinct intrapupillary images into the eye, and each parallax view may in turn require a different primary color image to form a full-color image. In some embodiments, the formation of a full-color image using primary color images is dependent on the elucidation of the desired accommodation response. For example, without being limited by theory, it may be possible to induce a desired accommodation response with monochromatic light. In such a case, the parallaxly distinct intrapupillary images used to elicit an accommodation response would be in only one color. As a result, it would not be required to form parallaxly distinct intrapupillary images using light of other colors, thereby freeing up time within the flicker fusion threshold for other types of images to be displayed. For example, a set of larger parallaxly distinct intrapupillary images may be generated to better approximate the wavefront.

[0083] In some other embodiments, the control system 1024 may be configured to spend less time within the flicker fusion threshold to display images of colors of light to which the human visual system is less sensitive. For example, the human visual system is less sensitive to blue light than to green light. As a result, the display system may be configured to generate images formed using more green light than images formed using blue light.

[0084] Referring now to FIG. 11A, another embodiment of a projection system 1003 for forming the diverging wavefront approximation of FIG. 7B is illustrated. Preferably, the projection system produces a relatively long depth of field, which may be controlled by a limiting aperture within the system. Without being limited by theory, it is believed that a projection system providing an image to the eye with an effective pupil diameter of approximately 0.5 mm forces the human visual system to operate in an "open-loop" mode because the eye cannot accommodate such an image. By providing an image with such an effective pupil diameter, the display system reduces the spot size on the retina for an image focused at infinity.

[0085] Continuing with reference to FIG. 11A , projection system 1003 forms images with parallax difference, as discussed herein. Images can be rapidly alternated to the viewer's eyes at a rate higher than the perceptual persistence of the human visual system (e.g., >60 Hz). As discussed herein, the illustrated projection system 1003 simultaneously produces images of an illumination source at a finite conjugate plane and an image of a pixel (image) source at infinity. In addition, selectively activated light-emitting regions 1026 a and 1026 b are spaced apart to produce a displacement of the optical system pupil, aligning the parallax images relative to each other within the viewer's pupil.

[0086] 11B, an example of the range of depth planes provided by the projection system of FIG. 11A is illustrated. The range extends from a far plane at optical infinity to a near plane closer to the eye 210. The far plane at optical infinity may be provided by collimated light beams 1010a and 1010b. The near plane may be provided using spatially displaced activated light-emitting regions as disclosed herein and may be understood to be the nearest depth plane provided by the display system. In some embodiments, the perceived proximity of the near plane to the eye 210 may be determined by the maximum parallax difference between light beams 1010a and 1010b, which may be determined by the maximum distance at which the display system allows selectively activated light-emitting regions 1026a and 1026b to be separated while still forming a sharp image of the light source 1026 at or near the viewer's pupil.

[0087] Advantageously, the use of light source 1026, comprising multiple discrete, selectively activated light emitters as discussed herein, provides the ability to produce a wide range of pupil or perceived image shapes, brightness profiles, and arrays (through manipulation of illumination source size, shape, and position) to achieve various depth-of-field effects. Light source 1026 also advantageously provides the ability to flexibly and bidirectionally change pupil shape as desired to provide high luminous efficiency while driving accommodation, and to accommodate horizontal parallax only, full parallax, or other combinations of parallax.

[0088] 12 , an example of a light source configuration for projection system 1003 is illustrated. Light source 1026 includes a single fixed illuminator 1028 and a spatial light modulator 1032 for adjusting the output of light from the illuminator 1028 to the spatial light modulator 1018 for forming an image. Light source 1026 may also include a condenser / collimator lens to direct light from the illuminator 1028 to the spatial light modulator 1032. The spatial light modulator 1032 may include pixels and / or shutters that allow or block light from passing therethrough, as desired. It should be understood that the pixels and / or shutters can be actuated to allow light to pass therethrough, and that the areas through which light passes are considered to be emissive regions (e.g., emissive regions 1026a and 1026b).

[0089] 13A, an example of a projection system 1003 for placing virtual objects at a default depth plane less than optical infinity is illustrated. As shown, the projection optics 1020 may have a focal length "F," and the spatial light modulator may be positioned less than F, which biases the system 1003 to have a far depth plane less than optical infinity by diverging the light beams 1010a and 1010b. The amount by which the light beams 1010a and 1010b diverge may be determined by the position of the spatial light modulator 1018 relative to the projection optics 1020; the closer the spacing, the greater the divergence. Because a certain amount of divergence is expected by default due to the spacing, in some embodiments, the size of the light-emitting regions 1026a and 1026b per intra-pupil image may be scaled up (e.g., by increasing the number of LEDs, increasing the number of light source illumination pixels activated to illuminate the spatial light modulator when forming the intra-pupil image, etc.), and the exit pupil associated with each intra-pupil image may be larger than 0.5 mm. As a result, the visual system may not function in open-loop mode. In some embodiments, the size of the light-emitting regions 1026a and 1026b may be set by a control system 1024 ( FIG. 9 ), which may be programmed to vary the size of the light-emitting regions 1026a and 1026b based on the desired default depth plane for the system 1003. In addition, the width of the cross section of the light beam for forming each intra-pupil image is preferably large enough relative to the optical structure of the projection optics 1020 so that the projection optics 1020 acts on the light to provide the desired divergence.

[0090] 13B, an example of a range of depth planes provided by the projection system of FIG. 13A is illustrated. The illustrated range extends from a far plane at a distance D less than optical infinity to a near plane relative to the eye 210. The far plane may be set by appropriate selection of the position of the spatial light modulator relative to the projection optics 1020. The near plane may be provided as disclosed above with respect to FIG. 11B.

[0091] 14 , an example of a projection system 1003 configured for spatially multiplexed display of intra-pupil images is illustrated. Rather than relying on spatial displacement between light-emitting regions to provide the desired parallax difference, the parallax difference may be provided by utilizing different areas of a spatial light modulator 1018 to form different intra-pupil images. The optical mechanism 1019 is configured to direct light from each of these different areas at different angles toward the projection optics 1020, which outputs light beams 1010 a and 1010 b toward the viewer's eye (not shown). In some embodiments, the areas of the spatial light modulator 1018 for forming the different intra-pupil images may be interleaved. For example, pixels providing image information for different intra-pupil images may be interleaved with one another. The optical mechanism 1019 may be configured to convert different locations (e.g., different pixels) from which the optical mechanism 1019 receives light to different angles at which the light from the pixels enters the projection optics 1020. In some embodiments, the optical mechanism 1019 P rhythm 、 and / or Lenslet arrays, etc. It may include a lens structure.

[0092] It should be understood that different non-overlapping regions of the spatial light modulator 1018 may be dedicated to providing image information for different intra-pupillary images. Because these regions are distinct from one another, they may, in some embodiments, be activated simultaneously. As a result, multiple intra-pupillary images may be presented to the eye simultaneously. This may advantageously reduce the speed at which the spatial light modulator 1018 is required to refresh images. As discussed above, to provide the perception that all images in a set of intra-pupillary images exist simultaneously to approximate a continuous wavefront, these images must all be presented within the flicker fusion threshold. In some embodiments, all or multiple images in a set of intra-pupillary images are presented simultaneously in different regions of the spatial light modulator such that rapid sequential display of these simultaneously presented images is not required for the human visual system to perceive the images as existing simultaneously.

[0093] As shown, light source 1028 provides light through lens structure 1014 to illuminate spatial light modulator 1018. In some embodiments, light source 1028 may be a single fixed illuminator without any selectively activated light-emitting regions.

[0094] In some other embodiments, the light source may include selectively activated light-emitting regions, which may advantageously provide additional control of parallax difference. Thus, the projection system may utilize both spatial and temporal multiplexing. Referring now to FIG. 15 , an example of a projection system 1003 configured for spatial and temporal multiplexing of intrapupil images is illustrated. The projection system 1003 may include a light source 1026, which may include selectively activated light-emitting regions, e.g., regions 1026a and 1026b. As discussed herein, spatial displacement between the light-emitting regions may be utilized to provide parallax difference for the alternately output light beams 1010a, 1010b, 1010c, and 1010d. Additionally, as discussed above with respect to FIG. 14 , the projection system 1000 may include an optical mechanism 1019 between the spatial light modulator 1018 and the projection optics 1020. The spatial light modulator 1018 and the optical mechanism 1019 may cooperate to provide spatial multiplexing. Thus, illumination of spatial light modulator 1018 with a single light-emitting region can produce multiple intra-pupil images. For example, activation of light-emitting region 1026a illuminates spatial light modulator 1018, which simultaneously generates image information for two intra-pupil images, with the light beams for each image directed in different directions by optical mechanism 1019. The light enters projection optics 1020 and exits as light beams 1010b and 1010d to form two distinct intra-pupil images. Similarly, subsequent activation of light-emitting region 1026b results in light beams 1010a and 1010d to form two other intra-pupil images.

[0095] Referring to both Figures 14 and 15, in some embodiments, as discussed above with respect to Figures 11A-11B and 13A-13B, the location of the spatial light modulator 1018 and optical mechanism 1019 relative to the projection optics 1020 may be selected to provide a desired default "home" depth plane, which may be less than optical infinity.

[0096] The projection system 1003 shown in various figures herein may be part of a hybrid system that utilizes a single finite focal length eyepiece, employing parallax-driven accommodation to place objects at a non-infinity depth plane as a default, while placing virtual objects at other depth planes. For example, the projection system 1003 may be configured to have a default depth plane at 0.3 dpt or 0.5 dpt, which may be close enough to optical infinity to be within the tolerance of the human visual system for accommodation-vergence-divergence mismatch. For example, without being limited by theory, it is believed that the human visual system can comfortably tolerate displaying content from optical infinity on a 0.3 dpt depth plane. In such a system, the beams of light 1010a and 1010b would have a wavefront divergence corresponding to the default depth plane. Advantageously, such a configuration may reduce the computational load on a processor (e.g., a graphics processing unit) within the display system, which may provide benefits such as lower power consumption, reduced latency, and increased processor options, among other benefits.

[0097] 16 , an example of a projection system 1003 is illustrated that includes a pupil relay combiner eyepiece 1030 for superimposing image content onto a user's view of the world. Preferably, the eyepiece 1030 is optically transmissive, allowing light from the world to propagate through the eyepiece and into the viewer's eye 210. In some embodiments, the eyepiece 1030 includes one or more waveguides having an internal coupling optical element 770 and an external coupling optical element 800. The internal coupling optical element 770 receives light from the projection optics 1020 and redirects it through the eyepiece 1030 by total internal reflection to propagate to the external coupling optical element 800. The external coupling optical element 800 outputs light to the viewer's eye 210. Advantageously, the eyepiece 1030 preserves all of the image attributes provided by the projection system 1003 so that rapid switching of parallax views is accurately depicted through the eyepiece 1030.

[0098] The incoupling optical element 770 and the outcoupling optical element 800 may be refractive or reflective structures. Preferably, the incoupling optical element 770 and the outcoupling optical element 800 are diffractive optical elements. Examples of diffractive optical elements include surface relief features, volume phase features, metamaterials, or liquid crystal polarization gratings.

[0099] It should be understood that the outcoupling optical element 800 or other optical elements forming part of the eyepiece 1030 may be configured to have a refractive power. In some embodiments, the refractive power may be selected to correct refractive errors of the eye 210, including refractive errors such as myopia, hyperopia, presbyopia, and astigmatism.

[0100] 17 , an example of a projection system 1003 is illustrated that includes an eye tracking system 1022 and a combiner eyepiece 1030 with a pupil expander 1034. The pupil expander replicates the projection system pupil across the eyepiece 1030. Because the pupil expander 1034 replicates the projection system pupil across a large area that may be traversed by the viewer's pupil through eye movement, the image formed by the spatial light modulator 1018 and the location of the light-emitting region of the light source 1026 can be updated in real time based on input from the eye tracking system 1022. Advantageously, this configuration allows for a larger eyebox for more comfortable viewing and reduces limitations on the relative positioning of the eye and combiner and variations in interpupillary distance.

[0101] Referring now to FIG. 18 , an example projection system 1003 is illustrated, comprising an eye tracking system 1022 and a combiner eyepiece 1030 with a pupil expander 1035 configured to produce a non-infinity depth plane. In some embodiments, the non-infinity depth plane may be at 3 meters, resulting in a budgeted accommodation of approximately 2.5 meters to infinity. For example, given the human visual system's tolerance for accommodation-vergence mismatch, virtual content at a distance of approximately 2.5 meters from the viewer to infinity may be placed on the 3-meter depth plane with little discomfort. In such a system, parallax-different intrapupillary images may be used to drive accommodation for a narrower range of depth planes, potentially all closer to the viewer than a fixed “default” focal plane. In some embodiments, the system also incorporates an eye tracking system 1022 to determine the distance of the viewer's fixation, for example, based on the vergence angle of the viewer's eyes.

[0102] In some embodiments, the light source 1026 may be replaced with a virtual light source formed on the image plane of the optical projection system. The optical projection system may include an actuator capable of scanning a beam of light across an area on the image plane corresponding to the virtual light source. To mimic the ability to activate discrete light-emitting areas of the light source 1026, the output of light by the projection system is synchronized with movement of the actuator, causing light to be output to desired locations on the image plane at specific times. Preferably, the rate at which the actuator can scan the beam of light across the image plane is sufficiently high so that all desired light output locations on the image plane can be accessed during the time frame at which any given intra-pupillary image is displayed. For example, during the amount of time at which a particular intra-pupillary image is displayed, the actuator is preferably capable of scanning the beam of light across the area of ​​the image plane corresponding to the virtual 2D light source at least once, and preferably multiple times.

[0103] 19 illustrates a light source 2026 comprising mirrors for directing light propagation to different light output locations. The light source 2026 comprises a light emitter 2028 and mirrors 2030 and 2032, which are moved by actuators 2031 and 2033, respectively. Examples of the light emitter 2028 include an LED and a laser. In some embodiments, a fiber optic cable may transmit light from a remotely located light emitter. As shown, light 1010a', 1010b' propagates from the emitter 2028 to mirror 2032, which reflects the light to mirror 2030, which then reflects the light, propagating through lens 2034 and focusing it onto an intermediate image plane 1026'. Mirrors 2030 and 2032 may be part of a two-axis galvanometer, and actuators 2031 and 2033 rotate the mirrors along different axes, e.g., orthogonal axes, thereby allowing light to be directed to an area defined along two axes of image plane 1026′. In some embodiments, actuators 2031, 2033 may be motors. Lens 2034 may be a linear transfer lens, such as an F-theta (F-θ or F-tan θ) lens, and may be configured to focus light onto a flat image plane 1026′. In some embodiments, light rays 1010a′, 1010b′ propagate from image plane 1026′ in a similar manner as light would propagate from light source 1026 (see, e.g., FIG. 9 ). In some embodiments, light source 2026 may also include a collimating lens 2036 to collimate light emitted by light emitter 2028 before it reaches mirror 2032.

[0104] The light source 2026 also preferably includes or communicates with a processing module 2038 that controls the output of light from the light emitter 2028 and synchronizes it with the movement of the actuators 2031, 2033 and the intra-pupillary image to be formed. For example, the processing module 2038 may coordinate the movement of the mirrors 2032, 2030 with the emission of light from the light emitter 2028. In some embodiments, the mirrors 2032, 2030 are continuously rotated or pivoted back and forth by the actuators 2031, 2033 on the axes along which the mirrors are designed to move. The emission of light (e.g., pulses of light) by the light emitter 2028 is timed with this movement so that the light is directed to a desired location on the intermediate image plane 1026′ at a given moment, the location and time also being determined based on the intra-pupillary image to be displayed (e.g., activation of a particular light output location coincides in time with the display of an intra-pupillary image having a parallax difference associated with that particular light output location). In some embodiments, the emission of light from the light emitter 2028 is controlled by switching the light emitter 2028 between an on state and an off state (e.g., by supplying or not supplying power to the light emitter, respectively). In some other embodiments, the emission of light from the light emitter 2028 may be controlled mechanically using a physical switch that selectively allows or blocks light from reaching the image plane 1026'.

[0105] 20 , the light source 2026 may include a fiber scanner 2027. The fiber scanner 2027 may include a light emitter 2028 and an actuator 2040 that moves a fiber 2042. Light 1010a′, 1010b′ propagates from the end of the fiber 2042 through a lens 2034 and is focused onto an image plane 2026′. It should be understood that the actuator 2040 may move the fiber 2042 at a known speed along a predetermined path (e.g., a circular path). As a result, the processing module 2038 may be configured to synchronize the propagation of light from the end of the fiber 2042 with the movement of the fiber 2042 so that the light propagates from the fiber 2042 at a desired light output location and, thus, is synchronized with the intrapupillary image to be displayed.

[0106] As noted above, light source 2026 may replace light source 1026 in any of the display systems discussed. For example, light source 2026 can replace light source 1026 in projection system 1003 or display system 1001 of any of Figures 9, 11A, 12, 13A, and 15-18.

[0107] 21 , an example of eyepiece 660 (which may correspond to eyepiece 1030 of FIGS. 14-16 ) is illustrated that includes a stacked waveguide assembly for outputting light of different wavelengths corresponding to different primary colors. In some embodiments, the waveguide assembly includes waveguides 670, 680, and 690. Each waveguide includes an associated internal coupling optical element (which may also be referred to as the light input area on the waveguide), for example, internal coupling optical element 700 is disposed on a major surface (e.g., upper major surface) of waveguide 670, internal coupling optical element 710 is disposed on a major surface (e.g., upper major surface) of waveguide 680, and internal coupling optical element 720 is disposed on a major surface (e.g., upper major surface) of waveguide 690. In some embodiments, one or more of the incoupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the respective waveguide 670, 680, 690 (particularly, one or more of the incoupling optical elements are reflective polarizing optical elements). As shown, the incoupling optical elements 700, 710, 720 may be disposed on the upper major surface of that respective waveguide 670, 680, 690 (or on top of the next lower waveguide), and particularly, the incoupling optical elements are transmissive polarizing optical elements. In some embodiments, the incoupling optical elements 700, 710, 720 may be disposed within the body of the respective waveguide 670, 680, 690. In some embodiments, as discussed herein, the incoupling optical elements 700, 710, 720 are wavelength selective, such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of the respective waveguides 670, 680, 690, it should be understood that the internal coupling optical elements 700, 710, 720 may be located within other areas of the respective waveguides 670, 680, 690 in some embodiments.

[0108] As shown, the in-coupling optical elements 700, 710, 720 may be laterally offset from one another. In some embodiments, each in-coupling optical element may be offset to receive light without that light passing through another in-coupling optical element. For example, each in-coupling optical element 700, 710, 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400, as shown in FIG. 6 , and may be separated (e.g., laterally spaced) from the other in-coupling optical elements 700, 710, 720 so as to receive substantially no light from others of the in-coupling optical elements 700, 710, 720. In some embodiments, the in-coupling optical elements 700, 710, 720 are vertically aligned and not laterally offset.

[0109] Each waveguide also includes an associated optically dispersive element, for example, optically dispersive element 730 is disposed on a major surface (e.g., the top major surface) of waveguide 670, optically dispersive element 740 is disposed on a major surface (e.g., the top major surface) of waveguide 680, and optically dispersive element 750 is disposed on a major surface (e.g., the top major surface) of waveguide 690. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on the bottom major surfaces of associated waveguides 670, 680, 690, respectively. In some other embodiments, optically dispersive elements 730, 740, 750 may be disposed on both the top and bottom major surfaces of associated waveguides 670, 680, 690, respectively, or optically dispersive elements 730, 740, 750 may be disposed on different ones of the top and bottom major surfaces in different associated waveguides 670, 680, 690, respectively.

[0110] Waveguides 670, 680, 690 may be spaced apart and separated, for example, by gas, liquid, and / or solid layers of material. For example, as shown, layer 760a may separate waveguides 670 and 680, and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediately adjacent waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or greater, or 0.10 or less, than the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may function as cladding layers that promote TIR of light through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layers 760a, 760b are formed from air. Although not shown, it should be understood that the top and bottom of the illustrated set of waveguides 660 may include immediate cladding layers.

[0111] Continuing with reference to FIG. 21, light rays 770, 780, 790 are incident on and launched into waveguides 670, 680, 690 by projection system 1003 (FIGS. 9 and 11-16).

[0112] In some embodiments, the light rays 770, 780, 790 have different properties, such as different wavelengths or different wavelength ranges, which may correspond to different colors. Each of the internal coupling optical elements 700, 710, 720 deflects the incident light so that the light propagates through a respective one of the waveguides 670, 680, 690 by TIR.

[0113] For example, in-coupling optical element 700 may be configured to deflect light ray 770 having a first wavelength or range of wavelengths. Similarly, transmitted light ray 780 impinges on and is deflected by in-coupling optical element 710, which is configured to selectively deflect light of a second wavelength or range of wavelengths. Similarly, light ray 790 is deflected by in-coupling optical element 720, which is configured to selectively deflect light of a third wavelength or range of wavelengths.

[0114] 21 , in-coupled light rays 770, 780, 790 are deflected by in-coupling optical elements 700, 710, 720, respectively, and then propagate by TIR within waveguides 670, 680, 690, respectively. Light rays 770, 780, 790 then impinge on optically dispersive elements 730, 740, 750, respectively. Optically dispersive elements 730, 740, 750 deflect light rays 770, 780, 790 to propagate toward out-coupling optical elements 800, 810, 820, respectively.

[0115] In some embodiments, the optically dispersive elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or disperse light into the out-coupling optical elements 800, 810, 820 and may increase the beam or spot size of this light as it propagates into the out-coupling optical elements. In some embodiments, for example, if the beam size is already the desired size, the optically dispersive elements 730, 740, 750 may be omitted and the in-coupling optical elements 700, 710, 720 may be configured to deflect light directly into the out-coupling optical elements 800, 810, 820. In some embodiments, the out-coupling optical elements 800, 810, 820 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light into the viewer's eye 210 (FIGS. 15-16). It should be understood that the OPE may be configured to increase the dimensions of the eyebox in at least one axis, and the EPE may increase the eyebox in an axis that intersects, e.g., is perpendicular to, the axis of the OPE.

[0116] Thus, in some embodiments, the eyepiece 660 includes, for each primary color, a waveguide 670, 680, 690, an in-coupling optical element 700, 710, 720, an optically dispersive element (e.g., OPE) 730, 740, 750, and an out-coupling optical element (e.g., EP) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with an air gap / cladding layer between each one. The in-coupling optical element 700, 710, 720 redirects or deflects the incoming light into its waveguide (with different in-coupling optical elements receiving light of different wavelengths). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the example shown, light ray 770 (e.g., blue light) is polarized by the first in-coupling optical element 700 in the manner described above, then continues bouncing down the waveguide, interacting with the optically dispersive element (e.g., OPE) 730 and then the out-coupling optical element (e.g., EP) 800. Light rays 780 and 790 (e.g., green and red light, respectively) pass through waveguide 670, with light ray 780 incident on and deflected by in-coupling optical element 710. Light ray 780 will then bounce down waveguide 680, via TIR, to its optically dispersive element (e.g., OPE) 740 and then the out-coupling optical element (e.g., EP) 810. Finally, light ray 790 (e.g., red light) passes through waveguide 690 and impinges on the optically in-coupling optical element 720 of waveguide 690. The light in-coupling optical element 720 deflects the light ray 790 so that it propagates by TIR to the light dispersive element (e.g., OPE) 750 and then by TIR to the out-coupling optical element (e.g., EP) 820. The out-coupling optical element 820 then finally out-couples the light ray 790 to a viewer, who also receives the out-coupled light from the other waveguides 670, 680.

[0117] 22, there is illustrated an example of a wearable display system 60. Display system 60 may correspond to display system 1001 of FIG. 9, with a projection system 1003 for each eye of viewer or user 90.

[0118] Display system 60 includes a display 70 and various mechanical and electronic modules and systems to support the functionality of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and configured to position the display 70 directly in front of the user's 90 eyes. The display 70, in some embodiments, may be considered eyewear. In some embodiments, a speaker 100 is coupled to frame 80 and configured to be positioned adjacent to the user's 90 ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / adjustable sound control). In some embodiments, the display system may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphones are configured to allow a user to provide input or commands (e.g., voice menu command selections, natural language queries, etc.) to system 60 and / or enable audio communication with other persons (e.g., other users of similar display systems). The microphone may further be configured as an ambient sensor to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system may also include ambient sensor 120a, which may be separate from frame 80 and attached to the body of user 90 (e.g., on the head, torso, limbs, etc. of user 90). Ambient sensor 120a, in some embodiments, may be configured to obtain data characterizing a physiological state of user 90. For example, sensor 120a may be an electrode.

[0119] 22 , the display 70 is operably coupled by a communication link 130, such as wired or wireless connectivity, to a local data processing module 140, which may be mounted in a variety of configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, integrated into headphones, or otherwise removably attached to the user 90 (e.g., in a backpack-style configuration, in a belt-coupled configuration). Similarly, the sensor 120 a may be operably coupled to the local processor and data module 140 by a communication link 120 b, e.g., wired or wireless connectivity. The local processing and data module 140 may comprise a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) captured from sensors (e.g., which may be operatively coupled to frame 80 or otherwise attached to user 90), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyroscopes, and / or other sensors disclosed herein, and / or b) data obtained and / or processed using remote processing module 150 and / or remote data repository 160 (including data related to virtual content), possibly for processing or retrieval and passage to display 70. Local processing and data module 140 may be operatively coupled to remote processing module 150 and remote data repository 160 by communications links 170, 180, such as via wired or wireless communication links, such that these remote modules 150, 160 are operatively coupled to each other and available as resources to local processing and data module 140.In some embodiments, local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be mounted to frame 80 or may be a stand-alone structure that communicates with local processing and data module 140 by a wired or wireless communication path. In some embodiments, local processing and data module 140 may include one or more graphics processors and may correspond to control system 1024 (FIG. 9).

[0120] 22 , in some embodiments, remote processing module 150 may comprise one or more processors configured to analyze and process data and / or image information. In some embodiments, remote data repository 160 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a “cloud” resource configuration. In some embodiments, remote data repository 160 may include one or more remote servers, which provide information, e.g., information for generating augmented reality content, to local processing and data module 140 and / or remote processing module 150. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from the remote module. [Example]

[0121] Brightfield and focal stack decomposition Bright field and focal stack decompositions can be used to determine the light output of the display system 1001, including the output of the light sources 1026, 2026 and the spatial light modulator 1018. More details regarding the decompositions are discussed below.

[0122] 1. Focal Stack Decomposition The focal stack y is decomposed into a series of time-multiplexed patterns to be displayed on two spatial light modulators A and B, located at the pupil and image planes, respectively. In some embodiments, spatial light modulators A and B may correspond to light source 1026, 2026 and spatial light modulator 1018, respectively. All quantities are represented as a focal stack with a vertical resolution of m pixels, a horizontal resolution of n pixels, and s focal slices. [ka] is a single vector [ka] The tensor notation is then vectorized, so that it would be represented as: Bold symbols are used below for discrete vectors. Unless otherwise specified, different color channels are ignored and assumed to be independent. Table 1 provides an overview of the tensor notation and operators employed herein. [ka]

[0123] The spatial light modulator at image plane B also has a resolution of m×n pixels, but in addition, k time-multiplexed patterns can be displayed in rapid succession such that they will be perceptually averaged by the viewer. These spatiotemporal patterns are vectorized into a matrix such that every spatial pixel forms the row index of this matrix, and the k time steps are the column index of the matrix. [ka] Similarly, the pupil plane SLMA is expressed as the matrix [ka] where o is the total number of addressable SLM pixels in the pupil plane and the column index is again the time step.

[0124] Hence, the goal of decomposing the focal stack y into a set of time-multiplexed patterns can be written as a non-convex optimization problem. [ka] In the formula, the projection operator [ka] performs a linear transformation (using a shift-and-add algorithm) from a 4D bright field to a 3D focal stack. The problem is a non-negative matrix decomposition embedded in the deconvolution problem. The Alternating Direction Method of Multipliers (ADMM) (Boyd et al., 2001, “Distributed Optimization and statistical learning via the alternating direction method of multipliers”, Foundations and Trends in Machine Learning 3, 1, 1-122) may be used to solve it.

[0125] Equation 1 can be rewritten as an equivalence problem to put it into standard ADMM form. [ka] In the formula, the matrix [ka] is the operator [ka] in matrix form, where the operator [ka] simply vectorizes the matrix into a single 1D vector (eg, using column-major order as done by the software MATLAB available from MathWorks, Natick, Massachusetts).

[0126] The augmented Lagrangian method for this system is then formulated as follows: [ka] In scaled form, the present augmented Lagrangian method can be written as follows: [ka] During the ceremony, [ka] is.

[0127] The ADMM algorithm then consists of three separate updates (or proximity operators) that are performed iteratively as follows: [ka] Here, the operator [ka] reshapes the vector into a matrix, and the operator [ka] undoes what was done to vectorize the matrix. Equations 5-7 can be solved iteratively, each time using the most recent output from the previous step.

[0128] 1.1 Efficient z-update Equation 5 is an unconstrained linear problem that can be rewritten as the following single linear system of equations: [ka] Although the system is large, all operations can be expressed as matrix-free function handles, so matrices are not explicitly formed. A variety of different solvers can be used to solve the system for z. For example, MATLAB's very simple simultaneous algebraic reconstruction method (SART) may be utilized.

[0129] To increase computational efficiency, it would be desirable to derive a closed-form solution for the z-update, which could facilitate real-time implementation of the entire algorithm. One approach to deriving a closed-form solution starts with the normal equations for Equation 8. [ka] To find a closed-form solution for this, [ka] The inverse of is derived. Since P transforms the bright field to a focal stack and the Fourier slice theorem dictates that refocusing in the primary domain is slicing in the Fourier domain, a closed-form solution in the frequency domain can be derived. Using this insight, it can be written as follows: [ka] where s is the number of slices in the focal stack and O i is a focal slice in the 4D frequency domain i is a diagonal matrix representing the slicing operator on F 4D and [ka] represent the discrete 4D Fourier transform and its inverse, respectively.

[0130] The expected algebraic expression for the inverse matrix is: [ka] It should be appreciated that such closed-form solutions may provide a solution more quickly than iterative algorithms, since no iterations are required, which may also be preferred if sufficient computational resources are available.

[0131] 1.2 Efficient A,B-update The A,B-update (Equation 6) is a Nonnegative Matrix Factorization (NMF) problem. In this case, it is the simplest possible NMF problem. Standard solutions for this and more advanced NMF approaches are detailed in Section 2 below.

[0132] 1.3 Handling Color Channels In the above derivation, grayscale separation was assumed, or each color channel was assumed to be processed independently. In some cases, this may not provide a satisfactory approximation. For example, a two-color SLM may introduce color crosstalk, which is not modeled above. Additionally, in some embodiments, the display system may use a combination of a grayscale LCoS at the image plane and a color LED or OLED array at the pupil plane. In this case, all color channels are linked.

[0133] When considering linked color channels, neither the z-update nor the u-update changes; that is, they can be calculated independently per color channel per ADMM iteration. However, the matrix decomposition routines A and B-update do change. R,G,B / B R,G,B Instead of independently decomposing A, a single decomposition is performed simultaneously for all color channels of A (B contains no color channels in this case), as follows: [ka]

[0134] 2. Matrix decomposition transformation Nonnegative matrix decomposition is an approach to decomposing a matrix into a sum of nonnegative rank-1 matrices. The decomposition problem is nonconvex, and therefore the solution is not simple. The problem and possible solutions will be discussed here.

[0135] The problem can be stated as decomposing the matrix X into a sum of rank-1 matrices. [ka] During the ceremony, [ka] The sum of rank-1 matrices yields a rank-K approximation of the original matrix.

[0136] A minimum squared error solution to this problem can be found by optimizing the following objective function: [ka] where the Frobenius norm of the matrix is [ka] is given as:

[0137] 2.1 Alternating least squares approach Cost function [ka] is both nonlinear and nonconvex, with a certain number of local minima. Fixing either A or B makes the solution for the other matrix convex. An alternating least-squares approach, which is expected to converge if nonnegativity constraints are not considered, can be employed to solve this decomposition problem. To this end, each decomposition matrix is ​​updated while fixing the other in an alternating fashion. Individual updates are computed using gradient descent. [ka] During the ceremony, [ka] is the derivative of the cost function for each decomposition matrix, and α A,B is its individual step length. As will be shown in the following section, one approach to choosing the step length is to choose one such that the update rule is multiplicative. Before discussing step length, the gradient will be considered and can be given as follows: [ka] In matrix form, the gradient can be written as: [ka]

[0138] 2.2 Multiplicative Update Rules As mentioned above, the key to choosing the step length is that by combining it with the steepest descent direction, the additive update rule (Equation 15) can be written in simple multiplication. ij ≧0 and A, B are initialized with positive values, the multiplicative update rule provides that the decomposition matrix remains positive throughout the iterative update process. The following step lengths result in the multiplicative update rule: [ka] Combining equations 15, 17 and 18 yields: [ka] The following multiplicative update rule is a simplified version of Equation 19: [ka] Starting from an initial guess containing only positive values ​​(usually random noise), and assuming the data matrix X is non-negative, these update rules are expected to keep A and B positive throughout the iterative process. In practice, a small value is added to the divisor to avoid division by zero. [ka] 2.3 Weighted non-negative matrix factorization The multiplicative update rule is that matrix element x ij may be modified to include weighting for each. [ka] where W is a weighting matrix of the same size as X. 2.4 Projected NMF Projection NMF adds an additional projection matrix P to the objective function that remains fixed throughout the optimization procedure. [ka] Here, A and B remain invariant in their dimensions to the previous surjection, but [ka] teeth, [ka] The gradient with respect to this formula is: [ka] This can be written in matrix form as follows: [ka] If the step length is selected as follows: [ka] This leads to the following multiplicative update rule: [ka]

[0139] 2.5 Projection-weighted NMF For projective NMF, the weights may be added to the light field, resulting in the following update rule: [ka]

[0140] It should be understood that each of the processes, methods, and algorithms described herein and / or depicted in the figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and thereby be fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuitry, etc. Code modules may be compiled and linked into an executable program, installed within a dynamic link library, or written in an interpreted programming language. In some implementations, particular operations and methods may be performed by circuitry specific to a given function.

[0141] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special-purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real time. For example, a video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0142] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage, including hard drives, solid-state memory, random-access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. In some embodiments, the non-transitory computer-readable medium may be part of one or more of the local processing and data module (140), the remote processing module (150), and the remote data repository (160). The methods and modules (or data) may also be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) generated over various computer-readable transmission media, including wireless-based and wired / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored, persistently or otherwise, in any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0143] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code, comprising one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality may be combined, rearranged, added, deleted, modified, or otherwise changed from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the program components, methods, and systems described may generally be integrated together in a single computer product or packaged into multiple computer products.

[0144] In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the invention. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

[0145] Indeed, it should be understood that the systems and methods of the present disclosure each have several innovative aspects, no single one of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure.

[0146] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operative in a combination and may even be initially claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. No single feature or group of features is required or essential to every embodiment.

[0147] It should be understood that conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, in particular, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to imply that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps should be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprise," "include," "have," and the like are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense); thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a," "an," and "the," as used in this application and the appended claims, should be interpreted to mean "one or more" or "at least one," unless otherwise specified. Similarly, while operations may be depicted in the figures in a particular order, it should be recognized that such operations need not be performed in the particular order shown, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Furthermore, the figures may diagrammatically depict one or more exemplary processes in the form of a flowchart. However, other operations not depicted may be incorporated into the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, concurrently with, or during any of the depicted operations.Additionally, operations may be rearranged or reordered in other embodiments. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0148] Thus, the claims are not intended to be limited to the implementations shown herein but are to be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.

Claims

1. 1. A head-mountable display system, comprising: a light source configured to emit light from different locations; a spatial light modulator configured to receive the light emitted by the light source and modulate the received light to form virtual content, wherein the light emitted by the light source from the different locations is received and modulated in different respective areas of the spatial light modulator; a projection optical element configured to direct the modulated light from the spatial light modulator toward an eye of a wearer of the head-mountable display system, the projection optical element being further configured to collimate the light output from the spatial light modulator at different angles; and A head-mountable display system comprising:

2. The head-mountable display system of claim 1 , wherein the light sources are configured to emit the light from the different locations at different times.

3. The head-mountable display system of claim 1 , wherein the light source includes at least two light emitters that emit light of different colors.

4. 1. A head-mountable display system, comprising: a light source configured to emit light from different locations; a spatial light modulator configured to receive the light emitted by the light source and modulate the received light to form virtual content, wherein the light emitted by the light source from the different locations is received and modulated in different respective areas of the spatial light modulator; a projection optical element configured to direct the modulated light from the spatial light modulator toward the eyes of a wearer of the head-mountable display system; and Equipped with 1. A head-mountable display system, wherein the light source comprises a fiber and an actuator configured to move the fiber so that ends of the fiber are at the different locations at different times, and the light is emitted from the ends of the fiber.

5. 2. The head-mountable display system of claim 1, wherein the light source comprises a plurality of selectively activated light-emitting regions, each of the plurality of selectively activated light-emitting regions comprising the different locations, each of the light-emitting regions comprising at least one light emitter, and the light from the different light-emitting regions of the light source is received and modulated by the different respective areas of the spatial light modulator.

6. The head-mountable display system of claim 5 , wherein at least two of the light-emitting regions overlap on the light source.

7. The head-mountable display system of claim 5 , wherein the light-emitting regions are arranged separately on the light source without overlapping.

8. The head-mountable display system of claim 5 , wherein the at least one light emitter includes at least one light emitting diode.

9. The head-mountable display system of claim 5 , wherein at least one of the light-emitting regions is substantially rectangular in shape or substantially circular in shape.

10. The head-mountable display system of claim 5 , wherein at least two of the light-emitting regions are configured to emit light of different intensities.

11. The head-mountable display system of claim 5 , wherein one or more of the at least one light emitter is configured to emit light of a selectively variable intensity.

12. The head-mountable display system of claim 5 , wherein the light from different light-emitting regions of the light source is output from the spatial light modulator at different angles.

13. The head-mountable display system of claim 1 , further comprising a lens structure between the light source and the spatial light modulator.

14. 14. The head-mountable display system of claim 13, wherein the lens structure is configured to convert a spatial difference in the light emitted from the light source into an angular difference in propagation of the light to the spatial light modulator.

15. The head-mountable display system of claim 1 , further comprising a polarizing beam splitter arranged to direct at least a portion of the light from the light source toward the spatial light modulator.

16. The head-mountable display system of claim 1 , wherein the spatial light modulator comprises one or more of a liquid crystal panel on silicon, a transmissive liquid crystal panel, a digital light processing device, or a MEMS device.

17. 1. A head-mountable display system, comprising: a light source configured to emit light from different locations; a spatial light modulator configured to receive the light emitted by the light source and modulate the received light to form virtual content, wherein the light emitted by the light source from the different locations is received and modulated in different respective areas of the spatial light modulator; a projection optical element configured to direct the modulated light from the spatial light modulator toward the eyes of a wearer of the head-mountable display system; a lenslet array between the spatial light modulator and the projection optics; wherein the lenslet array is configured to direct the light output from the different respective areas of the spatial light modulator at different angles to the projection optics.

18. 1. A head-mountable display system, comprising: a light source configured to emit light from different locations; a spatial light modulator configured to receive the light emitted by the light source and modulate the received light to form virtual content, wherein the light emitted by the light source from the different locations is received and modulated in different respective areas of the spatial light modulator; a projection optical element configured to direct the modulated light from the spatial light modulator toward the eyes of a wearer of the head-mountable display system; a prism between the spatial light modulator and the projection optical element; wherein the prism is configured to direct the light output from the different respective areas of the spatial light modulator at different angles to the projection optical element.

Images