A display system and method for providing a variable focus cue using a plurality of intra-pupillary disparity views formed by an emitter array

Abstract

To provide a display system and a method for providing variable accommodation cues using multiple intra-pupil parallax views formed by light emitter arrays.SOLUTION: A display system is configured to direct a plurality of parallactically-disparate intra-pupil images into the viewer's eye. The parallactically-disparate intra-pupil images provide different parallax views of a virtual object, and impinge on the pupil from different angles. In the aggregate, wavefronts of light forming the images approximate a continuous divergent wavefront and provide selectable accommodation cues for a user, depending on an amount of parallax disparity between the intra-pupil images. The amount of parallax disparity may be selected using an array of shutters that selectively regulate entry of image light into the eye. Each opened shutter in the array provides a different intra-pupil image, and locations of the open shutters provide a desired amount of parallax disparity between the images.SELECTED DRAWING: None

Description

Technical Field

[0001] (Claim of Priority) This application claims the benefit of priority of U.S. Provisional Application No. 62 / 812,142, filed on February 28, 2019, and U.S. Provisional Application No. 62 / 815,225, filed on March 7, 2019, which are hereby incorporated by reference herein.

[0002] (Incorporation by Reference) This application incorporates by reference in their entirety each of the following patent applications and publications: U.S. Application Publication No. 2018 / 0113311, published on April 26, 2018; U.S. Patent Application No. 14 / 555,585, filed on November 27, 2014; U.S. Patent Application No. 14 / 690,401, filed on April 18, 2015; U.S. Patent Application No. 14 / 212,961, filed on March 14, 2014; U.S. Patent Application No. 14 / 331,218, filed on July 14, 2014; U.S. Patent Application No. 15 / 072,290, filed on March 16, 2016; WO2016 / 179246, published on November 10, 2016; and U.S. Provisional Application No. 62 / 800363, filed on February 1, 2019.

[0003] The present disclosure relates to optical devices, including augmented reality and virtual reality imaging and visualization systems.

Background Art

[0004] Modern computing and display technologies are facilitating the development of systems for so-called "virtual reality" or "augmented reality" experiences in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or is perceived to be real. Virtual reality, i.e., "VR" scenarios, typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, and augmented reality, i.e., "AR" scenarios, typically involve the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user. Mixed reality, i.e., "MR" scenarios, are a type of AR scenario that typically involves virtual objects integrated into and responsive to the natural world. For example, in an MR scenario, AR image content can be perceived as being blocked by objects within the real world or otherwise interacting therewith.

[0005] Referring to FIG. 1, an augmented reality scene 10 is depicted, and to a user of AR technology, a real-world park-like setting 20 characterized by people, trees, buildings in the background, and a concrete platform 30 is visible. In addition to these items, a user of AR technology also "sees" and perceives "virtual content" such as a robot figure 40 standing on the real-world platform 30 and an avatar character 50 in the form of a flying cartoon that appears anthropomorphic like a honeybee, but these elements 50, 40 do not exist in the real world. The human visual perception system is complex, and it is difficult to produce AR technology that facilitates a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0006] The systems and methods disclosed herein address various challenges associated with AR or VR technology. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0007] In some embodiments, a head-mounted display system is provided. The display system includes an image projection system that comprises a microdisplay configured to output image light that defines an image, and a projection optical system configured to direct the image light from the microdisplay toward the viewer's eye for propagation. The display system also comprises an array of shutters that are selectively activated to selectively transmit the image light to the eye from different locations. The array of shutters that are selectively activated is disposed within an eyebox volume of the projection optical system.

[0008] In some other embodiments, a method for displaying image content is provided. The method includes the step of projecting into the viewer's eye, from a head-mounted display system, a set of pupil images that are disparately different for a virtual object. Each image of the set of pupil images is provided by forming the image on a microdisplay of the head-mounted display system, outputting the image light from the microdisplay through the projection optical system, and opening a shutter of the array of shutters to propagate the image light through the opened shutter to the eye. The array of shutters is disposed within an eyebox volume of the projection optical system. Different images of the set of disparately different pupil images propagate through different opened shutters.

[0009] In yet other embodiments, a head-mounted display system is provided. The display system comprises a microdisplay comprising an array of groups of light emitters, an array of optical collimators above the light emitters, and a projection optical system. Each optical collimator is associated with one of the groups of light emitters and extends across all of the light emitters of the associated group of light emitters. The array of optical collimators is between the light emitters and the projection optical system. The display system is configured to display a virtual object on a depth plane by projecting into the viewer's eye a set of pupil images that are disparately different for the object.

[0010] In some other embodiments, a method for displaying image content is provided. The method includes projecting into a viewer's eye a set of intra-pupil images that are disparately different from a head-mounted display system. The step of projecting the set of intra-pupil images that are disparately different includes providing an array of groups of light emitters, providing an array of light collimators over the groups of light emitters, where each light collimator is associated with a group of light emitters, providing a projection optical system, where the array of light collimators is between the array of groups of light emitters and the projection optical system, projecting a first intra-pupil image that is disparately different into the eye by emitting light from a first light emitter of the group of light emitters, and projecting a second intra-pupil image that is disparately different into the eye by emitting light from a second light emitter of the group of light emitters.

[0011] In addition, various innovative aspects of the subject matter described in this disclosure may be implemented in the following examples.

[0012] (Example 1) A head-mounted display system, An image projection system, the image projection system including A microdisplay configured to output image light defining an image, A projection optical system configured to direct the image light from the microdisplay towards a viewer's eye for propagation, An image projection system comprising An array of selectively activatable shutters for selectively transmitting image light to the eye from different locations, the array of selectively activatable shutters being disposed within an eyebox volume of the projection optical system A display system comprising.

[0013] (Example 2) Further comprising a control system, the control system comprising one or more processors and a memory for storing instructions, which when executed by the one or more processors, cause the display system to determine a desired depth plane for the virtual object, based on the desired depth plane, determine shutters of an array of shutters to be selectively activated to be opened, synchronize the presentation of different images and the opening of different ones of the shutters by an image projection system, the different images providing different views of the virtual object, perform operations including those described in Example 1.

[0014] (Example 3) The shutter is a movable physical structure, the display system according to any one of Examples 1 or 2.

[0015] (Example 4) The physical structure is a MEMS-based microstructure, the display system according to Example 3.

[0016] (Example 5) The shutter is a ferroelectric shutter, the display system according to Example 3.

[0017] (Example 6) The shutter comprises a chemical species having a reversibly changeable state, the state providing different amounts of light transmission, the display system according to any one of Examples 1 or 2.

[0018] (Example 7) The chemical species comprises liquid crystal, and the shutter is formed by pixels of a pixelated liquid crystal display, the display system according to Example 6.

[0019] (Example 8) The microdisplay is the display system according to any one of Examples 1-7, which is a light-emitting microdisplay including an array of light emitters.

[0020] (Example 9) The display system according to Example 8, wherein the light emitter is a micro LED.

[0021] (Example 10) The display system according to any one of Examples 1-9, further including an array of light collimators between the light emitter and the projection optical system.

[0022] (Example 11) The display system according to Example 10, wherein each of the arrays of light collimators extends across a plurality of light emitters, and each light collimator corresponds to a pixel in the image output by the image projection system.

[0023] (Example 12) The microdisplay is one of a plurality of monochromatic microdisplays forming a projection system, and each monochromatic microdisplay is configured to emit light of a different primary color. The display system according to any one of Examples 1-11.

[0024] (Example 13) The display system according to Example 12, further including an X-cube prism, and each monochromatic microdisplay is arranged to output image light into different faces of the X-cube prism.

[0025] (Example 14) The display system according to any one of Examples 1-13, further including a pupil relay combiner eyepiece configured to relay image light to the viewer's eye, and an array of shutters selectively activated is configured to adjust the propagation of image light to the pupil relay combiner eyepiece.

[0026] (Example 15) The pupil relay combiner eyepiece includes a waveguide, and the waveguide an internal coupling optical element for internally coupling image light into the waveguide, and an external coupling optical element for externally coupling the internally coupled image light out of the waveguide The display system according to Example 14.

[0027] (Example 16) The waveguide is one of a plurality of waveguides including an internal coupling optical element and an external coupling optical element, and the display system according to Example 15.

[0028] (Example 17) The projection system has a pupil diameter of 0.2 to 0.5 mm, and the display system according to any one of Examples 1-16.

[0029] (Example 18) A method for displaying image content, the method comprising: inputting a set of pupil images with different parallax of virtual objects into the viewer's eyes from a head-mounted display system including each image of the pupil images forming an image on a microdisplay of a head-mounted display system; outputting the image light from the microdisplay through a projection optical system; opening the shutters of an array of shutters and propagating the image light through the opened shutters to the eyes, wherein the array of shutters is disposed within an eyebox volume of the projection optical system; and provided by different images of the set of pupil images with different parallax propagate through different opened shutters.

[0030] (Example 19) All images of the set of pupil images with different parallax are input into the eyes within a flicker fusion threshold, and the method according to Example 18.

[0031] (Example 20) The flicker fusion threshold is the method described in Example 18, which is 1 / 60 second.

[0032] (Example 21) Determining a desired depth plane for a virtual object to be displayed to a viewer, Based on the desired depth plane, determining shutters of an array of shutters to be selectively activated to be opened, Synchronizing the presentation of different ones of a set of intra-pupil images that are different in parallax and the opening of different ones of the shutters The method according to any one of Examples 18-20, further comprising.

[0033] (Example 22) Using an eye tracking sensor to determine the line of sight of an eye, Based on the determined line of sight of the eye, selecting content for an intra-pupil image The method according to any one of Examples 18-21, further comprising.

[0034] (Example 23) The microdisplay is a light-emitting microdisplay, the method according to any one of Examples 18-22.

[0035] (Example 24) The array of shutters comprises a physically movable structure that is selectively movable, the method according to any one of Examples 18-23.

[0036] (Example 25) The array of shutters comprises a chemical species having a reversibly changeable state, and the state provides different amounts of light transmission, the method according to any one of Examples 18-23.

[0037] (Example 26) The different images provide different views of the virtual object, the method according to any one of Examples 18-25.

[0038] (Example 27) A head-mounted display system comprising a microdisplay comprising an array of groups of light emitters, an array of light collimators above the light emitters, each light collimator being associated with one of the groups of light emitters and extending across all of the light emitters of the associated group of light emitters, the array of light collimators, a projection optical system, the array of light collimators being between the light emitters and the projection optical system, the projection optical system and being configured to project different views of a virtual object onto a depth plane by projecting a set of parallax-different intra-pupil images of the object into the viewer's eye. The display system is configured to display a virtual object on a depth plane by projecting a set of parallax-different intra-pupil images of the object into the viewer's eye.

[0039] (Example 28) further comprising one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the display system to determine each light emitter of a group of light emitters for activation based on a desired level of parallax difference for an image formed by the light emitters, activate a first light emitter of the group of light emitters to form a first parallax-different intra-pupil image, activate a second light emitter of the group of light emitters to form a second parallax-different intra-pupil image, and perform operations including wherein the first and second parallax-different intra-pupil images provide different views of the virtual object, the display system according to Example 27.

[0040] (Example 29) Activating the first light emitter of the group of light emitters overlaps in time with activating the second light emitter of the group of light emitters and projects first and second disparately different intra-pupil images into the eye simultaneously, the display system described in Example 27.

[0041] (Example 30) The light collimator is a microlenslet, the display system described in any of Examples 27-29.

[0042] (Example 31) An array of selectively activatable shutters for selectively transmitting image light to the eye from different locations, the array of selectively activatable shutters being disposed within the eye box volume of the projection optical system, the array of selectively activatable shutters further comprising the display system described in any of Examples 27-30.

[0043] (Example 32) The array of shutters comprises a physically movable structure that is selectively movable, the display system described in Example 31.

[0044] (Example 33) The array of shutters comprises chemical species having reversibly changeable states, the states providing different amounts of light transmission, the display system described in Example 31.

[0045] (Example 34) further comprising one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the display system to determine a desired depth plane for a virtual object and based on the desired depth plane, determine shutters of the array of selectively activatable shutters to be opened and Synchronizing the presentation of different images and the opening of different shutters by an image projection system, where the different images provide different views of a virtual object, and A display system according to any one of Examples 31-33, which performs an operation including this.

[0046] (Example 35) The light collimator is a lenticular lenslet configured to provide different light beams from a group of associated light emitters to different locations along a first axis, An array of shutters is arranged to form sub-pupils along a second axis orthogonal to the first axis, a display system according to any one of Examples 27-34.

[0047] (Example 36) The microdisplay is a light-emitting microdisplay, and the light emitter is a micro LED, a display system according to any one of Examples 27-35.

[0048] (Example 37) The microdisplay is one of a plurality of monochrome microdisplays, and each monochrome microdisplay is configured to emit light of a different primary color, a display system according to any one of Examples 27-36.

[0049] (Example 38) Further comprising an X-cube prism, and each monochrome microdisplay is arranged to output image light into different faces of the X-cube prism, a display system according to Example 37.

[0050] (Example 39) Further comprising a pupil relay combiner eyepiece configured to relay image light to the viewer's eye, and an array of shutters that is selectively activated is configured to adjust the propagation of image light to the pupil relay combiner eyepiece, a display system according to any one of Examples 27-38.

[0051] (Example 40) The pupil relay combiner eyepiece includes a waveguide, and the waveguide an internal coupling optical element for internally coupling image light into the waveguide, and an external coupling optical element for externally coupling the internally coupled image light out of the waveguide and is the display system according to Example 39.

[0052] (Example 41) The waveguide is one of a plurality of waveguides including an internal coupling optical element and an external coupling optical element, and is the display system according to Example 40.

[0053] (Example 42) A method for displaying image content, the method comprising: inputting a set of parallax-different intra-pupil images into the viewer's eye from a head-mounted display system, inputting the set of parallax-different intra-pupil images includes: providing an array of a group of light emitters; providing an array of upper-layer light collimators for the light emitters, each light collimator being associated with a group of light emitters; providing a projection optical system, the array of light collimators being between the array of the group of light emitters and the projection optical system; inputting a first parallax-different intra-pupil image into the eye by emitting light from a first light emitter of the group of light emitters; inputting a second parallax-different intra-pupil image into the eye by emitting light from a second light emitter of the group of light emitters and the method includes the above steps.

[0054] (Example 43) Images of a set of intra-pupil images that are different in parallax are each projected into the eye at different angles, and all images of the set of intra-pupil images that are different in parallax are projected into the eye within the flicker fusion threshold, according to the method described in Example 42.

[0055] (Example 44) The flicker fusion threshold is 1 / 60 second, according to the method described in Example 43.

[0056] (Example 45) The different images provide different views of the virtual object, according to the method described in any of Examples 42-43.

[0057] (Example 46) Projecting the first intra-pupil image that is different in parallax and projecting the second intra-pupil image that is different in parallax are performed simultaneously, according to the method described in any of Examples 42-45.

[0058] (Example 47) Providing an array of shutters that are selectively activated to selectively transmit image light to the eye from different locations, wherein the array of shutters that are selectively activated is disposed within the eye box volume of the projection optical system further included, according to the method described in any of Examples 42-46.

[0059] (Example 48) The light collimator is a lenticular lenslet configured to provide different light beams from a group of light emitters associated with a light emitter to different locations along a first axis, and the array of shutters is arranged to form sub-pupils along a second axis orthogonal to the first axis, according to the method described in any of Examples 42-47.

[0060] (Example 49) Spatially multiplexing a plurality of images formed by different light emitters of a group of light emitters, positioning a display sub-pupil along a first axis, and temporally multiplexing the plurality of images by synchronizing the opening of a shutter with the activation of the corresponding light emitter, the method according to any of Examples 47-48.

[0061] (Example 50) The method according to any of Examples 47-49, wherein the shutter array comprises a physically movable structure that is selectively movable.

[0062] (Example 51) The method according to any of Examples 47-49, wherein the shutter array comprises a chemical species having a reversibly changeable state, and the state provides different amounts of light transmission.

[0063] (Example 52) Projecting a first intra-pupil image that is different in parallax and projecting a second intra-pupil image that is different in parallax includes routing light from a light emitter through a pupil relay combiner eyepiece to the eye, the method according to any of Examples 42-51.

[0064] (Example 53) The pupil relay combiner eyepiece comprises a waveguide, and the waveguide an internal coupling optical element for internally coupling image light into the waveguide, and an external coupling optical element for externally coupling the internally coupled image light out of the waveguide The method according to Example 52.

[0065] (Example 54) Further including projecting a second set of intra-pupil images that are different in parallax into the second eye of the viewer from the head-mounted display system, the method according to any of Examples 42-53. The present invention provides, for example, the following. (Item 1) A head-mounted display system, An image projection system, wherein the image projection system comprises: A microdisplay configured to output image light defining an image; A projection optical system configured to direct the image light from the microdisplay towards the viewer's eye for propagation; An image projection system comprising the same; An array of selectively activatable shutters for selectively transmitting the image light to the eye from different locations, the array of selectively activatable shutters being disposed within the eyebox volume of the projection optical system; A display system comprising the same. (Item 2) Further comprising a control system, the control system comprising one or more processors and a memory storing instructions, which when executed by the one or more processors cause the display system to: Determine a desired depth plane for a virtual object; Based on the desired depth plane, determine shutters of the array of selectively activatable shutters to be opened; Synchronize the presentation of different images and the opening of different ones of the shutters by the image projection system, the different images providing different views of the virtual object; The display system according to item 1, which performs operations including the above. (Item 3) The shutter of the display system according to item 1 is a movable physical structure. (Item 4) The physical structure of the display system according to item 3 is a MEMS-based microstructure. (Item 5) The shutter of the display system according to item 3 is a ferroelectric shutter. (Item 6) The display system according to item 1, wherein the shutter comprises a chemical species having a reversibly changeable state, and the state provides different amounts of light transmission. (Item 7) The display system according to item 6, wherein the chemical species comprises liquid crystal, and the shutter is formed by pixels of a pixelated liquid crystal display. (Item 8) The display system according to item 1, wherein the microdisplay is a light-emitting microdisplay comprising an array of light emitters. (Item 9) The display system according to item 8, wherein the light emitter is a micro LED. (Item 10) The display system according to item 8, further comprising an array of light collimators between the light emitter and the projection optical system. (Item 11) The display system according to item 10, wherein each of the array of light collimators extends across a plurality of the light emitters, and each light collimator corresponds to a pixel in the image output by the image projection system. (Item 12) The display system according to item 1, wherein the microdisplay is one of a plurality of monochrome microdisplays forming the projection system, and each of the monochrome microdisplays is configured to emit light of a different primary color. (Item 13) The display system according to item 12, further comprising an X-cube prism, and each of the monochrome microdisplays is arranged to output image light into different faces of the X-cube prism. (Item 14) The display system according to item 1, further comprising a pupil relay combiner eyepiece configured to relay the image light to the viewer's eye, and the array of selectively activatable shutters is configured to adjust the propagation of the image light to the pupil relay combiner eyepiece. (Item 15) The pupil relay combiner eyepiece includes a waveguide, and the waveguide has an internal coupling optical element for internally coupling the image light into the waveguide, and an external coupling optical element for externally coupling the internally coupled image light out of the waveguide The display system according to item 14, comprising. (Item 16) The display system according to item 15, wherein the waveguide is one of a plurality of waveguides including an internal coupling optical element and an external coupling optical element. (Item 17) The display system according to item 1, wherein the projection system has a pupil diameter of 0.2 to 0.5 mm. (Item 18) A method for displaying image content, the method comprising: Inputting a set of pupil images with different parallax of virtual objects from a head-mounted display system into the viewer's eyes including Each image of the pupil images forming the image on a microdisplay of the head-mounted display system; outputting image light from the microdisplay through a projection optical system; opening the shutters of an array of shutters and propagating the image light through the opened shutters to the eye, wherein the array of shutters is disposed within an eyebox volume of the projection optical system provided by wherein different images of a set of pupil images with different parallax propagate through different opened shutters. (Item 19) The method according to item 18, wherein all images of a set of pupil images with different parallax are input into the eye within a flicker fusion threshold. (Item 20) The method according to item 18, wherein the flicker fusion threshold is 1 / 60 second. (Item 21) Determining a desired depth plane for the virtual object to be displayed to the viewer; Based on the desired depth plane, determining shutters of an array of the selectively activatable shutters to be opened; Synchronizing the presentation of different ones of a set of intra-pupil images that are parallax-different with the opening of different ones of the shutters; The method according to item 18, further comprising. (Item 22) Using an eye tracking sensor to determine the line of sight of the eye; Based on the determined line of sight of the eye, selecting content for the intra-pupil image; The method according to item 18, further comprising. (Item 23) The method according to item 18, wherein the microdisplay is an emissive microdisplay. (Item 24) The method according to item 18, wherein the array of shutters comprises a physically movable structure that is selectively movable. (Item 25) The method according to item 18, wherein the array of shutters comprises a chemical species having a reversibly changeable state, and the state provides different amounts of light transmission. (Item 26) The method according to item 18, wherein the different images provide different views of the virtual object. (Item 27) A head-mounted display system, comprising: A microdisplay comprising an array of a group of light emitters; An array of light collimators above the light emitters, each light collimator being associated with one of the group of light emitters and extending across all of the associated group of light emitters; an array of light collimators; A projection optical system, wherein the array of light collimators is between the light emitters and the projection optical system; a projection optical system; Comprising; The display system is configured to display a virtual object on a depth plane by projecting a set of pupil images that are disparately different for an object into the viewer's eye. (Item 28) The display system further includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the display system to determine, based on a desired level of disparity difference for an image formed by a light emitter, each light emitter of a group of light emitters for activation; activate a first light emitter of the group of light emitters to form a first disparately different pupil image; activate a second light emitter of the group of light emitters to form a second disparately different pupil image; perform operations including; The display system according to item 27, wherein the first and second disparately different pupil images provide different views of the virtual object. (Item 29) Activating the first light emitter of the group of light emitters overlaps in time with activating the second light emitter of the group of light emitters, and projects the first and second disparately different pupil images into the eye simultaneously. The display system according to item 27. (Item 30) The display system according to item 27, wherein the light collimator is a microlenslet. (Item 31) An array of selectively activatable shutters for selectively transmitting the image light to the eye from different locations, the array of selectively activatable shutters being disposed within an eyebox volume of the projection optical system. The array of selectively activatable shutters The display system according to item 27, further comprising. (Item 32) The display system according to item 31, wherein the shutter array comprises a physically movable structure that can be selectively moved. (Item 33) The display system according to item 31, wherein the shutter array comprises a chemical species having a reversibly changeable state, and the state provides different amounts of light transmission. (Item 34) The display system further comprises one or more processors and a memory storing instructions, and when the instructions are executed by the one or more processors, the display system is caused to determine a desired depth plane for a virtual object, determine shutters of the shutter array that are to be selectively activated and opened based on the desired depth plane, synchronize the presentation of different images and the opening of different shutters by the image projection system, wherein the different images provide different views of the virtual object, and perform operations including the above, in the display system according to item 31. (Item 35) The light collimator is a lenticular lenslet configured to provide different light beams from an associated group of light emitters of the light emitter to different locations along a first axis, and the shutter array is arranged to form sub-pupils along a second axis orthogonal to the first axis, in the display system according to item 27. (Item 36) The microdisplay is a light-emitting microdisplay, and the light emitter is a micro LED, in the display system according to item 27. (Item 37) The microdisplay is one of a plurality of monochromatic microdisplays, and each of the monochromatic microdisplays is configured to emit light of a different primary color, in the display system according to item 27. (Item 38) The display system according to item 37, further comprising an X-cube prism, wherein the monochromatic microdisplays are each arranged to output image light into different faces of the X-cube prism. (Item 39) The display system according to item 27, further comprising a pupil relay combiner eyepiece configured to relay the image light to the viewer's eye, wherein an array of shutters that are selectively activated is configured to regulate the propagation of the image light to the pupil relay combiner eyepiece. (Item 40) The pupil relay combiner eyepiece comprises a waveguide, and the waveguide an internal coupling optical element for internally coupling the image light into the waveguide, and an external coupling optical element for externally coupling the internally coupled image light out of the waveguide The display system according to item 39, comprising. (Item 41) The display system according to item 40, wherein the waveguide is one of a plurality of waveguides comprising an internal coupling optical element and an external coupling optical element. (Item 42) A method for displaying image content, the method comprising: projecting a set of parallax-distinct intra-pupil images from a head-mounted display system into the viewer's eye comprising projecting the set of parallax-distinct intra-pupil images providing an array of a group of light emitters; providing an array of upper-layer light collimators for the light emitters, each light collimator being associated with the group of light emitters; providing a projection optical system, the array of light collimators being between the array of the group of light emitters and the projection optical system; projecting a first parallax-distinct intra-pupil image into the eye by emitting light from a first light emitter of the group of light emitters By emitting light from a second light emitter of the group of light emitters, introducing into the eye a second parallax - different intra - pupil image A method comprising the above. (Item 43) The images of the set of parallax - different intra - pupil images are each introduced into the eye at different angles, and all the images of the set of parallax - different intra - pupil images are introduced into the eye within the flicker fusion threshold. The method according to item 42. (Item 44) The flicker fusion threshold is 1 / 60 second. The method according to item 43. (Item 45) The different images provide different views of the virtual object. The method according to item 42. (Item 46) Introducing the first parallax - different intra - pupil image and introducing the second parallax - different intra - pupil image are carried out simultaneously. The method according to item 42. (Item 47) Providing an array of selectively activated shutters for selectively transmitting the image light to the eye from different locations, wherein the array of selectively activated shutters is arranged within the eye - box volume of the projection optical system The method according to item 42, further comprising the above. (Item 48) The light collimator is a lenticular lenslet configured to provide different light beams to different locations along a first axis from an associated group of light emitters of the light emitters, and the array of shutters is arranged along a second axis orthogonal to the first axis to form sub - pupils. The method according to item 47. (Item 49) Spatially multiplexing a plurality of images formed by different light emitters of the group of light emitters, positioning the display sub-pupil along the first axis, and synchronizing the activation of the light emitter corresponding to the opening of the shutter to temporally multiplex the plurality of images, the method according to item 48. (Item 50) The method according to item 47, wherein the array of shutters comprises a physically movable structure that is selectively movable. (Item 51) The method according to item 47, wherein the array of shutters comprises a chemical species having a reversibly changeable state, and the state provides different amounts of light transmission. (Item 52) Injecting the first disparately different intra-pupil image and injecting the second disparately different intra-pupil image include routing light from the light emitter through a pupil relay combiner eyepiece to the eye, the method according to item 42. (Item 53) The pupil relay combiner eyepiece comprises a waveguide, and the waveguide an internal coupling optical element for internally coupling the image light into the waveguide, and an external coupling optical element for externally coupling the internally coupled image light out of the waveguide The method according to item 52. (Item 54) The method according to item 42, further comprising injecting a second set of disparately different intra-pupil images into a second eye of the viewer from the head-mounted display system.

Brief Description of the Drawings

[0066]

Figure 1

[0067]

Figure 2

[0068]

Figure 3

[0069]

Figure 4

[0070]

Figure 5

[0071]

Figure 6A

[0072]

Figure 6B

[0073]

Figure 7A

[0074]

Figure 7B

[0075]

Figure 8

[0076]

Figure 9

[0077]

Figure 10

[0078]

Figure 11

[0079]

Figure 12

[0080]

Figure 13

[0081]

Figure 14

[0082]

Figure 15

[0083]

Figure 16A

[0084]

Figure 16B

[0085]

Figure 17A

[0086]

Figure 17B

[0087]

Figure 18

[0088]

Figure 19

[0089]

Figure 20

[0090]

Figure 21

[0091]

Figure 22

[0092] The human visual system can perceive an image presented by a display as being "three-dimensional" by providing slightly different presentations of the image to each of the viewer's left and right eyes. Depending on the images presented to each eye, the viewer perceives "virtual" objects within the image as being at a distance (e.g., a certain "depth plane") selected by the viewer (also referred to herein as the "user"). However, simply providing different presentations of the image to the left and right eyes can cause discomfort to the viewer. As further discussed herein, viewing comfort can be increased by causing the eyes to accommodate to an image that the eyes would accommodate to a real object in the depth plane in which the virtual object is placed.

[0093] Proper accommodation for a virtual object on a given depth plane can be induced by presenting the image to the eyes using light having a wavefront divergence that matches the wavefront divergence resulting from the light from a real object on that depth plane. Some display systems use distinct structures having distinct refractive powers to provide the appropriate wavefront divergence. For example, one structure may provide a specific amount of wavefront divergence (for placing a virtual object on one depth plane), and another structure may provide a different amount of wavefront divergence (for placing a virtual object on a different depth plane). Thus, there can be a one-to-one correspondence between the physical structure and the depth planes within these display systems. Due to the need for a separate structure for each depth plane, such display systems can be bulky and / or heavy, which may not be desirable for some applications such as portable head-mounted displays. In addition, such display systems can have a limited number of different accommodation responses they can induce in the eyes due to practical limitations on the number of structures with different refractive powers that can be utilized.

[0094] It has been found that a continuous wavefront, such as a continuous diverging wavefront, can be approximated by projecting disparately different intra-pupil images into the eye. For example, a display system can provide a range of accommodation responses without requiring a one-to-one correspondence between the accommodation response and the optical structure within the display. The display system may output light with a selected amount of perceived wavefront divergence corresponding to a desired depth plane by projecting a set of disparately different intra-pupil images into the eye. These images can be referred to as "disparately different" intra-pupil images because each image can be considered a different parallax view of the same virtual object or scene on a given depth plane. These are "intra-pupil" images because a set of images having a parallax difference is projected into the pupil of one eye, e.g., the right or left eye of the viewer. The images can have some overlap, but the light beams forming these images will have at least some areas without overlap and will impinge on the pupil at slightly different angles. In some embodiments, the viewer's other eye, e.g., the left eye, may be provided with its own set of disparately different intra-pupil images. The sets of disparately different intra-pupil images projected into each eye can be somewhat different; for example, the images can show somewhat different views of the same scene due to somewhat different viewpoints provided by each eye.

[0095] When the wavefronts of light that form each of the intra-pupil images of different views are projected into the eye's pupil, they can generally approximate a continuous diverging wavefront as a whole. The perceived amount of divergence of this approximated wavefront can be varied by varying the disparity difference between the intra-pupil images, i.e., a change in the disparity difference changes the angular range covered by the wavefront of light that forms the intra-pupil image. Preferably, this angular range mimics the angular range covered by the continuous wavefront being approximated. In some embodiments, the wavefronts of light that form the individual intra-pupil images are collimated or quasi-collimated as discussed herein. An example of a system for providing intra-pupil images is disclosed in U.S. Application Publication No. 2018 / 0113311, published on April 26, 2018. Some of the embodiments disclosed in that application utilize a light emitter to illuminate a spatial light modulator, which encodes the light from the light emitter with image information, i.e., a plurality of light emitters may be provided and different intra-pupil images may be formed at different locations using the light emitters to provide the desired amount of disparity difference between the images.

[0096] Preferably, a set of intra-pupil images for approximating a particular wavefront is introduced into the eye rapidly enough so that the human visual system does not detect that the images were presented to the eye at different times. Without being limited by theory, the term "flicker fusion threshold" can be used to indicate the duration for which images presented to the human eye are perceived as being presented simultaneously, i.e., the visual system can perceive images formed on the retina within the flicker fusion threshold as being presented simultaneously. In some embodiments, the step of approximating a wavefront may include sequentially introducing a beam of light into the eye for each set of intra-pupil images, in a state where the total duration for introducing all of the beam of light is less than the flicker fusion threshold. It should be understood that presenting a set of images over a duration that exceeds the flicker fusion threshold can result in the human visual system perceiving that at least some of the images were introduced separately into the eye. As an example, the flicker fusion threshold may be about 1 / 60 of a second. As a result, each set of intra-pupil images may consist of a particular number of disparity views, e.g., two or more views, three or more views, four or more views, etc., and preferably, all of these views are provided to the eye within the flicker fusion threshold.

[0097] Providing all of the desired views within the flicker fusion threshold presents a challenge for some display technologies, such as those that use a spatial light modulator that is moved to modulate the intensity of the light output by a physical element. The need to physically move these elements can limit the speed at which individual pixels can change state and also constrain the frame rate of displays that use these optical elements. Additionally, a spatial light modulator may require a separate light source, which can undesirably add to the complexity, size, and weight of the display system and potentially limit the brightness of the image being displayed.

[0098] In some embodiments, the display system includes a light-emitting microdisplay, which can advantageously provide different intra-pupil images at a significantly high rate. Additionally, the display system may include an array of shutters. The shutters of the shutter array may be selectively opened or activated individually to allow light transmission and enable light to propagate into the retina from different locations. The light-emitting microdisplay emits image light for forming an intra-pupil image, and the image light propagates to the shutter array. Different ones of the shutters at different locations may be selectively opened (made transmissive to the image light) to allow the image light to further propagate into the viewer's eye from those different locations. The amount of the parallax difference between the intra-pupil images may be varied by changing the locations where the shutters are opened. As a result, the spatial difference at the opened shutter locations can be converted into a difference in the path that the light follows into the eye. Different paths may correspond to different amounts of parallax difference. In some embodiments, an array of light collimators may be disposed proximate to the light-emitting microdisplay. For example, each pixel of the light-emitting microdisplay may have an associated light collimator. The light collimator may narrow the angular emission profile of the light emitted by the light-emitting microdisplay, thereby increasing the amount of the emitted light that ultimately reaches the viewer's eye.

[0099] It should be understood that the images formed by the emissive microdisplay may be synchronized in time with the shutter, which is opened within the array of shutters. For example, the opening of one shutter (or multiple adjacent or consecutive shutters) corresponding to one intra-pupil image may be synchronized with or simultaneous with the activation of the pixels in the microdisplay. Once another shutter at a desired location for the second intra-pupil image is opened, the microdisplay may emit light to form that second intra-pupil image. Additional intra-pupil images may be formed by synchronizing with the opening of shutters at different locations. The sequential input of the intra-pupil images to the eye in real-time may be referred to as time multiplexing or a temporally multiplexed display of the intra-pupil images. As a result, in some embodiments, the presentation of the intra-pupil images by the microdisplay may be temporally multiplexed such that different parallax views are provided at different times by the emissive microdisplay and are synchronized with the opening of different shutters that provide the desired parallax difference.

[0100] Preferably, the emissive microdisplay is a micro-LED display, which offers advantages for high brightness and high pixel density. In some other embodiments, the microdisplay is a micro-OLED display.

[0101] In some embodiments, the emissive microdisplay comprises an array of light emitters having a pitch of less than 10 μm, less than 8 μm, less than 6 μm, less than 5 μm, or less than 2 μm, including, for example, 1 - 5 μm, 1 - 4 μm, or 1 - 2 μm, and an emitter size of 2 μm or less, 1.7 μm or less, or 1.3 μm or less. In some embodiments, the emitter size is within a range having the upper limit of the above sizes and a lower limit of 1 μm. Examples of the ratio of emitter size to pitch include 1:1 - 1:5, 1:2 - 1:4, or 1:2 - 1:3.

[0102] In some embodiments, the display system may utilize a light-emitting microdisplay in conjunction with a light collimator array to provide different amounts of parallax difference. The collimator array may be configured to direct the light emitted by the microdisplay along different paths corresponding to different amounts of parallax difference. For example, the collimator array may be positioned adjacent to or directly on the light-emitting microdisplay. In some embodiments, the light collimator is a microlenslet. Each collimator of the collimator array may include a group of associated subpixels, each arranged at a different location with respect to the collimator. As a result, light from different subpixels of the group of subpixels interfaces differently with the collimator and is directed by the collimator along slightly different paths. These different paths may correspond to different amounts of parallax difference. Thus, each collimator may correspond to a different pixel in the intra-pupil image, and each subpixel may provide a different light path for its pixel such that the parallax difference between two or more pixels can be selected by appropriate activation of the subpixels forming those pixels. In some embodiments, advantageously, different intra-pupil images may be formed and provided to the eye simultaneously with a collimator array that directs the parallax difference determined by the location of the subpixels forming the image and the propagation of light from those subpixels.

[0103] As described above, light from different sub-pixels will follow different paths through the projection optical system and thus to the viewer's eye. As a result, lateral and / or vertical displacements of the active sub-pixels are converted into angular displacements of the light that exits the light collimator array and ultimately propagates through the projection optical system towards the viewer's pupil. In some embodiments, it can be understood that an increase in the lateral displacement between the activated sub-pixels used to form different images is converted into an increase in the angular displacement as measured with respect to the microdisplay. In some embodiments, the images within the pupil, which are used to approximate a particular wavefront, may each be formed by outputting light from different sub-pixels, thereby providing an angular displacement between the beams of light that form each of the images.

[0104] In some embodiments, the display system may include a projection optical system for directing light into the eye, whether or not it utilizes an array of collimators or shutters. The emissive microdisplay may be configured to output light encoded with image information to form an image within the pupil. The light then impinges on the projection optical system and propagates through it and ultimately to the viewer's eye.

[0105] In some embodiments, the display system may include a combiner eyepiece, which enables virtual image content to be overlaid with the viewer's view of the world or the surrounding environment. For example, the combiner eyepiece may be an optically transmissive waveguide that enables the viewer to see the world. Additionally, the waveguide may be utilized to receive, direct, and ultimately output light that forms an image within the pupil. Since the waveguide can be positioned between the viewer and the world, the light output by the waveguide can be perceived as forming virtual images placed on various depth planes within the world. In essence, the combiner eyepiece enables the viewer to receive a combination of light from the display system and light from the world.

[0106] In some embodiments, the display system may also include an eye tracking system that detects the viewing direction of the viewer. Such an eye tracking system enables appropriate content to be selected and displayed based on where the viewer is looking.

[0107] Preferably, the display system operates in an "open loop" mode where the depth of field provided by the light forming the individual intra-pupil images is substantially infinite and the visual system is unable to accommodate the eyes with respect to the individual intra-pupil images, and has a sufficiently small exit pupil. In some embodiments, the light beam forming the individual images occupies an area having a width or diameter of less than about 0.5 mm when incident on the eye. However, it should be understood that the light beams forming the set of intra-pupil images are at least partially non-overlapping, and the set of light beams preferably defines an area larger than 0.5 mm, provides sufficient information to the eye's lens, and induces a desired accommodation response based on the wavefront approximation formed by the wavefront of the light forming the intra-pupil images.

[0108] Without being limited by theory, the area defined by the set of light beams can be considered to mimic the synthetic aperture through which the eye views the scene. It should be understood that viewing the scene through a sufficiently small pinhole in the front of the pupil provides a substantially infinite depth of field. Given the small aperture of the pinhole, the eye's lens does not provide proper scene sampling to distinguish clearly different depths of focus. As the pinhole expands, additional information is provided to the eye's lens, enabling the natural optical phenomenon to allow a limited depth of focus to be perceived. Advantageously, the area defined by the set of light beams and the corresponding set of disparately different intra-pupil images can be made larger than the pinhole that produces an infinite depth of field, and the plurality of intra-pupil images can produce an approximation of the effect provided by the enlarged pinhole described above.

[0109] Some embodiments disclosed herein may provide various advantages. For example, since the microdisplay is light-emitting, external illumination is not required, thereby facilitating reduction in the size and weight of the projection system. The small size of these microdisplays enables the use of a single projector with separate primary color (e.g., red, green, blue) microdisplay panels without requiring an overly large or complex projector. In some embodiments, advantageously, due to the small size and weight of the various microdisplays disclosed herein, different projectors may be used for different primary colors. Additionally, in contrast to typical displays such as LCOS displays, polarization is not required to provide image information to light. As a result, the light losses associated with polarization can be avoided. Also, the individual light emitters of the microdisplay have high étendue, and as a result, the light from each pixel necessarily fills a very large pupil area, which can desirably provide a large eyebox volume. In some embodiments, the light-emitting microdisplay is a micro-LED display, which may have a significantly high frame rate (e.g., a frame rate of 1 kHz or more, including 1 - 2 kHz). Additionally, the light-emitting microdisplay may have a significantly small pixel pitch (e.g., 1 - 4 μm, including 2 - 4 μm or 1 - 2 μm) and a high pixel density, which can desirably provide a high image resolution.

[0110] Referring now to the drawings, like reference numerals refer to like parts throughout.

[0111] As discussed herein, the perception of an image as "3D" or "3-D" can be achieved by providing slightly different presentations of the image to each eye of the viewer. FIG. 2 illustrates a conventional scheme for simulating a 3D image for a user. One of two distinct images 190, 200 is output to the user for each eye 210, 220. Images 190, 200 are separated from eyes 210, 220 by a distance 230 along an optical or z-axis parallel to the viewer's line of sight. Images 190, 200 are flat, and eyes 210, 220 can be focused on the images by taking a single focused state. Such a 3-D display scheme relies on the human visual system to combine images 190, 200 and provide a perception of depth and / or scale of the combined images.

[0112] However, it should be understood that the human visual system is more complex and it is more difficult to provide a realistic perception of depth. For example, many viewers of conventional "3-D" display systems find such systems uncomfortable or may not perceive any sense of depth at all. Although not limited by theory, it is believed that a viewer of an object can perceive the object as "three-dimensional" due to the combination of convergence / divergence motion and accommodation. The movement of the convergence / divergence of the two eyes relative to each other (i.e., the rotation of the eyes where the pupils move towards each other or away from each other to converge the line of sight of the eyes and fixate on an object) is closely associated with the focusing (or "accommodation") of the eye's lens and pupil. Under normal conditions, changing the focus of the eye's lens (or accommodating the eye) from one object to another object at a different distance will automatically cause a corresponding change in the convergence / divergence motion to the same distance under a relationship known as the "accommodation-convergence / divergence reflex". Similarly, a change in the convergence / divergence motion will, under normal conditions, induce a corresponding change in accommodation associated with a change in the lens shape and pupil size. As described herein, many stereoscopic or "3-D" display systems use slightly different presentations (and thus slightly different images) to each eye to display a scene so that a three-dimensional viewpoint is perceived by the human visual system. However, such systems are uncomfortable for many viewers, especially because they simply provide different presentations of the scene, i.e., the eyes view all the image information, even the image information regarding objects at different depths, in a single accommodated state. However, this functions against the "accommodation-convergence / divergence reflex". A display system that provides better alignment between accommodation and convergence / divergence motion forms a more realistic and comfortable simulation of a three-dimensional image, which promotes the user to wear the display for a longer duration.

[0113] FIG. 3 illustrates a side view of an approach for simulating a three-dimensional image using a plurality of depth planes. Objects at various distances from eyes 210, 220 on the z-axis are focused by eyes 210, 220 such that those objects are in focus. That is, eyes 210, 220 assume a particular focused state and focus on objects at different distances along the z-axis. As a result, a particular focused state can be associated with a particular one of depth planes 240 having an associated focal length such that an object or a portion of an object in that particular depth plane is in focus when the eye is in the focused state for that depth plane. In some embodiments, the three-dimensional image may be simulated by providing different presentations of the image for each of eyes 210, 220, and the presentation of the image may also vary with respect to different depth planes. Although shown separately for clarity of illustration, it should be understood that the fields of view of eyes 210, 220 may overlap, for example, as the distance along the z-axis increases. Additionally, although shown as flat for ease of illustration, it should be understood that the contours of the depth planes may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular focused state.

[0114] The distance between the object and eye 210 or 220 can also vary the amount of divergence of light from that object as viewed by that eye. FIGS. 4A-4C illustrate the relationship between distance and divergence of light rays. The distance between the object and eye 210 is represented in the order of decreasing distances R1, R2, and R3. As shown in FIGS. 4A-4C, the light rays diverge more as the distance to the object decreases. As the distance increases, the light rays become more collimated. In other words, it can be said that the light field generated by a point (object or part of an object) has a spherical wavefront curvature that is a function of the distance the point is away from the user's eye. The curvature increases with a decrease in the distance between the object and eye 210. As a result, at different depth planes, the divergence of the light rays also differs, and the divergence increases with a decrease in the distance between the depth plane and the viewer's eye 210. Only the monocular 210 is illustrated in FIGS. 4A-4C and various other figures in this specification for clarity of illustration, but it should be understood that the discussion regarding the eye 210 can apply to both eyes 210 and 220 of the viewer.

[0115] Although not limited by theory, the human eye is thought to typically be able to interpret a finite number of depth planes and provide depth perception. As a result, a highly realistic simulation of the perceived depth can be achieved by providing different presentations of an image corresponding to each of these limited number of depth planes to the eye. The different presentations are focused separately by the viewer's eye, thereby serving to provide the user with depth cues based on the eye accommodation required to focus on different image features within a scene where different features are located on different depth planes. This can also cause other image features on other depth planes to appear out of focus, which provides an additional sense of depth to the viewer.

[0116] Since each depth plane has an associated wavefront divergence, some displays may utilize a waveguide having a refractive power to output light with the wavefront divergence corresponding to that depth plane in order to display image content that appears to be at a particular depth plane. A plurality of waveguides having similar but different refractive powers may be utilized to display image content on a plurality of depth planes. For example, such a system may utilize a plurality of such waveguides formed in a stack. FIG. 5 illustrates an example of a waveguide stack for outputting image information to a user. A display system 250 includes a stack 260 of waveguides that can be utilized to provide three-dimensional perception to the eye / brain using a plurality of waveguides 270, 280, 290, 300, 310 for outputting image information. Image input devices 360, 370, 380, 390, 400 may be utilized to input light containing image information into waveguides 270, 280, 290, 300, 310. In some embodiments, image input devices 360, 370, 380, 390, 400 may be understood as different light projection systems and / or different pupils of one or more projection systems. Waveguides 270, 280, 290, 300, 310 may be separated from other waveguides, for example, by air or another low refractive index material, and total internal reflection of light is facilitated through the individual waveguides. Each waveguide 270, 280, 290, 300, 310 may include a structure (e.g., optical gratings and / or lenses 570, 580, 590, 600, 610, respectively) that provides a refractive power such that each waveguide outputs light with a pre-set amount of wavefront divergence corresponding to a particular depth plane. Thus, each waveguide 270, 280, 290, 300, 310 disposes the image content on an associated depth plane determined by the amount of wavefront divergence provided by that waveguide.

[0117] However, it should be understood that a one-to-one correspondence between the waveguides and the depth planes can lead to a bulky and heavy device in a system where multiple depth planes are desired. In such embodiments, multiple depth planes would require multiple waveguides. Additionally, if a color image is desired, each depth plane may have multiple corresponding waveguides, and since one waveguide per primary color (e.g., red, green, or blue) may be required to form the color image, even more waveguides may be required.

[0118] Advantageously, various embodiments herein can provide a simpler display system that approximates a desired continuous wavefront by using a diverging light beam to form an intra-pupil image that presents different parallax views of an object or scene. Further, an optical projection system that utilizes a light-emitting microdisplay to reduce the size and weight of the display system relative to the projection system utilizes a separate spatial light modulator and a light source. Additionally, some embodiments can provide a significantly high frame rate and may offer advantages in terms of flexibility when providing a desired number of intra-pupil images within a given duration.

[0119] Referring now to FIG. 6A, the pre-accommodation state of the eye 210 and the post-accommodation state in response to receiving a continuous input wavefront 1000 are illustrated. Illustration a) shows the pre-accommodation state before the visual system focuses the wavefront 1000 onto the retina 211. It should be noted that the focal point 212 is not on the retina 211. For example, the focal point 212 may be in front of the retina 211 as shown. Illustration b) shows the post-accommodation state after the human visual system has flexed the pupil muscle tissue of the viewer's eye 210 and focused the wavefront 1000 onto the retina 211. As shown, the focal point 212 may be on the retina 211.

[0120] It has been found that continuous wavefronts such as wavefront 1000 in FIG. 6A can be approximated using a plurality of wavefronts. FIG. 6B illustrates the state of the eye 210 before accommodation and the state after accommodation in response to receiving a segmented approximation of the continuous wavefront 1000 in FIG. 6A. Illustration a) in FIG. 6B shows the state of the eye 210 before accommodation, and illustration b) shows the state after accommodation. The approximation may be formed using a plurality of component wavefronts 1010a, 1010b, and 1010c, each associated with a separate beam of light. As used herein, reference numerals 1010a, 1010b, and 1010c may denote both the light beam and the wavefront associated with that light beam. In some embodiments, the component wavefronts 1010a and 1010b may be planar wavefronts such as those formed by a collimated beam of light. As shown in illustration b), the wavefront approximation 1010 formed by the component wavefronts 1010a and 1010b is focused onto the retina 211 by the eye 210, and the focal point 212 is on the retina 211. Advantageously, the states before and after accommodation are similar to those produced by the continuous wavefront 1000 shown in FIG. 6A.

[0121] It should be understood that a continuous diverging wavefront may be formed using an optical projection system. As described above, U.S. Application Publication No. 2018 / 0113311, published on April 26, 2018, discloses an exemplary system that uses a light source to illuminate a spatial light modulator 1018. FIG. 7A illustrates an eye that accommodates for a diverging wavefront emanating from a finite focal length virtual image provided by a projection system. The system includes a spatial light modulator 1018 and a projection optical system 1020 with a focal length “F” and an external stop. A light source (not shown) provides light to the spatial light modulator 1018, and an image may be formed by the spatial light modulator 1018 by modulating the light. The modulated light containing image information may be directed through the projection optical system 1020 to the eye 210. As shown in FIG. 7A, the distance (less than F) between the microdisplay 2000 and the projection optical system 1020 may be selected such that the diverging wavefront 1000 is output toward the eye 210. As described above with respect to FIG. 6A, the eye 210 may then focus the wavefront 1000 onto the retina 211.

[0122] FIG. 7B illustrates a system for forming an approximation of the diverging wavefront of FIG. 7A using wavefront segments formed by an infinitely focused virtual image. As described above, the system includes a spatial light modulator 1018 and a projection optical system 1020. As described above, a light source (not shown) provides light to the spatial light modulator 1018, which modulates the light to form an image. The spatial light modulator 1018 can form two images that are offset relative to each other. The spatial light modulator 1018 is placed at a distance F from the back focal plane of the projection optical system 1020, which has a back focal length of F. A light beam 1010a containing image information regarding the first image propagates through the projection optical system 1020 into the eye 210. A light beam 1010b containing image information regarding the second image follows a different path through the projection optical system 1020 into the eye 210. The light beams 1010a and 1010b propagate through the projection optics 1020 along a path from the spatial light modulator into the eye 210 such that each light beam defines an angular range that matches the angular range of the diverging wavefront 1000 (FIG. 7A). It should be understood that the angular separation between the light beams 1010a and 1010b increases with an increase in the amount of wavefront divergence being approximated.

[0123] Referring now to FIG. 8, an embodiment of a parallax view forming a diverging wavefront approximation of FIG. 7B is illustrated. It should be understood that light beams 1010a, 1010b, and 1010c each form distinct images of one view of the same object or scene from slightly different viewpoints corresponding to different locations of an image in space. As illustrated, the images may be successively input into the eye 210 at different times. Alternatively, the images may be input simultaneously, if the optical system permits, or groups of images may be input simultaneously, as discussed herein. In some embodiments, the total duration that light forming all of the images is input into the eye 210 is less than the viewer's flicker fusion threshold. For example, the flicker fusion threshold may be 1 / 60 of a second, and light beams 1010a, 1010b, and 1010c are all input into the eye 210 over a duration less than that flicker fusion threshold. Thus, the human visual system integrates all of these images, and they appear to the eye 210 as if light beams 1010a, 1010b, and 1010c were input into that eye 210 simultaneously. Light beams 1010a, 1010b, and 1010c thus form the wavefront approximation 1010.

[0124] Referring now to FIG. 9, an embodiment of a display system 1001 is illustrated that includes a projection system 1003 for forming the diverging wavefront approximation 1010 of FIGS. 7B and 8. The projection system 1003 includes a microdisplay 2000 that includes a plurality of light emitters 2002 that function as pixels for forming an image. The light emitters 2002 emit light 2002a, 2002b, 2002c, which propagates through relay / projection optics 1020 and is output by the relay / projection optics 1020. The light then continues through the open shutter 20061 of the shutter array 2004. The open shutter 20061 can be understood to form a shutter aperture sub-pupil. Additionally, the light propagating through the open shutter 20061 can be understood as a light beam 1010a, which propagates into the eye 210 and forms an image within the pupil (see, e.g., the image formed by the light beam 1010a in FIG. 8). Preferably, the shutter array 2004 is located on or near the plane of the pupil of the eye 210.

[0125] It should be understood that the numerical aperture of the light received by the projection optical system 1020 is determined by the focal length and the diameter of the projection optical system. The light emerging from the projection optical system 1020 forms a pupil, which may be at the exit aperture of the projection optical system 1020 in some embodiments. Additionally, the lights 2002a, 2002b, 2002c that propagate through and exit from the projection optical system 1020 continue to propagate in different directions but also overlap to define a volume 2003. The volume 2003 is an eyebox and may be pyramid-shaped in some embodiments. As described above, the eyebox includes contributions from all of the light emitters 2002 of the microdisplay 2000. Preferably, the size (e.g., side dimension) of the projection optical system pupil and the size (e.g., side dimension) of the eyebox in which the eye 210 is placed are the same or larger (e.g., side dimension) than the pupil of the eye 210 when viewing an image from the microdisplay 2000. As a result, preferably, the entire image formed on the microdisplay 2000 can be viewed by the eye 210. Additionally, as described herein, the shutter array 2004 is preferably located on or proximate to the plane of the pupil of the eye 210 such that the shutter array 2004 is also within the eyebox volume 2003.

[0126] Continuing to refer to FIG. 9, as described herein, the microdisplay 2000 may comprise an array of light emitters 2002. Examples of light emitters include organic light emitting diodes (OLEDs) and micro light emitting diodes (microLEDs). It should be understood that OLEDs utilize organic materials to emit light and microLEDs utilize inorganic materials to emit light.

[0127] Advantageously, some microLEDs provide higher brightness and higher efficiency (from the perspective of lux / W) than OLEDs. In some embodiments, the microdisplay is preferably a microLED display. MicroLEDs may utilize inorganic materials, such as group III-V materials such as GaAs, GaN, and / or GaIn, for light emission. Examples of GaN materials include InGaN, which may be used to form blue or green light emitters in some embodiments. Examples of GaIn materials include AlGaInP, which may be used to form red light emitters in some embodiments. In some embodiments, the light emitter 2002 may emit light of an initial color, which may be converted to another desired color using a phosphor material or quantum dots. For example, the light emitter 20002 may emit blue light, which excites a phosphor material or quantum dots that convert blue wavelength light to green or red wavelengths.

[0128] In some embodiments, the display system 1001 may include an eye tracking device 1022, such as a camera, configured to monitor the viewer's line of sight. Such monitoring may be used to determine the direction the viewer is looking, which may be used to select appropriate image content with respect to that direction. The eye tracking device 1022 may track both eyes of the viewer, or each eye may include its own associated eye tracking device. In some embodiments, the convergence / divergence movement of both eyes of the viewer may be tracked, the point of convergence of the viewer's line of sight may be determined, and the direction and distance at which the eyes are directed may be determined.

[0129] Continuing to refer to FIG. 9, display system 1001 may also include a control system 1024 for determining the timing and type of image content provided by the display system. In some embodiments, control system 1024 comprises one or more hardware processors with memory storing a program for controlling display system 1001. For example, system 1024 may be configured to control the opening of individual shutters of shutter array 2004, the activation of emitters 2002 of microdisplay 2000, and / or the interpretation and reaction of display system 1001 to data received from eye tracking device 1022. Preferably, system 1024 is configured to receive an input regarding a desired depth plane or wavefront divergence and calculate an appropriate shutter to open in order to form a parallax view with an appropriate disparity for the desired depth plane or wavefront divergence, including a calculation module 1024a. Additionally, calculation module 1024a may be configured to determine the appropriate operation of light emitters 2002 of microdisplay 2000 and form an image of the desired parallax view. System 1024 may also include a synchronization module 1024b configured to synchronize the activation of a particular shutter of shutter array 2004 with the emission of light by light emitters 2002 of microdisplay 2000 to form an image providing the desired parallax view. Additionally, system 1024 may include an eye tracking module 1024c that receives an input from eye tracking device 1022. For example, eye tracking device 1022 may be a camera configured to image eye 210. Based on the image captured by eye tracking device 1022, eye tracking module 1024c may be configured to determine the orientation of the pupil and extrapolate the line of sight of eye 210. This information may be electronically communicated to calculation module 1024a. Calculation module 1024a may be configured to select image content based on the line of sight or gaze of eye 210 (preferably, also based on the line of sight or gaze of the other eye of the viewer, e.g., based on the fixation point of the viewer's eyes).

[0130] Referring now to FIG. 10A, an embodiment of the array 2004 of shutters 2006 is illustrated. The figure shown is a front view, as can be seen from the direction of the light from the microdisplay 2000 that impinges on the array 2004. As shown, the array 2004 may extend in two dimensions and may comprise rows and columns of shutters 2006. Preferably, each of the shutters 2006 may be individually activated or opened. The location of the opened shutters 2006 may correspond to the desired location in the intra-pupillary image. For example, the shutter array 2004 shown in FIG. 9 may have an opened shutter 2006a that corresponds to the location of the light beam 1010a shown in FIG. 8.

[0131] It should be understood that the array 2004 may have other overall configurations. For example, with respect to horizontal parallax-only driving focus adjustment, the array 2004 may be a horizontal line of shutters 2006, as shown in FIG. 10A. In some other embodiments, with respect to vertical parallax-only driving focus adjustment, the array 2004 may be a vertical line of shutters 2006, as shown in FIG. 10B.

[0132] Although shown as a square grid, the shutters 2006 may be arrayed in any configuration, which provides a certain parallax difference when different ones of the shutters 2006 are opened. For example, the shutters 2006 may be arranged in a hexagonal grid. In addition, the shutters 2006 may have a shape other than square. For example, the shutters 2006 may have a hexagonal shape, a circular shape, a triangular shape, etc. In some embodiments, multiple shutters may be opened, which may allow the size and / or shape of the opening to be selected as desired.

[0133] It should be understood that the shutters discussed herein are structures that can be made transmissive independently of light. As a result, an open shutter is controlled to be transmissive to light, and a closed shutter is controlled to block light or be opaque. In some embodiments, the shutter may be a physical structure that moves between a first position and a second position to transmit or block light by moving out of or into the light path, respectively. In some other embodiments, the shutter may include chemical species that reversibly change state or orientation and change the light transmissive properties of the shutter. Examples of structures suitable for use as shutters include pixels forming a transmissive pixelated liquid crystal display, MEMS-based micro-mechanical structures (e.g., that can move perpendicular and / or horizontally into and out of the light path), or other segmented structures of an array that can have a high switching rate between at least two states. Additional examples of shutters include ferroelectric shutters. Preferably, the shutter is capable of changing state (switching between a transmissive (open) state and a non-transmissive (closed) state) by a factor determined by the number of shutter aperture sub-pupils (or intra-pupil images) desired for the display system at a rate exceeding the frame update rate of the microdisplay 2000. For example, if the microdisplay 2000 produces images at 120 Hz and three sub-pupils are desired, the shutter 2006 of the shutter array 2004 should preferably be capable of a switching rate of at least three times the microdisplay frame rate or 3×120 Hz (e.g., 360 Hz or higher).

[0134] Preferably, the projection system 1003 (FIG. 9) produces a relatively long depth of field, which may be controlled by limiting the aperture within the system. Although not limited by theory, as discussed herein, the projection system is configured to provide an image to the eye with an effective pupil diameter small enough to force the human visual system to operate in an "open-loop" mode because the eye cannot individually focus such an image. However, multiple intra-pupillary images are thought to provide the eye with sufficient information to elicit the desired focusing response. As discussed herein, the wavefront of the light forming the intra-pupillary image generally provides a wavefront approximation to which the eye focuses. In some embodiments, the projection system 1003 may be configured to provide an image with an effective pupil diameter of about 0.5 mm or less (e.g., 0.2 - 0.5 mm), which is considered small enough to force the human visual system to operate in an "open-loop" mode. As a result, in some embodiments, the shutter 2006, when opened, provides an opening having a width of 0.5 mm or less (e.g., 0.2 - 0.5 mm).

[0135] Referring again to FIG. 9, it should be understood that the illustrated open shutter 20061 is opened at a first time (t = t1) to provide an initial intra-pupillary image and show a first parallax view. Subsequently, at a later time (t = t2), another shutter may be opened to provide a second intra-pupillary image and show a second parallax view.

[0136] Referring now to FIG. 11, an embodiment of the projection system of FIG. 9 is shown that provides different intra-pupillary images that are differentially parallaxed at a later time (e.g., time t = t2). As shown, the shutter 20062 is opened to provide a beam of light 1010b into the eye 210. The light beam 1010b may form an intra-pupillary image shown on the right side of FIG. 8. Additional shutters may be opened at other times to provide other intra-pupillary images, and it should be understood that the set of intra-pupillary images for approximating the wavefront curvature is preferably provided to the eye within the flicker fusion threshold.

[0137] Again, referring to FIG. 9, as described herein, the flicker fusion threshold of the human visual system imposes a time constraint on the number of images that can be projected into the eye 210 while still being perceived as being projected simultaneously. For example, the processing bandwidth of the control system 1024 and the ability to switch the light modulator of the light source 1026 and the microdisplay 2000 can limit the number of images that can be projected into the eye 210 within the duration enabled by the flicker fusion threshold. Assuming this finite number of images, the control system 1024 may be configured to make selections regarding the images to be displayed. For example, within the flicker fusion threshold, the display system may be required to project a set of intra-pupil images that are different in parallax into the eye, and thus, each parallax view may require images of various primary colors to form a full-color image. In some embodiments, the formation of full-color images using primary color images diverges from the elucidation of the desired depth adjustment response. For example, without being limited by theory, it may be conceivable that a desired depth adjustment response can be induced with monochromatic light. In such a case, the intra-pupil images that are different in parallax and used to induce the depth adjustment response would be in monochromatic light only. As a result, it would not be necessary to form intra-pupil images that are different in parallax using light of other colors, thereby freeing up time within the flicker fusion threshold for other types of images to be displayed. For example, a larger set of intra-pupil images that are different in parallax may be generated to better approximate the wavefront.

[0138] In some other embodiments, the control system 1024 may be configured to spend little time within the flicker fusion threshold to display images of colors of light to which the human visual system is less sensitive. For example, the human visual system is less sensitive to blue light than to green light. As a result, the display system may be configured to generate a greater number of images formed using green light than images formed using blue light.

[0139] It should be understood that the light emitters of the microdisplay 2000 can emit light with a large-angle emission profile (e.g., Lambertian angle emission profile). Unfortunately, such an angle emission profile can "waste" light because only a small portion of the emitted light can ultimately propagate through the projection optics 1020 and reach the viewer's eye.

[0140] Referring now to FIG. 12, in some embodiments, a light collimator may be utilized to narrow the angle emission profile of the light emitted by the light emitter. FIG. 12 illustrates an example of the projection system of FIGS. 9 and 11 with an array of light collimators 2010 between the emissive microdisplay 2000 and the projection optics 1020. The array of light collimators 2010 is disposed in front of the light emitters 2002. In some embodiments, each light emitter 2002 has a dedicated light collimator 2010, i.e., each light emitter 2002 is matched 1 / 1 with an associated light collimator 2010 (one light collimator 2010 per light emitter 2002). The light collimator may be configured to pick up light from an associated light emitter 2002 and image that light to optical infinity (or in some other embodiments, some other non-infinite plane).

[0141] The light collimator 2010 may be configured to narrow the angle emission profile of the incident light. For example, as shown, each light collimator 2010 receives light from an associated light emitter 2002 with a relatively wide initial angle emission profile and outputs that light with an angle emission profile narrower than the wide initial angle emission profile of the light emitter 2002. In some embodiments, the light rays of the light exiting the light collimator 2010 are more parallel than the light rays of the light received by the light collimator 2010 before exiting therefrom. Advantageously, the light collimator 2010 can increase the efficiency of the use of light for forming an image by enabling more of that light to be directed into the viewer's eye than would be the case if the collimator 2010 were not present.

[0142] Preferably, the optical collimator 2010 is positioned in close proximity to the optical emitter 2002 and captures a large proportion of the light output by the optical emitter 2002. In some embodiments, a gap may exist between the optical collimator 2010 and the optical emitter 2002. In some other embodiments, the optical collimator 2010 may be in contact with the optical emitter 2002. Preferably, all or most of the cone of light from the optical emitter 2002 is incident on a single associated optical collimator 2010. Thus, in some embodiments, each optical emitter 2002 is smaller (occupies a smaller area) than the light receiving surface of the associated optical collimator 2010. In some embodiments, each optical emitter 2002 has a width smaller than the spacing between adjacent optical emitters 2002.

[0143] The optical collimator 2010 may take various forms. For example, in some embodiments, the optical collimator 2010 may be a microlens or a lenslet, including a spherical or lenticular lenslet. As discussed herein, each microlens preferably has a width greater than the width of the associated optical emitter 2002. The microlens may be formed from a curved transparent material such as glass or polymer, including resins such as photoresist and epoxy. In some embodiments, the optical collimator 2010 may be a nanolens, such as a diffractive optical grating. In some embodiments, the optical collimator 2010 may be a metasurface and / or a liquid crystal grating. In some embodiments, the optical collimator 2010 may take the form of a reflective well.

[0144] Referring now to FIG. 13, another embodiment of a projection system 1003 for forming the diverging wavefront approximations of FIGS. 7B and 8 is illustrated. In some embodiments, the projection system 1003 may include a microdisplay 2000 and an array of optical collimators 2010 disposed between the microdisplay 2000 and the projection optics 1020. The projection system is part of a display system 1001 having a control system 1024 for controlling the projection system 1003 and the eye tracking device 1022, as described above with respect to FIG. 9.

[0145] Continuing to refer to FIG. 13, the microdisplay 2000 may include a plurality of light emitters 2002. The light emitters 2002 may be arranged in groups, each group comprising two or more light emitters. Each group of light emitters 2002 includes an associated optical collimator 2010. As shown, each group 2002g of optical collimators may include two light emitters 20021, 20022, and the associated optical collimator 20101 is configured to capture light from both light emitters. Preferably, the associated optical collimator 20101 has a width that extends across both light emitters 20021, 20022 such that the associated optical collimator 20101 encompasses an area that exceeds the area of both light emitters 20021, 20022. In some embodiments, the optical collimator 2010 is spaced apart from the light emitters 20021, 20022. In some other embodiments, the optical collimator 2010 may be in contact with the light emitters 20021, 20022. For example, the optical collimator 2002 may be a microlenslet, and the light emitters 20021, 20022 may be disposed within the associated microlenslet 20101.

[0146] It should be understood that the location of the light emitter relative to the associated light collimator can affect the direction of the light propagating out from the light collimator. The light emitters 20021, 20022 interface (e.g., are located at different locations) differently from the associated light collimator 20101, which projects the light from each of the light emitters 20021, 20022 along different paths to the projection optical system 1020 and then to the eye 210 for the light collimator 20101. Preferably, the locations of the light emitters 20021, 20022 and the physical parameters (e.g., size and shape) of the associated light collimator 20101 are configured to provide different light paths corresponding to different intra-pupil images. Two light emitters 20021, 20022 are provided for illustrative clarity, but it should be understood that each group of light emitters associated with a light collimator may have more than two light emitters. For example, each of the groups 2002g may have three or more light emitters, four or more light emitters, etc. A greater number of spatially distinct light emitters facilitates a greater number of potential levels of parallax differences.

[0147] In some embodiments, the image light from the light emitters of each group of light emitters may be directed by the associated light collimator 2010 and the projection optical system 1020 to form a first intra-pupil image as the light beam 1010a, and the image light from another light emitter of each group of light emitters may be directed by the associated light collimator 2010 and the projection optical system 1020 to form a second intra-pupil image as the light beam 1010b. As discussed herein, the light beams 1010a, 1010b preferably have a small enough diameter such that the eye cannot accommodate for the individual images formed by each light beam. In some embodiments, the diameter of each light beam 1010a, 1010b at the pupil of the eye 210 is about 0.5 mm or less (e.g., 0.2 - 0.5 mm).

[0148] Continuing to refer to FIG. 13, projection system 1003, which includes groups 2002g of light emitters 20021, 20022, may form different intra-pupil images using spatial multiplexing. With respect to such spatial multiplexing, light emitters at different locations may be used to form different intra-pupil images, i.e., different light emitters of each group of light emitters may provide image information regarding different intra-pupil images. Light from corresponding light emitters of each group of light emitters is directed by an optical structure (e.g., the collimators of collimator array 2010 and projection optics 1020) to form one of light beams 1010a, 1010b for forming a corresponding intra-pupil image. Advantageously, the intra-pupil images are provided to the eye simultaneously, eliminating the need to rapidly sequentially provide images below the flicker fusion threshold. This may reduce the requirements regarding the frame rate of microdisplay 2000 and assist control system 1024.

[0149] Referring now to FIG. 14, it should be understood that the projection system 1003 of FIG. 13 may include both a collimator with associated groups of light emitters and an array of shutters 2004 to provide a combination of spatial and temporal multiplexing. The array of shutters 2004 may further refine the location and / or size of the light beam incident on the eye and forming the intra-pupil image. For example, the light beam output by collimator array 2010 may be larger than desired to form the intra-pupil image (e.g., larger than 0.5 mm in diameter, larger than desired for open-loop operation of the eye). The shutters of shutter array 2004 may be utilized to limit the size of the light beam ultimately incident on eye 210. Additionally, each light beam provided by the collimators of collimator array 2010 may impinge on multiple shutters. As a result, the shutters may provide an additional level of parallax difference for each incident light beam, which may facilitate the presentation of virtual content on a larger number of depth planes than would be provided by the light beams output by collimator array 2010 alone.

[0150] In some embodiments, the display system may be time-multiplexed in one dimension (e.g., along the x-axis) and spatially multiplexed in another dimension (e.g., along an orthogonal dimension such as the y-axis). For example, the optical collimator may be a lenticular lenslet instead of a spherical lenslet, and may be configured to provide different beams of light from spatially distinct light emitters to different locations along a first axis that is orthogonal to a second axis that may be the elongation axis of the lenticular lenslet. The array of shutters may be utilized to form sub-pupils at different points and at different times along that second axis. For example, the positioning of the sub-pupils along the first axis may be achieved by spatial multiplexing by activating different light emitters of each group of light emitters associated with a lenticular lenslet (elongated along the second axis), while the positioning of the sub-pupils along the second axis may be provided by time multiplexing by opening the shutters on the second axis at a desired time (e.g., a time that overlaps with the activation of the corresponding light emitters used for spatial multiplexing).

[0151] Referring now to FIG. 15, an example of the range of depth planes provided by the projection system disclosed herein is illustrated. The range extends from a far plane at optical infinity to a near plane closer to the eye 210. The far plane at optical infinity may be provided by an intra-pupil image formed by light beams (e.g., light beams 1010a and 1010b) that are substantially collimated or parallel to each other. With respect to such a far plane, the light beams may be understood to have no or low parallax differences.

[0152] Continuing to refer to FIG. 15, closer depth planes may be provided using higher levels of disparity difference. In some embodiments, the perceived proximity of the proximity plane to the eye 210 may be determined by the maximum disparity difference between the light beams forming the intra-pupil image. For example, the maximum disparity difference between the light beams 1010a and 1010b may be determined by the maximum distance between the shutters of the shutter array 2004 (FIGS. 9-12 and 14) and / or the maximum difference in the direction provided by each collimator 1010a for the light from the associated group of light emitters 2002 (FIGS. 13 and 14) of the light emitter.

[0153] Referring now to FIG. 16A, an example of a display system is illustrated that includes a projection system including a shutter array and an intra-pupil relay combiner eyepiece for superimposing image content onto the view of a user of the world. Preferably, the eyepiece 1030 is optically transmissive so that the viewer can see the world and allows light 2012 from the world to propagate through the eyepiece into the viewer's eye 210. In some embodiments, the eyepiece 1030 includes one or more waveguides 1032 having internal coupling optical elements, such as internal coupling optical element 770 and external coupling optical element 800, and external coupling optical elements. The internal coupling optical element 770 receives the image light 2014 from the projection system 1003 and redirects the image light so that it propagates through the eyepiece 1030 to the external coupling optical element 800 by total internal reflection. The external coupling optical element 800 outputs the image light 2014 to the viewer's eye 210. Advantageously, the eyepiece 1030 preserves all of the attributes of the image provided by the projection system 1003, and thus the rapid-switching parallax views are accurately depicted through the eyepiece 1030. It should be understood that the image light 2014 is light emitted by the microdisplay 2000 and may correspond, for example, to light 200a, 2002b, 2002c (FIG. 9) or light provided through the array of collimators 2010 (FIGS. 13 and 14).

[0154] As shown, the projection system 1003 may include an array of shutters 2004 on the pupil plane 2005 through which light exits the projection system 1003 and travels towards the internal coupling optical element 770. A lens structure 2016 may be provided between the shutter array 2004 and the internal coupling optical element 770 to relay the image light 2014. In some embodiments, the lens structure 2016 may also collimate the image light 2014 for propagation within the waveguide 1032 that forms the eyepiece lens 1030.

[0155] The internal coupling optical element 770 and the external coupling optical element 800 may be refractive or reflective structures. Preferably, the internal coupling optical element 770 and the external coupling optical element 800 are diffractive optical elements. Examples of diffractive optical elements include surface relief features, three-dimensional phase features, metamaterials, or liquid crystal polarization gratings. Although shown disposed on the same side of the waveguide 1032, it should be understood that the internal coupling optical element 770 and the external coupling optical element 800 may be disposed on different sides of the waveguide 1032. Also, although shown on the side of the waveguide 1032 facing the projection optics 1020, one or both of the internal coupling optical element 770 and the external coupling optical element 800 may be disposed on the same side of the waveguide 1032 as the projection optics 1020.

[0156] FIG. 16B illustrates an embodiment of a display system comprising a projection system that includes an array of light collimators for providing different intra-pupil images and a pupil relay combiner eyepiece for superimposing image content onto the view of a user of the world. The display system of FIG. 16B is similar to that of FIG. 16A, except that the projection system 1003 is similar to that of FIG. 13 rather than the projection system of FIG. 9. As shown, the projection system 1003 may include a microdisplay 2000 and an array of light collimators 2010 between the microdisplay 2000 and the projection optics 1020. The microdisplay 2000 may include a plurality of light emitters formed within a group 2002g with one group per light collimator 2010. The location of the light emitters relative to the associated light collimator determines the direction of light propagating out from the light collimator and thus the parallax difference between images formed using different light emitters 20021, 20022 of the group of light emitters 2002g. As a result, as described herein, the activation of different light emitters within a group of light emitters sets the parallax difference between intra-pupil images formed using different light emitters. In some embodiments, the light from the first light emitter 20021 of each group 2002g of light emitters is directed by the associated light collimator 2010 and projection optics 1020 to form a first intra-pupil image with a light beam 1010a, and the light from the second light emitter 20022 of each group 2002g of light emitters may be directed by the associated light collimator 2010 and projection optics 1020 to form a second intra-pupil image with a light beam 1010b.

[0157] It should be understood that any of the embodiments of the projection systems disclosed herein may be utilized within the display systems of FIGS. 16A and 16B, for example, as the projection system 1003 of these figures. For example, the projection system 1003 may utilize both an array of light collimators and an array of shutters, as shown, for example, in FIGS. 12 and 14.

[0158] Figure 17A illustrates another embodiment of a projection system that includes an array of shutters and an intra-pupil relay combiner eyepiece for superimposing image content onto the view of a user around the world. The illustrated display system is similar to that of FIG. 16A, except for the optical structure associated with the eyepiece 1030. The eyepiece 1030 may include a partially transparent relay mirror 2020 to reflect the image light 2014 into the eye 210. As illustrated, the image light 2014 is internally coupled by the internal coupling optical element 770, which redirects the light 2014 to propagate through the waveguide 1032. The eyepiece 1030 includes an external coupling optical element 800, which directs the image light 2014 forward through a quarter-wave plate 2018 on the front surface of the waveguide 1032. As illustrated, the partially transparent relay mirror 2020 may be in front of and spaced from the quarter-wave plate 2018 and the waveguide 1032.

[0159] Figure 17B illustrates another embodiment of a display system that includes an array of light collimators for providing different intra-pupil images and an intra-pupil relay combiner eyepiece for superimposing image content onto the view of a user around the world. The illustrated display system is similar to that of FIG. 16B, except for the optical structure associated with the eyepiece 1030. The eyepiece 1030 and the associated structure are similar to those illustrated in FIG. 17A. As described above, the image light 2014 is internally coupled by the internal coupling optical element 770 and propagates through the waveguide 1032 by total internal reflection. The external coupling optical element 800 outputs the image light 2014 forward through the quarter-wave plate 2018 to the partially transparent relay mirror 2020 away from the eye 210. As illustrated, the partially transparent relay mirror 2020 may be in front of and spaced from the quarter-wave plate 2018 and the waveguide 1032 and may reflect the image light 2014 into the eye 210. In some embodiments, the partially transparent relay mirror 2020 may be configured to selectively reflect polarized light transmitted through the quarter-wave plate 2018.

[0160] Referring now to FIG. 18, an embodiment of a display system 1001 having a projection system 1003 is illustrated, which includes an eye tracking system 1022 and a combiner eyepiece 1030 with a pupil expander 1034. The pupil expander 1034 may include, for example, a diffractive optical element configured to replicate the projection system pupil across the eyepiece 1030. Since the pupil expander 1034 replicates the projection system pupil across a large area that can be traversed by the viewer's pupil through eye movement, the location of the image formed by the microdisplay 2000 and the emission region of the light source 1026 can be updated in real time based on the input from the eye tracking system 1022. Advantageously, this configuration allows for a larger eyebox for more comfortable viewing and relaxes the restrictions regarding the relative positioning of the eye and the combiner and the variation in interpupillary distance.

[0161] Referring now to FIG. 19, an embodiment of a display system 1001 having a projection system 1003 is illustrated, which includes an eye tracking system 1022 and a combiner eyepiece 1030 with a pupil expander 1035 configured to produce a non-infinite depth plane. In some embodiments, the non-infinite depth plane may be at 3 meters, which provides a focus adjustment within a budget of up to about 2.5 meters to infinity. For example, assuming the tolerances of the human visual system regarding focus-convergence / divergence motion mismatch, virtual content at a distance of up to about 2.5 meters from the viewer to infinity can be placed on the 3-meter depth plane with little discomfort. In such a system, the differentially different intra-pupil images may potentially all be used to drive the focus adjustment for a narrower range of depth planes that are closer to the viewer than the fixed "default" focal plane. In some embodiments, the system may also incorporate the eye tracking system 1022 and, for example, determine the viewer's fixation distance based on the convergence / divergence motion angles of both eyes of the viewer.

[0162] It should be understood that the microdisplay 2000 may be a monochrome display, and the display system 1001 may be configured to provide a monochrome image to the eye 210. More preferably, the display system 1001 is configured to provide a full-color image to the eye 210. In such embodiments, the microdisplay 2000 may be a full-color microdisplay. For example, a full-color image may be formed by providing different images formed by different primary colors (e.g., three or more primary colors such as red, green, and blue), which, in combination, are perceived by the viewer as a full-color image. The microdisplay 2000 may be configured to emit light of all primary colors. For example, different colors may be emitted by different light emitters.

[0163] In some other embodiments, the full-color image may be formed using primary color images provided by a plurality of microdisplays, at least some of which are monochrome microdisplays. For example, different monochrome microdisplays may be configured to provide different primary color images. The primary color images may be provided to the eye 210 simultaneously or multiplexed in time (e.g., all different primary color images for forming a single full-color image may be provided to the eye 210 within the flicker fusion threshold).

[0164] Figures 20A-20B illustrate an embodiment of a projection system having a plurality of light-emitting microdisplays 2000a, 2000b, 2000c. The microdisplays 2000a, 2000b, 2000c may each be similar to the microdisplay 2000 disclosed herein (see, e.g., FIGS. 9, 11-14, and 16A-17B). Light from the microdisplays 2000a, 2000b, 2000c is combined by an optical combiner 1050 and directed towards a projection optical system 1020 and ultimately towards the viewer's eye 210. As discussed herein, in some embodiments, the light from the projection optical system 1020 may be directed to an eyepiece 1030, which may be a waveguide assembly comprising one or more waveguides.

[0165] In some embodiments, the microdisplays 2000a, 2000b, 2000c may be monochromatic microdisplays. Each monochromatic microdisplay may output light of a different primary color and provide a different monochromatic image, which may be combined by the viewer to form a full-color image.

[0166] Continuing to refer to FIG. 20A, the optical combiner 1050 receives image light 2001a, 2001b, 2001c from each of the microdisplays 2000a, 2000b, 2000c, respectively, and combines this light such that the light propagates generally in the same direction, e.g., towards the projection optical system 1020. In some embodiments, the optical combiner 1050 may be a dichroic X-cube prism having reflective internal surfaces 1052, 1054 that redirect the image light 2001c, 2001a, respectively, towards the projection optical system 1020. The projection optical system 1020 converges or focuses the image light that impinges on an array of shutters 2004. The image light 2001a, 2001b, 2001c then propagates into the eye 210 through an open shutter (e.g., shutter 20061).

[0167] Referring now to FIG. 20B, an embodiment of a light projection system 1010 is illustrated, with a plurality of light emitting microdisplays 2000a, 2000b, 2000c and associated arrays 2010a, 2010b, 2010c of optical collimators. Arrays 2010a, 2010b, 2010c may each be similar to array 2010 disclosed herein (see, e.g., FIGS. 12 - 14, 16B, and 17B).

[0168] In some embodiments, arrays 2010a, 2010b, 2010c may each include an optical collimator configured to narrow the angular emission profile of image light 2001a, 2001b, 2001c emitted by microdisplays 1030a, 1030b, 1030c. Image light 2001a, 2001b, 2001c then propagates through optical combiner 1050 to projection optics 1020 and then into eye 210. Additionally, each optical collimator may have an associated group of light emitters (as illustrated, e.g., in FIGS. 13, 14, 16B, and 17B) as discussed herein, and light from different light emitters of each group of emitters may be directed along different paths corresponding to different amounts of parallax differences for forming different intra - pupil images.

[0169] Referring to FIGS. 20A and 20B, in some other embodiments, the microdisplays 2000a, 2000b, 2000c may each be full-color displays configured to output light of all primary colors. For example, the microdisplays 2000a, 2000b, 2000c may each include light emitters configured to emit red, green, and blue light. The microdisplays 2000a, 2000b, 2000c may be the same and may display the same image. However, using multiple microdisplays may provide the advantage of increasing the brightness and brightness dynamic range of the image by combining light from the multiple microdisplays to form a single image. In some embodiments, two or more (e.g., three) microdisplays may be used with an optical combiner 1050 configured to combine light from all of these microdisplays.

[0170] Continuing to refer to FIGS. 20A and 20B, in some embodiments, the monochrome microdisplay 2000b directly facing the output surface 1051 may advantageously output green light. It should be understood that the reflective surfaces 1052, 1054 may have optical losses when reflecting light from the microdisplays 2000c, 2000a, respectively. In addition, of the primary colors red, green, and blue, the human eye is most sensitive to the color green. As a result, the monochrome microdisplay 2000b facing the output surface of the optical combiner 1050 preferably outputs green light so that the green light can travel directly through the optical combiner 1050 without being reflected so that the green light is output from the optical combiner 1050. However, it will be understood that the green monochrome microdisplay may face other surfaces of the optical combiner 1050 in some other embodiments.

[0171] In some embodiments, as discussed herein, the display system may include an eyepiece (e.g., eyepiece 1030, FIGS. 16A - 19) and relay light output by the projection optical system 1020 to the eye 210. In some embodiments, the eyepiece may include a single waveguide configured to internally and externally couple light of all primary colors.

[0172] In some other embodiments, the eyepiece 1030 may include a plurality of waveguides that form a stack of waveguides. Each waveguide has an individual internal coupling optical element for internally coupling image light. For example, each waveguide may have an associated internal coupling optical element configured to internally couple light of a different primary color or a different range of wavelengths. In some embodiments, the number of waveguides is proportional to the number of primary colors provided by the microdisplays 2000a, 2000b, 2000c. For example, if there are three primary colors, the number of waveguides in the eyepiece 1030 may include a set of three waveguides or a plurality of sets of three waveguides. Examples of the arrangement of waveguides and associated structures are discussed in U.S. Provisional Application No. 62 / 800363, filed on February 1, 2019, the entire disclosure of which is incorporated herein by reference.

[0173] Referring now to FIG. 21, an embodiment of an eyepiece 660 (which may correspond to the eyepiece 1030 of FIGS. 16A - 19) is illustrated that includes a stacked waveguide assembly for outputting light of different wavelengths corresponding to different primary colors. In some embodiments, the waveguide assembly includes waveguides 670, 680, and 690, which may individually correspond to waveguide 1032 (FIGS. 16A - 17B). Each waveguide includes an associated internal coupling optical element. For example, internal coupling optical element 700 is disposed on a major surface (e.g., the upper major surface) of waveguide 670, internal coupling optical element 710 is disposed on a major surface (e.g., the upper major surface) of waveguide 680, and internal coupling optical element 720 is disposed on a major surface (e.g., the upper major surface) of waveguide 690. In some embodiments, one or more of the internal coupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the individual waveguides 670, 680, 690 (in particular, one or more of the internal coupling optical elements are reflective deflecting optical elements). As shown, the internal coupling optical elements 700, 710, 720 may be disposed on the upper major surface of their individual waveguides 670, 680, 690 (or the top of the next lower waveguide), and in particular, those internal coupling optical elements are transmissive deflecting optical elements. In some embodiments, the internal coupling optical elements 700, 710, 720 may be disposed within the body of the individual waveguides 670, 680, 690. In some embodiments, as discussed herein, the internal coupling optical elements 700, 710, 720 are wavelength selective such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of their individual waveguides 670, 680, 690, it should be understood that in some embodiments, the internal coupling optical elements 700, 710, 720 may be disposed within other areas of their individual waveguides 670, 680, 690.

[0174] As shown, the internally coupled optical elements 700, 710, 720 may be laterally offset from each other. In some embodiments, each internally coupled optical element may be offset such that its light is received without passing through another internally coupled optical element. For example, each of the internally coupled optical elements 700, 710, 720 may be configured to receive light from different image input devices 360, 370, 380, 390, and 400, as shown in FIG. 6, and may be separated (e.g., laterally spaced) from the other internally coupled optical elements 700, 710, 720 such that it receives substantially no light from the other internally coupled optical elements 700, 710, 720. In some embodiments, the internally coupled optical elements 700, 710, 720 are vertically aligned and not laterally offset.

[0175] Each waveguide may also include an associated light dispersing element. For example, the light dispersing element 730 may be disposed on a major surface (e.g., upper major surface) of the waveguide 670, the light dispersing element 740 may be disposed on a major surface (e.g., upper major surface) of the waveguide 680, and the light dispersing element 750 may be disposed on a major surface (e.g., upper major surface) of the waveguide 690. In some other embodiments, the light dispersing elements 730, 740, 750 may each be disposed on the bottom major surface of the associated waveguides 670, 680, 690. In some other embodiments, the light dispersing elements 730, 740, 750 may each be disposed on both the upper and bottom major surfaces of the associated waveguides 670, 680, 690, or the light dispersing elements 730, 740, 750 may each be disposed on different ones of the upper and bottom major surfaces within different associated waveguides 670, 680, 690.

[0176] Waveguides 670, 680, 690 may be separated and isolated, for example, by gas, liquid, and / or solid layers of material. For example, as shown, layer 760a may separate waveguides 670 and 680, and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed from a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediate vicinity of waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or more or 0.10 or less relative to the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the lower refractive index layers 760a, 760b may function as cladding layers that promote TIR of light through waveguides 670, 680, 690 (e.g., TIR between the upper and bottom major surfaces of each waveguide). In some embodiments, layers 760a, 760b are formed from air. It should be understood that the upper and bottom of the illustrated set 660 of waveguides may include an immediate cladding layer, although not shown.

[0177] Continuing to refer to FIG. 21, light rays 770, 780, 790 are incident on and introduced into waveguides 670, 680, 690 by projection system 1003 (FIGS. 9, 11-14, and 16A-17B).

[0178] In some embodiments, light rays 770, 780, 790 have different properties, such as different wavelengths or different wavelength ranges, which may correspond to different colors. Internal coupling optical elements 700, 710, 720 each deflect the incident light so that the light propagates through an individual one of waveguides 670, 680, 690 by TIR.

[0179] For example, the internal coupling optical element 700 may be configured to deflect the light ray 770 having a first wavelength or wavelength range. Similarly, the transmitted light ray 780 impinges on an internal coupling optical element 710 configured to selectively deflect light of a second wavelength or wavelength range, and is thereby deflected. Similarly, the light ray 790 is deflected by an internal coupling optical element 720 configured to selectively deflect light of a third wavelength or wavelength range.

[0180] Continuing to refer to FIG. 21, the internally coupled light rays 770, 780, 790 are each deflected by internal coupling optical elements 700, 710, 720, and then each propagate by TIR within the waveguides 670, 680, 690. The light rays 770, 780, 790 then each impinge on light dispersion elements 730, 740, 750. The light dispersion elements 730, 740, 750 deflect the light rays 770, 780, 790 so as to each propagate towards external coupling optical elements 800, 810, 820.

[0181] In some embodiments, the light dispersion elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPE may deflect or disperse light towards the external coupling optical elements 800, 810, 820 and may increase the beam or spot size of the present light as it propagates towards the external coupling optical elements. In some embodiments, for example, if the beam size is already the desired size, the light dispersion elements 730, 740, 750 may be omitted and the internal coupling optical elements 700, 710, 720 may be configured to deflect light directly towards the external coupling optical elements 800, 810, 820. In some embodiments, the external coupling optical elements 800, 810, 820 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light towards the viewer's eye 210 (see FIG. 18). It should be understood that the OPE may be configured to increase the size of the eyebox in at least one axis, and the EPE may increase the eyebox in an axis that intersects, for example, is orthogonal to, the axis of the OPE.

[0182] Thus, in some embodiments, the eyepiece 660 includes, for each primary color, waveguides 670, 680, 690, internal coupling optical elements 700, 710, 720, light dispersion elements (e.g., OPE) 730, 740, 750, and external coupling optical elements (e.g., EP) 800, 810, 820. The waveguides 670, 680, 690 may be stacked with a gap / cladding layer between each one. The internal coupling optical elements 700, 710, 720 redirect or deflect the incident light into their respective waveguides (using different internal coupling optical elements that receive light of different wavelengths). The light then propagates at an angle that will result in TIR within the individual waveguides 670, 680, 690. In the illustrated embodiment, the ray 770 (e.g., blue light) is polarized by the first internal coupling optical element 700 in the manner described above, then bounces along the waveguide and interacts with the light dispersion element (e.g., OPE) 730, then the external coupling optical element (e.g., EP) 800. The rays 780 and 790 (e.g., green and red light, respectively) pass through the waveguide 670, and the ray 780 collides with the internal coupling optical element 710 and is thereby deflected. The ray 780 then bounces along the waveguide 680 via TIR and will proceed to its light dispersion element (e.g., OPE) 740, then the external coupling optical element (e.g., EP) 810. Finally, the ray 790 (e.g., red light) passes through the waveguide 690 and collides with the internal coupling optical element 720 of the waveguide 690. The internal coupling optical element 720 deflects the ray 790 such that the ray propagates by TIR to the light dispersion element (e.g., OPE) 750, then by TIR to the external coupling optical element (e.g., EP) 820. The external coupling optical element 820 then finally externally couples the ray 790 to the viewer, who also receives the externally coupled light from the other waveguides 670, 680.

[0183] As described above, the internal coupling optical elements 700, 710, 720 may be laterally offset from each other. In such embodiments, the image light 2001a, 2001b, 2001c from different ones of the microdisplays 2000a, 2000b, 2000c may follow different paths to the eyepiece 1030 such that they impinge on different ones of the internal coupling optical elements 700, 710, 720. If the image light 2001a, 2001b, 2001c includes light of different primary colors, the associated internal coupling optical elements 700, 710, 720 may each be configured to selectively internally couple light of different wavelengths, as discussed herein.

[0184] The different optical paths for the image lights 2001a, 2001b, 2001c may be provided by the combiner 1050 (FIGS. 20A-20B), by one or more of the arrays 2010a, 2010b, 2010c, and / or by angling one or more of the microdisplays 2000a, 2000b, 2000c at an appropriate angle with respect to the reflective surfaces 1052, 1054 of the optical combiner 1050. For example, referring to FIGS. 20A-20B, the optical combiner 1050 may be configured to redirect the image lights 2001a, 2001b, 2001c emitted by the microdisplays 2000a, 2000b, 2000c so that the image lights propagate along different optical paths for the image lights to impinge on the associated ones of the internal coupling optical elements 700, 710, 720. Thus, the optical combiner 1050 combines the image lights 2001a, 2001b, 2001c in the sense that the light may exit the optical combiner in slightly different directions, but the image lights are output from a common plane of the optical combiner 1050. For example, the reflective internal surfaces 1052, 1054 of the X-cube prism may each be angled to direct the image lights 2001a, 2001b, 2001c along different paths to the eyepiece lens 1030. As a result, the image lights 2001a, 2001b, 2001c may impinge on different associated ones of the internal coupling optical elements 700, 710, 720. In some embodiments, the microdisplays 2000a, 2000b, 2000c may be appropriately angled with respect to the reflective internal surfaces 1052, 1054 of the X-cube prism to provide the desired optical paths to the internal coupling optical elements 700, 710, 720. For example, the surfaces of one or more of the microdisplays 2000a, 2000b, 2000c may be angled to match the surfaces of the optical combiner 1050 such that the image light emitted by the microdisplay impinges on the reflective internal surfaces 1052, 1054 at an appropriate angle and propagates towards the associated internal coupling optical elements 700, 710, 720. In addition to a cube, it should be understood that the optical combiner 1050 may take the form of various other polyhedra.For example, the optical combiner 1050 may be in the shape of a right-angled prism having at least two non-square faces.

[0185] As discussed herein, the perception of a full-color image by a user may be achieved using time-division multiplexing in some embodiments. For example, different ones of the micro-LED arrays 2000a, 2000b, 2000c may be activated at different times to generate different primary color images. In such embodiments, the different primary color images that form a single full-color image may be sequentially displayed sufficiently quickly such that the human visual system does not perceive the primary color images as being displayed at different times, i.e., all of the different primary color images that form a single full-color image may be displayed within a duration that is sufficiently short such that the user perceives the primary color images as being presented simultaneously rather than being temporally separated. For example, the primary color images may be sequentially displayed at a frame rate higher than the flicker fusion threshold of the human visual system. As an example, the flicker fusion threshold may be 60 Hz, which is considered to be sufficiently fast such that most users do not perceive the primary color images as being displayed at different times. In some embodiments, the different primary color images are sequentially displayed at a rate higher than 60 Hz. It should be understood that time-division multiplexing may advantageously reduce the computational load on a processor (e.g., a graphics processor) utilized to form the displayed image. In some other embodiments, such as when sufficient computational resources are available, all of the primary color images that form a full-color image may be simultaneously displayed by the micro-displays 2000a, 2000b, 2000c.

[0186] Referring now to FIG. 22, an example of a wearable display system 60 is illustrated. The display system 60 may correspond to the display system 1001 of FIGS. 9 and 13 and include a projection system 1003 for each eye of the viewer or user 90.

[0187] The display system 60 includes a display 70 and various mechanical and electronic modules and systems for supporting the functions of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a display system user or viewer 90 and is configured to position the display 70 in front of the user's 90 eyes. The display 70 may be considered eyewear in some embodiments. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned adjacent to the user's 90 outer ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other outer ear canal to provide stereo / adjustable sound control). In some embodiments, the display system may also include one or more microphones 110 or other devices that may detect sound. In some embodiments, the microphone is configured to enable the user to provide an input or command (e.g., selection of a voice menu command, natural language question, etc.) to the system 60 and / or may enable audio communication with other persons (e.g., other users of similar display systems). The microphone may further be configured as a peripheral sensor to collect audio data (e.g., sound from the user and / or the environment). In some embodiments, the display system 60 may further include one or more outwardly directed environmental sensors 112 configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user. For example, the environmental sensor 121 may include one or more cameras, which may be positioned outwardly facing to capture an image similar to at least a portion of the user's 90 normal field of view. In some embodiments, the display system may also include a peripheral sensor 120a that is separate from the frame 80 and may be attached to the user's 90 body (e.g., on the user's 90 head, torso, limbs, etc.). The peripheral sensor 120a may be configured to obtain data for characterizing the physiological state of the user 90 in some embodiments. For example, the sensor 120a may be an electrode.

[0188] Continuing to refer to FIG. 22, the display 70 is operatively coupled to a local data processing module 140 by a communication link 130, such as a wired conductor or wireless connectivity, which may be mounted in various configurations, such as fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, built into headphones, or removably attached to the user 90 in another manner (e.g., in a backpack configuration, in a belt attachment configuration). Similarly, the sensor 120a may be operatively coupled to a local processor and data module 140 by a communication link 120b, such as a wired conductor or wireless connectivity. The local processing and data module 140 may include a hardware processor and a digital memory, such as a non-volatile memory (e.g., flash memory or hard disk drive), both of which may be utilized to assist in the processing, caching, and storage of data. The data may include a) data captured from sensors such as an image capture device (e.g., a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, a gyroscope, and / or other sensors disclosed herein (e.g., operatively coupled to the frame 80 or otherwise attachable to the user 90), and / or b) data obtained and / or processed using the remote processing module 150 and / or the remote data repository 160 (including data related to virtual content), possibly for passage to the display 70 after processing or retrieval. The local processing and data module 140 may be operatively coupled to the remote processing module 150 and the remote data repository 160 by communication links 170, 180, such as via a wired or wireless communication link, such that these remote modules 150, 160 are operatively coupled to each other and available as resources to the local processing and data module 140.In some embodiments, the local processing and data module 140 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be attached to the frame 80 or may be of a stand-alone structure that communicates with the local processing and data module 140 via a wired or wireless communication path. In some embodiments, the local processing and data module 140 may include one or more graphic processors and may correspond to the control system 1024 (FIGS. 9 and 13).

[0189] Continuing to refer to FIG. 22, in some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 160 may include a digital data storage facility, which may be available through the Internet or other networking configurations in a “cloud” resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers that provide information, such as information for generating augmented reality content, to the local processing and data module 140 and / or the remote processing module 150. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, enabling completely autonomous use from the remote module.

[0190] The processes, methods, and algorithms described in this specification and / or depicted in the figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being capable of being fully or partially automated. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be installed in a dynamic link library that is compiled and linked into an executable program, or can be written in an interpreted type programming language. In some implementations, certain operations and methods can be implemented by circuits specific to a given function.

[0191] Furthermore, because the functional implementations of the present disclosure are sufficiently mathematically, computationally, or technically complex, a hardware specific to a particular use or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, a video can include many frames, each frame can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0192] A code module or any type of data can be stored on any type of non-transitory computer-readable medium such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. In some embodiments, the non-transitory computer-readable medium may be part of one or more of a local processing and data module (140), a remote processing module (150), and a remote data repository (160). The methods and modules (or data) may also be transmitted as data signals generated on various computer-readable transmission media including wireless-based and wire / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal) and may take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0193] Any process, block, state, step, or functionality in a flowchart described in and / or depicted in the accompanying figures herein is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality may be combined, rearranged, added to, removed from, modified from, or otherwise changed from the illustrative embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states may be performed in a suitable other sequence, e.g., sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed illustrative embodiments. Further, the separation of the various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the program components, methods, and systems described may generally be integrated together in a single computer product or packaged into multiple computer products.

[0194] In the foregoing specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a limiting sense.

[0195] In fact, it should be understood that the systems and methods of the present disclosure each have several innovative aspects, none of which individually contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure.

[0196] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Further, features may be described above as acting in a certain combination and may further be claimed as such, but one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination. No single feature or group of features is necessary or essential to every embodiment.

[0197] In particular, conditional statements used herein such as “~ can”, “~ could”, “~ might”, “~ may”, “for example (e.g.)”, and equivalents, generally convey that while one embodiment includes certain features, elements, and / or steps, other embodiments do not, unless specifically stated otherwise or understood otherwise within the context in which they are used. Thus, such conditional statements are not generally intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are to be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms “comprising”, “including”, “having”, and equivalents are synonyms and are used inclusively in a non-limiting manner, without excluding additional elements, features, acts, operations, etc. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Additionally, the articles “a”, “an”, and “the” as used in this application and the appended claims should be construed to mean “one or more” or “at least one” unless otherwise defined. Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is not necessary for achieving the desired result, and that such operations may be performed in the particular order shown or in a sequential order, or that all of the illustrated operations need not be performed. Further, the drawings may schematically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted can also be incorporated into the exemplary methods and processes schematically illustrated. For example, one or more additional operations can be performed before, after, simultaneously with, or during any of the illustrated operations.Additionally, the operations may be rearranged or reordered in other embodiments. In certain situations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0198] Accordingly, the claims are not intended to be limited to the embodiments shown herein but are to be accorded the widest scope consistent with the disclosure, principles, and novel features disclosed herein.

Claims

1. A head mounted display system, comprising: a microdisplay comprising an array of groups of light emitters; an array of optical collimators on top of the light emitters, each optical collimator being associated with one of the groups of light emitters and extending across all the light emitters of the associated group of light emitters; a projection optical system, wherein the array of light collimators is between the light emitter and the projection optical system, and the head mounted display system is configured to display the virtual object on a depth plane by projecting a set of parallax-distinct intra-pupillary images of the virtual object into the same eye of a viewer; and one or more processors; and a memory for storing instructions. Equipped with The instructions, when executed by the one or more processors, cause the head mounted display system to: determining each light emitter of the group of light emitters for activation based on a level of parallax difference associated with an image formed by the light emitter; forming a first parallaxically distinct intrapupillary image by activating a first light emitter of the group of light emitters; forming a second parallaxically distinct intrapupillary image by activating a second light emitter of said group of light emitters; Performing an operation including the first and second parallaxically distinct intrapupillary images providing different views of the virtual object to a same eye of the viewer; a head mounted display system, wherein activating the first light emitter of the group of light emitters overlaps in time with activating the second light emitter of the group of light emitters to project the first and second parallaxically distinct intrapupillary images into the same eye of the viewer within a flicker fusion threshold of 1 / 60 seconds.

2. The head mounted display system of claim 1 , wherein the first and second parallaxically distinct intrapupillary images are simultaneously projected into the same eye of the viewer.

3. The head mounted display system of claim 1 , wherein the light collimator is a lenslet.

4. A head-mounted display system as described in claim 1, further comprising an array of selectively activated shutters for selectively transmitting image light from different locations to the eye, the array of selectively activated shutters being positioned within an eyebox volume of the projection optical system.

5. the light collimators being lenticular lenslets configured to provide different beams of light from light emitters of an associated group of light emitters to different locations along a first axis; 5. The head mounted display system of claim 4, wherein the array of selectively activated shutters is arranged to form a sub-pupil along a second axis orthogonal to the first axis.

6. 5. The head mounted display system of claim 4, further comprising a pupil relay combiner eyepiece configured to relay the image light to the eye of the viewer, the array of selectively activated shutters configured to adjust propagation of the image light to the pupil relay combiner eyepiece, the pupil relay combiner eyepiece comprising a plurality of waveguides, each waveguide comprising an internal coupling optical element and an external coupling optical element.

7. 5. The head mounted display system of claim 4, wherein the array of selectively activated shutters comprises chemical species having reversibly changeable states, the states providing different amounts of light transmission.

Images