Optical system including selective illumination

Abstract

In an embodiment, an apparatus is disclosed that includes at least one processor. The at least one processor is configured to select a light source from a plurality of light sources based at least in part on a location of a pupil of an eye relative to an eye motion box. The selected light source is configured to illuminate a portion of the eye motion box corresponding to the location of the pupil with a light beam. The at least one processor is further configured to activate the selected light source to illuminate the portion of the eye motion box.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 130,957, entitled "DISPLAYS EMPLOYING SELECTIVE EYE PUPIL ILLUMINATION WITH OPTIONAL LIGHT FIELD PROJECTION," filed December 28, 2020, and U.S. Provisional Application No. 63 / 121,937, entitled "DISPLAYS EMPLOYING SELECTIVE EYE MOTION BOX ILLUMINATION," filed December 6, 2020, the entire contents of each of which are incorporated herein by reference. [Background technology]

[0002] The present disclosure relates to optical systems, and more particularly, to optical systems with selective illumination that may be used in near-eye display systems in some embodiments.

[0003] Optical systems, such as near-eye display systems, typically illuminate a user's eye in a manner that can lead to potential aberrations that result in reduced image quality. For example, an optical system may illuminate the entire pupil with a light beam of an image. Due to aberrations in the light beam, such as coma, astigmatism, or other aberrations, portions of the image may become blurred and distorted as they pass through the pupil and reach the retina. Summary of the Invention

[0004] In an embodiment, an apparatus is disclosed that includes at least one processor. The at least one processor is configured to select a light source from a plurality of light sources based at least in part on a location of a pupil of an eye relative to an eye motion box. The selected light source is configured to illuminate a portion of the eye motion box corresponding to the location of the pupil with a light beam. The at least one processor is further configured to activate the selected light source to illuminate the portion of the eye motion box.

[0005] In some embodiments, the selected light source is configured to illuminate with a light beam a portion of the eye motion box corresponding to only a portion of the pupil.

[0006] In an embodiment, the at least one processor is configured to determine a distortion to be applied to the light beam based at least in part on the selected light source, and to cause a modification of the light beam based at least in part on the determined distortion.

[0007] In another embodiment, determining the distortion to apply to the light beam based at least in part on the selected light source includes determining a correction to the light beam for aberrations induced by a collimator.

[0008] In some embodiments, causing the modification of the light beam based at least in part on the determined distortion includes causing a spatial light modulator to modify the light beam based at least in part on the determined distortion.

[0009] In an embodiment, the light beam illuminates a portion of the eye motion box based at least in part on a plurality of elements in the coupling-out arrangement, and at least one of the reflectivity and intensity of each of the elements is selectively adjustable between at least two states. The at least one processor is further configured to determine a target state for a given element of the plurality of elements based at least in part on the selected light source, and to cause the given element to be set to the target state.

[0010] In another embodiment, the light source is a first light source, and the at least one processor is configured to select a second light source from the plurality of light sources, the second light source configured to illuminate a portion of the eye motion box.

[0011] In some embodiments, a method is disclosed that includes selecting a light source from a plurality of light sources based at least in part on a location of a pupil of an eye relative to an eye motion box, the selected light source being configured to illuminate a portion of the eye motion box corresponding to the location of the pupil with a light beam, the method further including activating the selected light source to illuminate the portion of the eye motion box.

[0012] In some embodiments, the selected light source is configured to illuminate with a light beam a portion of the eye motion box corresponding to only a portion of the pupil.

[0013] In an embodiment, the method further includes determining a distortion to be applied to the light beam based at least in part on the selected light source, and causing a modification of the light beam based at least in part on the determined distortion.

[0014] In another embodiment, determining the distortion to apply to the light beam based at least in part on the selected light source includes determining a correction to the light beam for aberrations induced by a collimator.

[0015] In some embodiments, causing the modification of the light beam based at least in part on the determined distortion includes causing a spatial light modulator to modify the light beam based at least in part on the determined distortion.

[0016] In an embodiment, the light beam illuminates a portion of the eye motion box based at least in part on a plurality of elements in a coupling-out arrangement, wherein at least one of the reflectivity and intensity of each of the elements is selectively adjustable between at least two states. The method further includes determining a target state for a given element of the plurality of elements based at least in part on the selected light source, and causing the given element to be set to the target state.

[0017] In another embodiment, the light source is a first light source and the method further includes selecting a second light source from the plurality of light sources, the second light source configured to illuminate a portion of the eye motion box.

[0018] In one embodiment, an optical system is disclosed. The optical system includes a plurality of light sources, and the light-directing optical element includes a coupling-out arrangement configured to direct light beams received from the plurality of light sources toward an eye-motion box of the optical system. The optical system further includes a controller configured to select a light source from the plurality of light sources based at least in part on a location of a pupil of the eye relative to the eye-motion box. The selected light source is configured to emit a light beam that illuminates a portion of the eye-motion box corresponding to the location of the pupil when directed by the coupling-out arrangement. The controller is further configured to activate the selected light source to illuminate the portion of the eye-motion box.

[0019] In some embodiments, the optical system further includes an eye motion tracking system configured to determine a location of a pupil, and the controller is configured to determine a portion of the eye motion box that corresponds to the location of the pupil determined by the eye motion tracking system.

[0020] In an embodiment, the optical system further includes a spatial light modulator disposed between the plurality of light sources and the light-directing optical element, the controller configured to determine a distortion to be applied to the light beam based at least in part on the selected light sources, and the spatial light modulator configured to modify the light beam based at least in part on the determined distortion.

[0021] In another embodiment, the optical system further includes an optical arrangement configured to direct a light beam from a selected light source toward the spatial light modulator, the optical arrangement including a first lens, a second lens, a first microlens array disposed between the first lens and the second lens, and a second microlens array disposed between the first microlens array and the second lens.

[0022] In some embodiments, the plurality of light sources are located in a focal plane of a first lens, the second microlens array is located in a focal plane of the first microlens array, and the spatial light modulator is located in a focal plane of the second lens.

[0023] In an embodiment, the coupling-out arrangement includes a plurality of elements, and the controller is configured to selectively adjust at least one of the reflectivity and intensity of each of the elements between at least two states.

[0024] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. In the drawings, like reference numbers indicate identical or functionally similar elements. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram of an exemplary optical system, according to an embodiment. [Figure 2A]2 is a schematic diagram illustrating an exemplary image projection assembly of the optical system of FIG. 1 for 2D pupil dilation, according to an embodiment. [Figure 2B] 2 is a schematic diagram illustrating an exemplary image projection assembly of the optical system of FIG. 1 for 2D pupil dilation, according to an embodiment. [Figure 2C] 2 is a schematic diagram illustrating an exemplary image projection assembly of the optical system of FIG. 1 for 2D pupil dilation, according to an embodiment. [Figure 3] 2 is a schematic diagram illustrating an exemplary image projection assembly of the optical system of FIG. 1 for 1D pupil dilation, according to an embodiment. [Figure 4] 1 is a diagram illustrating the projection of a light beam onto a surface, according to an embodiment. [Figure 5] 5 illustrates the projection of the light beam of FIG. 4 through a sub-aperture onto a surface, according to an embodiment. [Figure 6A] 2A-2C are schematic diagrams illustrating beams of light projected by a full aperture of a projection optical device (POD) of the exemplary image projection assembly of FIGS. 2A-2C, according to an embodiment. [Figure 6B] 6B illustrates the beam of light of FIG. 6A being projected onto the eye, according to an embodiment. [Figure 7A] 2A-2C is a schematic diagram illustrating a beam of light projected by a first sub-aperture of the POD of the exemplary image projection assembly of FIGS. 2A-2C, according to an embodiment. [Figure 7B] 7B illustrates the beam of light of FIG. 7A being projected onto the eye, according to an embodiment. [Figure 8A] 2A-2C is a schematic diagram illustrating a beam of light projected by a second sub-aperture of the POD of the exemplary image projection assembly of FIGS. 2A-2C, according to an embodiment. [Figure 8B] 8B illustrates the beam of light of FIG. 8A being projected onto the eye, according to an embodiment. [Figure 9A] 8A-8B are diagrams illustrating exemplary aberration plots of the beams of light of FIGS. 6A-8B projected onto the eye, according to embodiments. [Figure 9B]8A-8B are diagrams illustrating exemplary aberration plots of the beams of light of FIGS. 6A-8B projected onto the eye, according to embodiments. [Figure 9C] 8A-8B are diagrams illustrating exemplary aberration plots of the beams of light of FIGS. 6A-8B projected onto the eye, according to embodiments. [Figure 10] 2 is a schematic diagram illustrating an example POD of the optical system of FIG. 1, according to an embodiment. [Figure 11] 2 is a schematic diagram illustrating an example POD of the optical system of FIG. 1, according to an embodiment. [Figure 12] 12 is a schematic diagram illustrating the optical system of FIG. 1 with a controller controlling the spatial light modulator (SLM) of the POD of FIGS. 10 and 11, according to an embodiment. [Figure 13A] 10A-10C illustrate an exemplary process for partial eye pupil illumination and selective eye motion box (EMB) illumination, according to an embodiment. [Figure 13B] 13B illustrates exemplary aberration plots from the exemplary process of FIG. 13A, according to an embodiment. [Figure 13C] 13B illustrates exemplary aberration plots from the exemplary process of FIG. 13A, according to an embodiment. [Figure 13D] FIG. 13B is a flow diagram of the example process of FIG. 13A, according to an embodiment. [Figure 14] 2 is a schematic diagram illustrating an example POD of the optical system of FIG. 1 having a microlens array (MLA), according to an embodiment. [Figure 15] 2 is a schematic diagram illustrating an example POD of the optical system of FIG. 1 having a microlens array (MLA), according to an embodiment. [Figure 16A] 2 is a schematic diagram illustrating an example image projection assembly of the optical system of FIG. 1, according to an embodiment. [Figure 16B] 2 is a schematic diagram illustrating an example image projection assembly of the optical system of FIG. 1, according to an embodiment. [Figure 16C] 2 is a schematic diagram illustrating an example image projection assembly of the optical system of FIG. 1, according to an embodiment. [Figure 17] 11 is a schematic diagram illustrating a light beam projected by the POD of FIG. 10 into a light-guiding optical element (LOE) of the optical system of FIG. 1 and directed to an EMB, according to an embodiment. [Figure 18A] 11A-11C are diagrams illustrating the paths of light beams from different apertures and different fields of view (FOVs) of the POD of FIG. 10 relative to the location of the pupil of the eye, according to an embodiment. [Figure 18B] 11A-11C are diagrams illustrating the paths of light beams from different apertures and different fields of view (FOVs) of the POD of FIG. 10 relative to the location of the pupil of the eye, according to an embodiment. [Figure 18C] 11A-11C are diagrams illustrating the paths of light beams from different apertures and different fields of view (FOVs) of the POD of FIG. 10 relative to the location of the pupil of the eye, according to an embodiment. [Figure 19A] 2 is a schematic diagram illustrating a POD and illumination elements of the optical system of FIG. 1 having an MLA, according to an embodiment. [Figure 19B] 2 is a schematic diagram illustrating a POD and illumination elements of the optical system of FIG. 1 having an MLA, according to an embodiment. [Figure 19C] 2 is a schematic diagram illustrating a POD and illumination elements of the optical system of FIG. 1 having an MLA, according to an embodiment. [Figure 20] 10A-10C are diagrams illustrating the effect of accommodation changes in the eye on a light beam projected onto the pupil, according to an embodiment. [Figure 21] 10A-10C are diagrams illustrating the effect of accommodation changes in the eye on a light beam projected onto the pupil, according to an embodiment. [Figure 22] 2A-2C showing light beams forming a time-multiplexed bright field image, according to an embodiment; FIG. [Figure 23] 4 is a schematic diagram illustrating the image projection assembly of FIG. 3 showing light beams forming a time-multiplexed bright field image, according to an embodiment. [Figure 24] 1A-1C are diagrams illustrating time-multiplexed bright-field images, according to an embodiment. [Figure 25A]2 is a schematic diagram illustrating an image projection assembly of the optical system of FIG. 1 having a dynamic facet structure, according to an embodiment. [Figure 25B] 2 is a schematic diagram illustrating an image projection assembly of the optical system of FIG. 1 having a dynamic facet structure, according to an embodiment. [Figure 25C] 2 is a schematic diagram illustrating an image projection assembly of the optical system of FIG. 1 having a dynamic facet structure, according to an embodiment. [Figure 26] 26 is a schematic diagram illustrating the dynamic facet structure of FIG. 25, according to an embodiment. [Figure 27A] 2 is a schematic diagram illustrating an example image projection assembly of the optical system of FIG. 1 having a switchable Bragg reflector (SBR), according to an embodiment. [Figure 27B] 2 is a schematic diagram illustrating an example image projection assembly of the optical system of FIG. 1 having a switchable Bragg reflector (SBR), according to an embodiment. [Figure 28] 13 is a schematic diagram illustrating the optical system of FIG. 12 having a controller that further controls at least one of the reflectivity and intensity of the facets of the LOE, according to an embodiment. [Figure 29] FIG. 1 is a flow diagram of an exemplary process for partial eye pupil illumination and selective eye motion box (EMB) illumination using time-multiplexed brightfield imaging, according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] In optical systems, such as near-eye display systems, light rays are output from the display system to a target surface, such as a user's eye, that is proximate to the display system. Often, such optical systems illuminate the entire eye or the entire pupil of the eye when projecting an image. In some cases, such global illumination of the eye or pupil, combined with aberrations in the optical projection system, can degrade the quality of the resulting image for the user. For example, when a light beam passes through the lens of the eye and is focused onto the retina, some portions of the image may be blurred, distorted, or have other aberrations as viewed by the user.

[0027] In some embodiments, the disclosed optical system is configured to reduce or inhibit such aberrations by selectively illuminating only the portion of the pupil required for a user to view the image with good quality. Such selective illumination is also referred to herein as partial-eye pupil illumination. For example, partial-eye pupil illumination may be beneficial for achieving improved image quality compared to full-eye illumination and may utilize a less complex optical system. In some embodiments, partial-eye pupil illumination, in combination with projected image displacement, may create a time-multiplexed bright-field image that may provide a solution to the problem of vergence-accommodation conflict (VAC). VAC occurs when the brain receives cues of mismatch between the distance of a virtual three-dimensional (3D) object, sometimes referred to as vergence, and the focusing distance required for the eyes to focus on the virtual 3D object, sometimes referred to as accommodation.

[0028] In some embodiments, the disclosed optical systems are also, or alternatively, configured to illuminate only a portion of the eye motion box (EMB) at a time, for example, only the portion of the EMB where the eye's pupil is currently located, also referred to herein as selective EMB illumination. Selective EMB illumination may provide increased power efficiency in the optical system compared to illumination of the full EMB because image illumination is distributed over a smaller area with partial EMB illumination.

[0029] Partial eye pupil illumination, time-multiplexed bright field imaging, and selective EMB illumination can be utilized separately or together to provide the above-mentioned and other advantages to an optical system that can be configured as a near-eye display system.

[0030] 1 and 2A-2C, an exemplary optical system 100 is described. Optical system 100 includes an image projection assembly 110, a controller 140, and an eye tracking system 160. Eye tracking system 160, which may be optional, is configured to track the location of a pupil of a user's eye 180 and provide corresponding location information to controller 140. Image projection assembly 110 includes a projection optical device (POD) 112 and a light-directing optical element (LOE) 114 and is configured to project an image onto user's eye 180 using two-dimensional (2D) pupil dilation.

[0031] The POD 112 includes an image generator, a spatial light modulator (SLM) 304 ( FIG. 10 ), or other components typically included in an image projection assembly. Some or all of these components may be arranged on the surface of one or more polarizing beam splitter (PBS) cubes or other prism arrangements. The image generator includes an illumination source that provides illumination, such as a light beam or laser beam, that corresponds to the image projected onto the user's eye 180. Exemplary illumination sources may include light emitting diodes (LEDs), micro-LEDs, or other illumination sources.

[0032] SLM 304 may be implemented as an emissive SLM, including components such as an organic light-emitting diode (OLED) display element, a backlit liquid crystal display (LCD) panel, a micro-LED display, a digital light processing (DLP) chip, or another emissive component, or may be implemented as a reflective SLM, such as a liquid crystal on silicon (LCOS) chip. A beamsplitter cube-block may be interposed between collimating optics and SLM 304 to allow delivery of illumination to the surface of SLM 304.

[0033] SLM 304 is configured to modulate the projected intensity of each pixel of illumination to generate an image. In some embodiments, SLM 304 may provide a light beam from each pixel of the display that diverges in the plane of LOE 114, for example, in the plane of major exterior surfaces 116 and 118 described below. The light beam may be collimated in the plane of LOE 114 after reflection from reflective optical arrangement 122 of LOE 114. In some embodiments, the light beam may be collimated in the plane of LOE 114, but may not be collimated in a plane orthogonal to LOE 114.

[0034] Alternatively, the POD 112 may include a scanning arrangement, e.g., a high-speed scanning mirror, which scans illumination from a light source across the image plane of the POD 112 while varying the intensity of the illumination synchronously with the pixel-by-pixel motion to project the desired intensity for each pixel.

[0035] The POD 112 also includes a coupling-in arrangement, such as a coupling-in reflector, an angled coupling prism, or any other coupling-in arrangement, for injecting image illumination into the LOE 114. In some embodiments, the coupling between the POD 112 and the LOE 114 may include direct coupling, e.g., the POD 112 may contact a portion of the LOE 114, or may include coupling via an additional aperture enlargement arrangement to enlarge the size of the aperture through which the image is injected into the plane of the LOE 114.

[0036] 2A , the POD 112 may be configured to output a light beam 126 using a first aperture size such that the light beam 126, once collimated by the reflective optical arrangement 122 of the LOE 114, has a width D, and to output a second light beam 130 using a second aperture size smaller than the first aperture size such that the light beam 130, once collimated by the reflective optical arrangement 122 of the LOE 114, has a width d that is smaller than the width D.

[0037] LOE 114 includes a waveguide including parallel first and second major exterior surfaces 116 and 118, and edges that are not optically active, as shown, for example, in FIG. 2C . LOE 114 also includes a coupling-out arrangement 120 and a reflective optical arrangement 122, such as a lens. Reflective optical arrangement 122 is configured to collimate illumination passing through LOE 114 while redirecting it back toward coupling-out arrangement 120, as seen, for example, in FIG. 2A . While reflective optical arrangement 122 is described above as a reflective lens, a wide range of other lens types and implementations may alternatively be utilized, including, but not limited to, spherical, aspherical, or freeform refractive lenses, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above, formed from glass or plastic.

[0038] Coupling-out arrangement 120 is configured to direct illumination from LOE 114 toward EMB 128 for projection onto user's eye 180. In some embodiments, coupling-out arrangement 120 is illustrated as a plurality of parallel partially reflective surfaces, also referred to herein as facets 124, disposed within LOE 114 at oblique angles relative to major exterior surfaces 116 and 118 of LOE 114. Facets 124 include angle-dependent coatings that provide high transmission at certain angles and partial reflection at other angles.

[0039] For example, light beam 126 travels through LOE 114 toward reflecting optical arrangement 122 by reflecting off major exterior surfaces 116 and 118, as seen in FIG. 2C. Light beam 126 travels through facet 124 to reflecting optical arrangement 122, which reflects, redirects, and collimates light beam 126 back toward facet 124 with width D, due to, for example, partial reflection at the traveling angle of light beam 126. After reflection by reflecting optical arrangement 122, when collimated light beam 126 encounters facet 124, light beam 126 reflects back into facet 124 with width D, approximately the same as the width of EMB 128, due to, for example, partial reflection at the traveling angle of light beam 126. EMB, and is redirected by facet 124 towards EMB 128.

[0040] Although the description herein refers to a facet-based coupling-out arrangement, any other coupling-out arrangement may alternatively be used, including, for example, a coupling-out arrangement having a diffractive optical element.

[0041] Reflecting optical arrangement 122 may have cylindrical optical power that reflects at least a portion of the illumination back toward facet 124 in an in-plane direction by internal reflection from major exterior surfaces 116 and 118. The illumination after reflection from reflecting optical arrangement 122 is collimated both in planes perpendicular and parallel to major exterior surfaces 116 and 118. Reflecting optical arrangement 122 may be integrated with an edge of LOE 114 and may have a cylindrical axis perpendicular to major exterior surfaces 116 and 118 of LOE 114. In some embodiments, reflecting optical arrangement 122 may include a diffractive optical element with cylindrical power integrated within LOE 114.

[0042] The reflective optical arrangement 122 may have a high reflectivity over a range of angles corresponding to the illumination as it propagates through the LOE 114, and a low reflectivity (e.g., transmissive or absorptive) at angles outside this range. In this manner, the reflective optical arrangement 122 reflects light emitted by the POD 112 propagating within the LOE 114 by internal reflection, while the reflection of light from any other light source may be inhibited. For example, light from an external light source, such as the sun, reaches the reflective optical arrangement 122 at angles in the low reflectivity range and is reflected away from the LOE 114 or absorbed. In this manner, the intensity of potential ghost images caused by external light sources is reduced. In some embodiments, the reflective optical arrangement 122 is formed with a reflectivity that depends on the angle of incidence of the light, for example, using multilayer coating techniques that provide selective reflectivity and the desired angle. In another embodiment, the reflective optical arrangement 122 may be formed using one or more volume Bragg gratings with high diffraction efficiency over a relatively narrow range of angles.

[0043] Referring to FIG. 3, an exemplary image projection assembly 210 according to another embodiment will be described. As illustrated in FIG. 3, like elements have like reference numerals as in the image projection assembly 110 of FIGS. 1 and 2A-2C. For example, the image projection assembly 210 includes a POD 212, an LOE 214, a major exterior surface (not shown), a coupling-out arrangement 220, a facet 224, and other components similar to those described above for the image projection assembly 110. The image projection assembly 210 is configured to project an image onto the user's eye 180 using one-dimensional (1D) pupil dilation. In the image projection assembly 210 embodiment, instead of using the reflective optical arrangement 122 to redirect illumination back onto the facet 224, the POD 212 is attached to the LOE 214 at its top. For example, illumination emitted from the POD 212 propagates through the LOE 214 and is gradually emitted toward the EMB (not shown) via the facet 224 of the coupling-out arrangement 220. In this embodiment, the illumination output by POD 212 is already collimated when it enters LOE 214 .

[0044] 1 , controller 140 includes a computing device having one or more processing devices, memory, or other components. For example, controller 140 may comprise a central processing unit (CPU), a field programmable gate array (FPGA), a microcontroller, a dedicated circuit, or any other component. Controller 140 is configured to control POD 112 to generate and output images to LOE 114 for projection to user's eye 180, as described in more detail below.

[0045] In some embodiments, controller 140 may be integrated into image projection assembly 110 or into a device that includes image projection assembly 110, such as, for example, glasses, a head-mounted display, or another device. In some embodiments, controller 140 may be located remotely from image projection assembly 110. For example, the image projection assembly may include a wired or wireless communication device configured to communicate with controller 140. As an example, controller 140 may be included in a mobile device, or a computing device separate from image projection assembly 110, or as part of a device that includes image projection assembly 110.

[0046] The eye tracking system 160 includes one or more eye tracking cameras, lasers, or other optical devices configured to determine the location of the pupil 182 of the user's eye 180 and generate location information, e.g., coordinates or other location information, corresponding to the location. The location information can be provided to the controller 140 for use in controlling the POD 112 to generate and output images to the LOE 114.

[0047] 2A-9C, partial eye pupil illumination according to some embodiments will now be described and explained in more detail.

[0048] 4, in an exemplary scenario, an ideal lens 400 has an aperture D0. The lens 400 is illuminated by a collimated beam 402, whose wavefront is not perfectly planar and contains one or more optical aberrations. In this exemplary scenario, the lens 400 creates an image at point P0 on a screen 404, which is located in the focal plane of the lens 400. Due to the presence of aberrations in the beam, the image is blurred and has a size d0.

[0049] 5, in another exemplary scenario, an ideal lens 400 is illuminated by the same aberrated beam 402. However, in this exemplary scenario, a diaphragm 406 is placed in front of the lens 400 so that only a subaperture of diameter D1 is illuminated by the beam 402, where D1 is smaller than D0. In this exemplary scenario, the lens 400 generally produces an image at point P1 on the screen 404, which may be different from point P0 in the exemplary scenario of FIG. 4. The image is blurred and has a size d1 that is smaller than the image size d0 of the image produced in the exemplary scenario of FIG. 4.

[0050] 4 and 5, reducing the diameter of the aperture that lens 400 exposes to beam 402 can improve image quality as long as the resulting geometric image size d1 is larger than the diffraction limit of the aperture. Depending on the type of aberrations present in beam 402, the location of the image will change based on the location of the illuminated subaperture D1.

[0051] 6A and 6B illustrate an example scenario using an image projection assembly 110 with a full aperture angle of the POD 112 illuminated by a beam 134. The width of the collimated beam 134 after reflection from the reflective optical arrangement 122 is D EMB FIG. 6B illustrates a user's eye 180, with the width D of the pupil 182 of the eye 180. EP Width greater than D EMB A collimated beam 134 having a size s illuminates the eye 180 and projects an image J onto the retina 186 of the eye 180 through the lens 184 of the eye 180. In this exemplary scenario, the resulting image J projected onto the retina 186 is blurred and has a size s due to aberrations in the beam 134 collected by the pupil 182.

[0052] 7A and 7B illustrate an exemplary scenario using image projection assembly 110, according to an exemplary embodiment, where only a portion of the angular aperture of POD 112 in the plane of LOE 114 is illuminated by beam 136. For example, a selective illumination system within POD 112, as described in more detail below, may be utilized to illuminate only a portion of the angular aperture of POD 112. In this exemplary scenario, the angular aperture of POD 112 in a plane perpendicular to the plane of major exterior surfaces 116 and 118 of LOE 114 may be fully illuminated. The width of collimated beam 136 after reflection from reflective optical arrangement 122 is adjusted to a width D of pupil 182, as shown in FIG. 7B, such that collimated beam 136 illuminates only a portion of pupil 182. EP Collimated beam 136 projects image J2 onto retina 186 through lens 184 of eye 180. Image J2 is projected onto retina 186 at a location different from the location of image J1, is blurred, and has a size s2 that is smaller than the size s1 of image J1.

[0053] 8A and 8B illustrate an exemplary scenario using image projection assembly 110, according to an exemplary embodiment, where only another portion of the angular aperture of POD 112 in the plane of LOE 114 is illuminated by beam 138. For example, a selective illumination system within POD 112 may be utilized to illuminate only a portion of the angular aperture of POD 112; in this example, the illuminated portion of the angular aperture is illuminated by beam 136, different from the portion shown in FIGS. 7A and 7B . Similar to beam 136, the angular aperture of POD 112 in the plane perpendicular to the plane of major external surfaces 116 and 118 of LOE 114 may be fully illuminated by beam 138. The width of collimated beam 138 after reflection from reflecting optical arrangement 122 is adjusted to a width D of pupil 182, as shown in FIG. 8B, such that collimated beam 138 illuminates only a portion of pupil 182. EPCollimated beam 138 projects image J3 onto retina 186 through lens 184 of eye 180. Image J3 is projected onto retina 186 at a location different from the locations of both images J1 and J2, is blurred, and has a size s3 that is smaller than the size s1 of image J1.

[0054] In each example scenario, the locations J1, J2, and J3 of the projected images on the retina 186 may be defined as the centroid of the illuminated location. In this manner, the locations J1, J2, and J3 of the projected images in Figures 6A-8B may be considered different even if one or more portions of the projected images overlap on the retina 186.

[0055] 9A-9C illustrate optical aberration plots of optical system 100 having a reflective optical arrangement 122, such as a cylinder mirror, at the end of LOE 114, according to each of the example scenarios described above with respect to FIGS. 6A-8B. In FIGS. 9A-9C, axis px refers to the pupil coordinate axis and coincides with axis X in FIGS. 6A-8B, axis ey refers to the lateral ray error as a function of pupil entrance radius, ac, bc, and cc refer to aberrations resulting from reflective optical arrangement 122, af refers to the width of the beam with a fully illuminated aperture angle, and bf and cf refer to the subaperture widths of POD 112.

[0056] 9A illustrates aberration plots for the exemplary scenario of FIGS. 6A and 6B, where beam 134 illuminates the entire area of ​​pupil 182. As can be seen in FIG. 9A, the aberrations a c resulting from reflective optical arrangement 122 are relatively large compared to those seen in FIGS. 9B and 9C.

[0057] 9B illustrates an aberration plot for the example scenario of FIGS. 7A and 7B, where beam 136 illuminates a first portion of pupil 182. As can be seen in FIG. 9B, the aberration bc forms a first dashed rectangle on the aberration plot, but is smaller than the aberration ac resulting from full illumination of pupil 182.

[0058] 8A and 8B, where beam 138 illuminates a second portion of pupil 182. As can be seen in FIG. 9C, aberration cc forms a second dashed rectangle on the aberration plot that is smaller than aberration ac resulting from full illumination of pupil 182. Additionally, as can be seen in FIG. 9B and 9C, the types of aberrations formed by beams 136 and 138 are different; for example, beam 138 may have reduced aberration cc due to reflective optical arrangement 122, compared to aberration bc of beam 136.

[0059] 10-15 , exemplary optical architectures and configurations of optical system 100 and POD 112 according to various embodiments are described. POD 112 in each embodiment includes illumination system 300, projection optics 302, and SLM 304. Selective eye-pupil illumination or selective EMB illumination in these embodiments is achieved by illumination system 300, which can be utilized by POD 112 as an image generator for LOE 114 in a 2D magnification system ( FIGS. 2A-2C ) or LOE 214 in a 1D magnification system ( FIG. 3 ). Projection optics 302 is configured to collimate light beams coming from pixels of SLM 304, with each pixel generating a collimated beam, and with the collimated beams from different pixels propagating in different directions. Projection optics 302 is also configured to inject the collimated beams from each of the pixels into LOE 114 or LOE 214. For example, in some embodiments, projection optics 302 can include a single lens with SLM 304 located at the focal plane of the lens. In other embodiments, projection optics 302 may include one or more additional or alternative optical elements, including, for example, lenses, mirrors, wave plates, beam splitter prisms, or other optical elements.

[0060] 10 and 11 show an exemplary configuration of a POD 112 in which selective eye pupil illumination or selective EMB illumination may be achieved according to an embodiment. In this embodiment, an illumination system 300 includes a light source 306, e.g., an array of LEDs or other selectively activatable light sources, located in the focal plane of an optical arrangement 308, such as a collimating lens. Although the optical arrangement 308 is described as a collimating lens, a wide range of lens types and implementations may be utilized, including, but not limited to, spherical, aspherical, or freeform refractive lenses, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above, formed from glass or plastic.

[0061] The array of light sources 306 may include red, green, and blue light sources, or multi-color light sources configured to generate red, green, blue, or other colors. The light sources are configured to generate color images in a color-sequential operation mode of the SLM 304. While the array 306 is illustrated in FIGS. 10 and 11 as having a specific number of light sources, the array 306 may alternatively include any other number of light sources. For example, additional light sources may be included in the array 306 to achieve smoother EMB scanning. In some embodiments, aperture scanning may be performed in the YZ plane as shown in FIG. 10, while in the XZ plane, the full aperture of the POD 112 may be illuminated as shown in FIG. 11. In some embodiments, the illumination system 300 may also include an optional diffuser 310 to increase the divergence of the light beam in the XZ plane, as shown in FIG. 11. In other embodiments, a cylindrical lens may be utilized in each of the light sources instead of the diffuser 310 to reduce the beam divergence in the XY plane.

[0062] The output light beam of each light source in array 306 from illumination system 300 is collimated or near-collimated illumination that is provided to SLM 304 via projection optics 302. The angle of the collimated light beam at SLM 304 depends on which light sources in array 306 are activated, and the divergence of the illumination generated by the light sources depends on the size of the light sources and the scattering angle range of optional diffuser 310. Each light source in array 306 corresponds to a different sub-aperture angle of POD 112, where switching the illumination between sub-aperture angles of POD 112 can be achieved, for example, by switching on and off each light source in array 306. For example, as shown in FIG. 10 , light sources 312 and 314 generate light beams 316 and 318, respectively, which are collimated by optical arrangement 308 and optionally scattered by diffuser 310 before being provided to SLM 304. As seen in FIG. 10, light beams 316 and 318 each illuminate a field of view (FOV) region between FOVA and FOVB of POD 112.

[0063] 12 shows an exemplary embodiment of optical system 100 using the exemplary configuration of POD 112 described above with respect to FIGS. 10 and 11. Eye tracking system 160 is configured to measure the location of pupil 182 of eye 180 relative to LOE 114 and provide this measured location as location information to controller 140. Based on the location information, controller 140 is configured to determine a light source in array 306 that can be turned on or otherwise activated to illuminate an aperture that projects an image onto pupil 182 or a subaperture of pupil 182 at the measured location. In some embodiments, controller 140 is also configured to calculate or determine any distortion to be applied to the image provided to SLM 304 to compensate the image for any aberrations caused by POD 112 and reflective optical arrangement 122 or other components of LOE 114.

[0064] The distortion applied to the image provided to the SLM 304 may depend, for example, on which light sources in the array 306 are activated, the location of the pupil 182, which portion of the EMB is illuminated, or any other criteria. The location of the image on the retina 186 for the same FOV may differ depending on, for example, the subaperture viewed by the pupil 182, as seen, for example, in FIGS. 7A-8B . Because the location of the projected image on the retina differs for each subaperture, the type and amount of distortion applied by the controller 140 to the image at the SLM 304 may depend on which subaperture or corresponding light source of the POD 112 is activated. By applying distortion based on the image for which subaperture or light source is activated, and in some embodiments, according to the location on the retina 186 where the image is projected, the image projected onto the retina 186 from each subaperture or light source may be aligned so that the user sees the same or nearly the same image regardless of which subaperture or light source is activated to provide the image. For example, control of the SLM 304 and the array of light sources 306 may be synchronized by the controller 140 to allow rapid switching between light sources based on the location of the pupil 182 as tracked by the eye tracking system 160. By synchronizing control of the SLM 304 and the array of light sources 306, an image corrected for aberrations may be projected regardless of changes in the location of the pupil 182 or corresponding changes in which light sources are activated by the controller 140.

[0065] 13A-13D, an exemplary process will now be described for determining the image distortion applied to a light beam by SLM 304 based on which light sources are activated in array 306. The exemplary process may be utilized for a single FOV point or for a small localized region of the FOV.

[0066] An exemplary process for operating optical system 100 will now be described with reference to FIG. 13D. The process may be performed at least in part by controller 140, eye tracking system 160, and POD 112, or may be performed at least in part by other portions of optical system 100. The process of FIG. 13D includes steps 500-508. Although the process of FIG. 13D is described herein as having particular steps or a particular order of steps, in other embodiments, the process may alternatively perform steps in any order, include additional steps, include fewer steps, or perform only a portion of the steps described below.

[0067] In step 500, eye tracking system 160 locates the position of pupil 182 within EMB 128, for example, as shown in FIG. 13A. By way of example, eye tracking system 160 may utilize one or more eye tracking cameras or other optical elements to locate the position of pupil 182. Eye tracking system 160 provides location information, e.g., coordinates or other information, corresponding to the determined location to controller 140.

[0068] In step 502, the controller 140 determines which light sources in the array 306 can be activated to project an image onto portions of the pupil 182. For example, the controller 140 may maintain a coordinate map of the EMB 128 indicating which light sources correspond to each portion of the EMB 128. The controller 140 may select the light sources to activate based at least in part on a comparison between the location information and the coordinate map, for example, by determining the location of the pupil 182 relative to the EMB 128 and identifying the corresponding light sources based on the coordinate map. In some embodiments, the controller 140 is configured to identify, for each light source from the light source array 306, which region of the EMB 128 is illuminated for each FOV point or small localized FOV region. Given the location of the pupil 182 and the FOV point to be projected, the controller 140 can identify the light sources to be turned on. For example, as seen in FIG. 13A , such light sources generate beams 142 that illuminate a region spanning the pupil 182. In FIG. 13A, coordinates (x0, z0) correspond to the center of the illuminated area within pupil 182.

[0069] In step 504, controller 140 determines which distortion to apply to the image at SLM 304 based at least in part on the selected light source. In some embodiments, the distortion may also or alternatively be determined based at least in part on location information, for example, when multiple light sources may be used to illuminate the same location but with different collimation angles.

[0070] 13B and 13C show aberration curves similar to those shown in FIGS. 9A-9C for optical system 100 with reflective optical arrangement 122. The aberration curves in FIGS. 13A and 13B show the positions ex and ey of rays corresponding to a given FOV point at the user's retina 186 as a function of the ray's position at the exit pupil, e.g., EMB 128, along the x-axis. Assuming that the nominal image corresponds to a ray passing through the center of the exit pupil, the distances dy and dx in FIGS. 13B and 13C give the local displacement of the image along the y- and x-axes, respectively. Distortions equal to dx and dy can be applied to an image projected through a subaperture defined by the intersection of beam 142 and pupil 182.

[0071] In some embodiments, controller 140 may determine the distortion to apply, for example, using a lookup table with predefined distortion values ​​based on the activated target location and light source. For example, the lookup table may be generated using various techniques, including, for example, by simulating or modeling aberrations and potential distortions, or in any other manner, using an inverse method to adjust the distortion based on the resulting image as projected onto eye 180 or a representation of the eye. In some embodiments, the illuminated beam aperture may be slightly different for each of the RGB light sources. In such cases, controller 140 may also apply distortions to the red, green, and blue images to correct for such small differences in aperture and take these differences into account when correcting for possible color distortions of the projection optics, such as lateral color.

[0072] In step 506, the controller 140 activates the selected light source to output the image.

[0073] In step 508, controller 140 provides SLM 304 with the determined distortion to be applied to the image before providing the image to LOE 114. The image is then projected by LOE 114 onto a portion of pupil 182, and the process returns to step 500 and continues for each frame of the image. In this manner, changes in the location of pupil 182 are taken into account, corresponding light sources are activated, and appropriate distortions are applied to produce an image with as little distortion as possible.

[0074] In some embodiments, controller 140 is configured to sequentially activate each light source in array 306 to perform a full EMB scan in which distortions can be determined and applied to the image at SLM 304 for each light source. As an example, if eye tracking system 160 is not present or active and the location of pupil 182 is not known, such sequential activation can be used to rapidly present images to each portion of EMB 128 to ensure that at least one distortion-corrected image is projected onto the location of pupil 182.

[0075] 14 and 15 show an exemplary configuration of POD 112 in which selective eye pupil illumination or selective EMB illumination may be achieved, according to another embodiment. In this embodiment, illumination system 600 replaces illumination system 300 in POD 112. Illumination system 600 includes an array of light sources 606 similar to the array of light sources 306, a first optical arrangement 608, a first microlens array (MLA) 610, a second microlens array 612, and a second optical arrangement 614.

[0076] The first and second optical arrangements 608 and 614 may include lenses, such as Fresnel lenses or diffractive lenses that may be used to collimate light beams for images. Although the first and second optical arrangements 608 and 614 are described above as including particular types of lenses or optical components, a wide range of other lens types or optical components and implementations may alternatively be utilized, including, but not limited to, spherical, aspherical, or freeform refractive lenses formed from glass or plastic, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above.

[0077] The first MLA 610 and the second MLA 612 each include an array of lenses that can function as a single element. In some embodiments, the lenses of the first and second MLAs 610 and 612 may include refractive lenses. In some embodiments, a baffle arrangement (not shown) may be interposed between the first MLA 610 and the second MLA 612 to reduce crosstalk between the collimating optics. Although the first MLA 610 and the second MLA 612 are described above as including particular types of lenses or optical components, a wide range of other lens types or optical components and implementations may be utilized, including, but not limited to, spherical, aspherical, or freeform refractive lenses, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above, formed from glass or plastic.

[0078] The lenses in the first optical arrangement 608 and the first MLA 610 are together configured to create an image of the light sources of the array 606 at the plane of the second MLA 612. The second MLA 612 and the second optical arrangement 614 are together configured to create an image of each of the lenslet elements of the first MLA 610 at the plane of the SLM 304. In an exemplary configuration, the array of light sources 606 is located in the focal plane of the optical arrangement 608, the second MLA 612 is located in the focal plane of the first MLA 610, and the SLM 304 is located in the focal plane of the optical arrangement 614. Selective illumination of the EMB 128 is achieved by switching the light sources of the array 606 on and off in a coordinated and timed manner so that only a target portion of the EMB 128 and a corresponding portion of the pupil 182 are illuminated. 14 and 15 , image light beams 616 and 618 may be generated, for example, by selectively activating light sources 620 and 622 at different times. Light beams 616 and 618 travel through first optical arrangement 608, through one or more lenses of first MLA 610, through one or more lenses of second MLA 612, and through second optical arrangement 614, and are output from illumination system 600, via projection optics 302, as collimated beams to the FOVA and FOVB of SLM 304. SLM 304 then applies distortions to the images of light beams 616 and 618 and outputs them to LOE 114 for projection onto pupil 182, in a manner similar to that described above for the process of FIG. 13 .

[0079] 16A and 16B, we now describe schematic diagrams of an image projection assembly 710 in the YZ and XZ planes, respectively, according to some embodiments. The image projection assembly 710 may include similar components to those described above for the image projection assembly 110, where such components have similar reference numbers. For example, the image projection assembly 710 includes a POD 712 and an LOE 714. The LOE 714 includes major exterior surfaces 716 and 718, a coupling-out arrangement 720 including, for example, facets 724, and a reflective optical arrangement 722 similar to the components of the LOE 114.

[0080] The image projection assembly 710 further includes a wedge 750 positioned between the POD 712 and the LOE 714. The wedge 750 is configured to reduce the aberrations of the cylinder mirror illustrated in FIGS. 9A-9C. The light beam 726 is coupled from the POD 712 into the LOE 714 through the wedge 750, and a waveguide aperture stop is located behind the wedge 750. In some embodiments, surfaces 752 and 754 of the wedge 750 may optionally have one- or two-dimensional optical power to compensate for optical aberrations of the POD 712 and improve image quality. The LOE 714 further includes a mixer 756, e.g., a semi-reflective flat surface parallel to the major exterior surfaces 716 and 718. In some scenarios, for example, the output beam 726 of the POD 712 may not completely fill the LOE 714. Mixer 756 is utilized to distribute beam 726 across the full aperture of LOE 714. In some embodiments, mixer 756 may be located inside LOE 714 between wedge 750 and coupling-out arrangement 720, for example, as shown in FIG. 16A . In other embodiments, mixer 756 may be located inside LOE 714 between coupling-out arrangement 720 and reflective optical arrangement 722. Reflective optical arrangement 722 may also include a wave plate 723 disposed between coupling-out arrangement 720 and reflective optical arrangement 722, e.g., a quarter-wave plate. In embodiments in which mixer 756 is located between coupling-out arrangement 720 and reflective optical arrangement 722, the light beam passes through mixer 756 twice before being directed from LOE 714 toward the EMB or pupil by coupling-out arrangement 720. In such a case, the mixer 756 may have a shorter length along the z-direction compared to an embodiment in which the mixer 756 is located between the wedge 750 and the coupling-out arrangement 720 .

[0081] The image projection assembly 710 may further include a polarizer 725 disposed between the LOE 714 and the wedge 750. In some embodiments, the coupling-out arrangement 720 includes a surface, e.g., a facet 724, that is partially reflective to one polarization, e.g., polarization(s), but essentially transparent to the orthogonal polarization, e.g., polarization (p). If the input light beam 726 output from the POD 712 is p-polarized, it propagates toward the reflecting optical arrangement 722 without being coupled out by the coupling-out arrangement 720. After reflecting from the reflecting optical arrangement 722 and passing through the wave plate 723, the light becomes s-polarized and is coupled out of the LOE 714 by the coupling-out arrangement 720 as it propagates back from the reflecting optical arrangement 722 toward the POD 712. The polarizer 725 is configured to block the counter-propagating light beam from entering the POD 712.

[0082] In some embodiments, the LOE 714 may also include an optional cover plate 727 disposed on the major exterior surfaces 716 and 718 of the LOE 714. The cover plate 727 results in the thickness of the coupling-out arrangement 720 in a direction perpendicular to the waveguide major surfaces being less than the total thickness of the LOE 714. The LOE 714 may also include an optional polarizer (not shown) disposed parallel to the major exterior surfaces 716 and 718 of the LOE 714 configured to block the passage of p-polarized light. For example, such an optional polarizer (not shown) may be disposed in front of the LOE 714, e.g., to the left of the LOE 714 in FIG. 16A , and is configured to block the light beam coupled out of the LOE 714 to the outside world by the coupling-out arrangement 720. Another optional polarizer (not shown) may be disposed behind the LOE 714, for example, to the right of the LOE 714 in FIG. 16A, and is configured to block the light beam coupled out by the coupling-out arrangement 720 and reflected by the left major external surface 716 toward the user.

[0083] An illumination system 800 of POD 712 is shown schematically in FIG. 16A and in more detail in FIG. 16B . Illumination system 800 includes an array of light sources 802, a polarizing beam splitter disposed between an arrangement of prisms 804 and 806, a quarter-wave plate 808, and a collimating optical arrangement 810, such as a reflective lens or mirror. Light beams emitted by the light sources in array 802 become collimated after reflection from optical arrangement 810 and are projected into an LOE 812, e.g., a waveguide, having two major planar parallel surfaces and a set of semi-reflective facets 814 for extracting light toward prism 816 of POD 712. POD 712 may also include an optional diffuser 818 between LOE 812 and prism 816 of POD 712. Illumination system 800 may also include an optional diffuser (not shown) between prisms 804 and 806 and LOE 812. Light beams coupled out from prisms 804 and 806 into LOE 812 propagate through LOE 812 by total internal reflection and are coupled out of LOE 812 by facet 814 towards optional diffuser 818 of POD 712 and prism 816. Exemplary paths of light beam propagation within illumination system 800 are shown as arrows in FIG.

[0084] 16A, a light beam received from illumination system 800 enters prism 816 and is redirected toward SLM 820. A polarizing beam splitter 822 may be disposed between prism 816 and another prism 824 of POD 712. SLM 820 may be similar to SLM 304 and configured to be controlled by controller 140. In this embodiment, SLM 820 may be implemented as a reflective SLM or a light-emitting SLM, as described above with reference to SLM 304.

[0085] The illumination light beam is modulated by SLM 820, for example, in a manner similar to that described above for SLM 304, and directed through polarizing beam splitter 822 and prism 824 toward reflecting optical arrangement 826 of POD 712. The light beam reflected by reflecting optical arrangement 826 is then directed by prism 824 toward wedge 750.

[0086] FIG. 16C illustrates an illumination system 900 that can be used with the POD 712 and LOE 714 according to another embodiment. In this embodiment, the illumination system 900 replaces the illumination system 800 of FIG. 16B. The illumination system 900 includes an array of light sources 902 and an imaging system including an optical element 904. The optical element 904 may include a refractive lens, a Fresnel lens, a diffractive lens, or a phase lens, such as a Pancharatnam-Berry lens, or any other type of lens in any combination. Light beams emitted by sources in the array 902 are collimated after passing through the optical element 904 and projected into the LOE 906 through a prism 908. The light propagates within the LOE 906 by total internal reflection and is directed into the prism 816 of the POD 712 by a semi-reflective facet 910. The light propagation paths within the illumination system 900 are indicated by arrows in FIG. 16C. Similar to illumination system 800, optional diffusers (not shown) may also be included, for example, between LOE 906 and prism 816, and between optical element 904 and prism 908.

[0087] 17 and 18A-18C illustrate an embodiment of the image projection assembly 110 in which two pixels of the SLM 304 (FIG. 10) reflect a light beam from the illumination system 300 (FIG. 10) into two light beams 1000 and 1002. The beams 1000 and 1002 correspond to two different points within the FOV, FOV A and FOV B, respectively, but also correspond to the same sub-aperture angle of the POD 112 as defined by the active illumination system source. The beams 1000 and 1002 are expanded by the facet 124 of the coupling-out arrangement 120 and projected onto the EMB 128.

[0088] 18A-18C illustrate an example scenario in which beams 1000 and 1002 of different FOVs are projected onto the EMB 128 and the pupil 182 (FIG. 6B) of the eye 180 (FIG. 6B). In FIG. 18A-18C, the beams 1000 and 1002 are illustrated as dashed lines, and the position of the pupil 182 is illustrated as a circle.

[0089] 18A , an exemplary scenario is described in which two possible positions 1004 and 1006 of the pupil are shown. At position 1004, both beams 1000 and 1002 illuminate the pupil, showing the FOVA and FOVB to the user. At position 1006, only beam 1000 illuminates the pupil, showing only the FOVA to the user. The exemplary scenario of FIG. 18A illustrates that different subapertures of the POD 112 can be illuminated such that a particular FOV is visible to the pupil. For example, a different subaperture of the POD 112 can be illuminated such that the FOVB is visible to the pupil at position 1006.

[0090] 18B, another exemplary scenario is described in which the projections of light beams 1008 (FOVA) and 1010 (FOVB) on the EMB 128 are shown to correspond to different sub-apertures of the POD 112 that are illuminated. In this exemplary scenario, the same two possible positions of the pupil are shown, 1004 and 1006. At position 1004, both beams 1008 and 1010 illuminate the pupil, showing both the FOVA and FOVB to the user, but beam 1008 only partially illuminates the pupil 182 at position 1004. At position 1006, only beam 1010 illuminates the pupil, showing the FOVB to the user. Since the FOVB is visible to the pupil at position 1006, different sub-apertures of the POD 112 may be illuminated.

[0091] 18A and 18B, when the pupil is at position 1006, it is not possible to see either the FOVA or the FOVB. In some embodiments, controller 140 (FIG. 1) is configured to account for this problem by sequentially turning on and off one or more light sources in array 306 so that the complete set of FOVs is visible to the user's eye when the pupil is located at a particular position on EMB 128.

[0092] 18C , another exemplary scenario is described in which the projections of light beams 1012 (FOVA) and 1014 (FOVB) on the EMB 128 are shown to correspond to different subapertures of the POD 112 that are illuminated. In this exemplary scenario, the same two possible positions 1004 and 1006 of the pupil are shown. At position 1004, both beams 1012 and 1014 illuminate the pupil, showing both the FOVA and the FOVB to the user. At position 1006, the entire beam 1014 illuminates the pupil, showing the FOVB to the user, but only a portion of beam 1012 illuminates the pupil. Such partial illumination can result in a degradation of the image of the FOVA, for example, due to diffraction at the edges of the pupil. In some embodiments, the controller 140 may be configured to instruct the SLM 304 to project the FOVB but not the FOVA when illuminating this particular subaperture of the POD 112 while the pupil is at the position 1006, in order to inhibit the projection of a degraded image of the FOVA. Alternatively, the FOVA may be projected onto the same position 1006 by successive activation of another subaperture or light source of the POD 112.

[0093] 19A-19C, a lighting system 1100 according to another embodiment will now be described. The lighting system 1100 may, for example, replace the lighting system 300 of the POD 112 as shown in FIG. 10, the lighting system 600 of the POD 112 as shown in FIG. 14, the lighting system 800 of the POD 712 as shown in FIGS. 16A and 16B, the lighting system 900 of the POD 712 as shown in FIG. 16C, or may be used with any other POD. The lighting system 1100 may include components similar to those found in any of the lighting systems 300, 600, 800, and 900.

[0094] 19A, reference numbers corresponding to POD 712 are used in conjunction with the description of illumination system 1100. For example, as mentioned above, POD 712 includes prisms 816 and 824, SLM 820, polarizing beam splitter 822, reflective optical arrangement 826, and optional diffuser 828, all of which may function as described above with reference to FIGS. 16A and 16B after receiving the light beam from illumination system 1100.

[0095] As shown in FIGS. 19A-19C, illumination system 1100 includes an MLA 1102 and a matrix of light sources 1104, such as a micro LED display or other arrangement of light sources. Light beams output from illumination system 1100 are provided to prism 816. As shown in FIG. 19B, each microlens in MLA 1102 collimates the light from its respective light source. The direction or angle of the collimated illumination can change from microlens to microlens depending on which light source in matrix 1104 is activated. For example, when light source 1106 is activated, light beam 1108 is collimated by microlens 1110 and output at a first direction or angle, while when light source 1112 is activated, light beam 1114 is collimated by the same microlens 1110 but output at a second direction or angle different from the first direction or angle. The configuration of MLA 1102 and the matrix of light sources 1104 allows controller 140 to present different illumination angles for different regions of SLM 820. In another embodiment, an array of micromirrors (not shown) can be used in place of MLA 1102.

[0096] 20-29, various embodiments of the optical system 100 described above may be configured for the projection of time-multiplexed bright-field images are disclosed.

[0097] 20 and 21, the function of the eye 180 when the lens 184 is focused at infinity or a finite distance will now be described, respectively.

[0098] As seen in FIG. 20 , the lens 184 of the eye 180 is focused at infinity, and the pupil 182 is illuminated by two beams of light, beam 1200 and beam 1202, which create images P1 and P2, respectively, at the retina 186. FIG. 21 shows that the lens 184 of the eye 180 focuses at a finite distance rather than at infinity, and the lens 184 has a shorter focal length. Because the focal length of the lens 184 in FIG. 21 is short, the images P1 and P2 converge into a single image. Because the images P1 and P2 are projected onto the pupil 182 through a small sub-aperture, changing the focal length of the lens 184 of the eye 180 reduces the blurring of the images P1 and P2.

[0099] 22 and 23 illustrate embodiments in which image projection assemblies 110 and 210 project beams 1300 and 1302 onto EMB 128 that correspond to different points within the FOV and different sub-aperture angles of PODs 112 and 212, illuminating different regions of pupil 182 as shown in FIGS. 20 and 21. While shown as having specific components, image projection assemblies 110 and 210 may each include any of the components of the LOE, POD, illumination system, or other parts of optical system 100 found in various embodiments described herein.

[0100] 24, in some embodiments, controller 140 is configured to divide the projection of a single image into multiple frames, e.g., frame 1, frame 2...frame n, and to sequentially project each frame onto user's eye 180. Controller 140 in this embodiment generates successive frames 1-n of images 13001, 13002, ...13003, as shown in FIG. n Images 13001, 13002, ... 1300 in each successive frame 1, 2, ... n are shifted slightly relative to the previous frame. nThe controller 140 is configured to project the frames one at a time in rapid succession, and in some embodiments may project one or more of the shifted frames using different sub-apertures of the POD 112 by activating different light sources in the array 306. For example, as seen in FIG. 24 , the image 1300 of frame n n is displaced by a distance e relative to image 13001 of frame 1. In this way, controller 140 can simulate a one-dimensional bright field using time-multiplexed projections of frames of images.

[0101] In some cases, the time-multiplexed brightfield projection described above is created in only one dimension, for example, along axis X in Figure 22. Along axis Z, the image beam is wide and illuminates the entire aperture of the eye in the Z direction due to the dilation of the pupil by facets 124. As a result, the image in the Z direction is sharp only when focused at infinity, and as the eye's accommodation changes to a finite focal length, the image in the Z direction becomes blurry.

[0102] 25A-25C, an embodiment of image projection assembly 210 is illustrated in which blurring in the Z direction of time-multiplexed bright-field projection as described above may be overcome. For example, the aperture of beam 1400 in the Z direction may be limited by dynamically increasing or decreasing at least one of the reflectivity and intensity of some of facets 224, e.g., by dynamically making them more reflective or more transparent. By way of example, one or both of the reflectivity and intensity of facets 224 of LOE 214 are configured to be dynamically adjusted in the embodiment shown in FIGS. 25A-25C. Although described with reference to image projection assembly 210, in other embodiments, image projection assembly 110 may alternatively be utilized.

[0103] Although the size of beam 1400 in the X dimension is limited due to components of an illumination system, such as any of illumination systems 300, 600, 800, and 900 described herein, in the Z direction, beam 1400 illuminates the complete EMB 128 by reflections from facets 1402, 1404, 1406, and 1408, as shown in FIG. 25B. In the example shown in FIG. 25B, reflections from facets 1404 and 1406 illuminate pupil 182 at a particular location within EMB 128, while reflections from other facets, such as facets 1402 and 1408, do not illuminate pupil 182 at that particular location within EMB 128. If facet 1406 were transparent (non-reflective), as shown in FIG. 25C, only a portion of pupil 182 in the Z direction would be illuminated. By dynamically adjusting one or both of the reflectivity and intensity of each facet, the image of a point on the retina 186 can be sharpened in the Z direction for any position of the pupil 182 and any accommodation of the lens 184 of the eye 180.

[0104] Referring now to FIG. 26 , a dynamic facet structure 1500 for controlling one or both of the reflectivity and intensity of a facet 1502 is illustrated according to some embodiments. The facet 1502 may be highly transmissive for p-polarized light and partially reflective for s-polarized light. The dynamic facet structure 1500 includes a facet 1502 disposed between a first liquid crystal layer 1504 and a second liquid crystal layer 1506. In some embodiments, the liquid crystal layers 1504 and 1506 are parallel and may be parallel to the facet 1502. The state of the liquid crystals in each of the liquid crystal layers 1504 and 1506 is controlled by a voltage applied to the layer, for example, by the controller 140 ( FIG. 1 ). In the “on” state, the liquid crystals in each of the liquid crystal layers 1504 and 1506 act as a half-wave plate, rotating the polarization of the beam 1400 by 90 degrees. In the “off” state, the polarization state of the beam 1400 remains unchanged after passing through the layer of liquid crystals.

[0105] In this embodiment, beam 1400 propagating within LOE 214 is s-polarized, and facet 1502 is highly transmissive for p-polarized light and partially reflective for s-polarized light, as mentioned above. In other embodiments, beam 1400 propagating within LOE 214 may be p-polarized, and facet 1502 may be highly transmissive for s-polarized light and partially reflective for p-polarized light.

[0106] When liquid crystal layers 1504 and 1506 are in the "off" state, the polarization of beam 1400 when it encounters facet 1502 is s-polarized, and facet 1502 is partially reflective to beam 1400, as shown, for example, by facet 1404 in FIG. 25C. When liquid crystal layers 1504 and 1506 are in the "on" state, the beam polarization is p-polarized at facet 1502, and facet 1502 is transparent to beam 1400, as shown, for example, by facet 1406 in FIG. 25C.

[0107] Note that because of the presence of liquid crystal layers disposed on either side of facet 1502, e.g., liquid crystal layer 1504 disposed on one side of facet 1502 and liquid crystal layer 1506 disposed on the opposite side of facet 1502, when in the "on" state, the polarization of beam 1400 changes from s-polarized before encountering dynamic facet structure 1500 to p-polarized after passing through liquid crystal layer 1504. Beam 1400 encounters facet 1502 while possessing p-polarized light, and passes through due to the high transmittance of facet 1502 for p-polarized light. Beam 1400 then encounters liquid crystal layer 1506 on the opposite side of facet 1502, where it changes back from p-polarized to s-polarized. Beam 1400 then exits dynamic facet structure 1500 still possessing s-polarized light. In this way, each dynamic facet structure 1500 can be independently controlled by controller 140 (FIG. 1) to reflect or transmit beam 1400 without affecting the polarization of beam 1400 relative to any of the other facets.

[0108] In another embodiment, one or both of the reflectivity and intensity of the facets 224 may alternatively be dynamically controlled using electrically switchable Bragg reflectors. For example, in some embodiments, each of the facets 224 of the coupling-out arrangement 220 may include an electrically switchable Bragg reflector whose reflectivity, intensity, or both may be electrically controlled by the controller 140.

[0109] 27A and 27B, an image projection assembly 1610 according to some embodiments will now be described. The image projection assembly includes a POD 1612, which may include similar components and functionality as the POD 112, 212, or any other POD disclosed herein. The image projection assembly includes an LOE 1614, which may include at least some similar components and functionality as the LOE 114, 214, or any other LOE disclosed herein, except as described in more detail below. The POD 1612 is configured to output a light beam 1626 to the LOE 1614, which is directed by a coupling-out arrangement 1620 to the EMB 128 and pupil 182.

[0110] 27A and 27B, coupling-out arrangement 1620 includes a switchable Bragg reflector (SBR) 1624 disposed on one of major exterior surfaces 1616 or 1618 of LOE 1614, for example, as shown in FIGS. 27A and 27B. For example, in some embodiments, SBR 1624 may be integrated within major exterior surface 1616. When in the “on” state, SBR 1624 is configured to reflect light beam 1626 toward the location of pupil 182 within EMB 128. When in the “off” state, SBR 1624 provides total internal reflectivity such that light beam 1626 propagates within the waveguide. In some embodiments, SBR 1624 is divided into multiple selectively activatable regions, including region 1628, which may be independently controlled by controller 140, for example. By switching a selected region of the SBR 1624 "on," a target portion of the pupil 182 can be illuminated, for example, as shown in selected region 1628 in Figure 27B. In another embodiment, a transmittance switchable grating can alternatively be used in place of the SBR 1624.

[0111] 28, an embodiment of optical system 100 with 2D bright field projection will now be described in which controller 140 is further configured to control dynamic facet structure 1500 or SBR 1624. For example, eye tracking system 160 determines the location of pupil 182 relative to LOE 114 and provides this location to controller 140 as location information. Controller 140 uses the location information to determine which light source or light sources in array 306 can be activated to project an image onto pupil 182 at the determined location, and to determine the distortion to apply to the image by SLM 304 to compensate for image shift on retina 186 due to aberrations in POD 112 and reflective optical arrangement 122 of the 2D magnification system of FIGS. 2A-2C or POD 212 of the 1D magnification system of FIG. 3, as described above.

[0112] In this embodiment, controller 140 is also configured to determine which of the facets 124 in LOE 114 are selected to be set to an "on" state (semi-reflective) and an "off" state (transmissive), e.g., to improve image clarity in the Z direction due to changes in accommodation of lens 184 of eye 180, as described above. For example, to project an image of an object located at infinity, only a single image needs to be projected. However, to project an image of an object located at a final distance, multiple images need to be projected, e.g., as described above with reference to Figures 20-24.

[0113] In one example scenario, an image of an object located at a finite distance from the user is projected. For a given position of the pupil 182, multiple images, e.g., two images, three images, ... 100 or more images, are projected through different sub-apertures of the pupil 182. For each sub-aperture projection, the image is distorted twice. The first distortion is configured to compensate for distortions caused by aberrations, as shown in Figures 13A-13C. The second distortion is configured to shift the image to create a bright field, as shown in Figures 20 and 21, for example.

[0114] 29, an exemplary process for operating optical system 100 of FIG. 28, including control of selectively activatable facets 124, will now be described. This process may be performed at least in part by controller 140, eye tracking system 160, POD 112, and LOE 114, or may be performed at least in part by other portions of optical system 100.

[0115] The process of Figure 29 includes steps 1700 through 1712. Although the process of Figure 29 is described herein as having particular steps or a particular order of steps, in other embodiments, the process may alternatively perform steps in any order, may include additional steps, may include fewer steps, or may perform only a portion of the steps described below.

[0116] In step 1700, the eye tracking system 160 determines the location of the pupil 182, for example, using one or more eye tracking cameras or other optical elements, and provides location information, for example, coordinates or other information, corresponding to the determined location to the controller 140.

[0117] In step 1702, controller 140 determines light sources in array 306 that may be activated to project an image onto portions of pupil 182. For example, controller 140 may maintain a coordinate map of EMB 128 that indicates which light sources correspond to each portion of EMB 128. Controller 140 may select the light sources to activate based at least in part on a comparison between the location information and the coordinate map, for example, by determining the location of pupil 182 relative to EMB 128 and identifying the corresponding light sources based on the coordinate map.

[0118] In step 1704, the controller 140 determines which of the facets 124 should be set to an "on" state and which of the facets 124 should be set to an "off" state, for example, as described above.

[0119] In step 1706, controller 140 determines which distortion to apply to the image at SLM 304 based at least in part on the identified light sources that are activated. In some embodiments, the distortion may also, or alternatively, be determined based at least in part on location information, for example, when multiple light sources may be utilized to illuminate the same location but with different collimation angles. In some embodiments, controller 140 may determine the distortion to apply in a manner similar to that described above for step 504 of FIG. 13 or in any other manner.

[0120] In step 1708, the controller 140 applies the appropriate control signals to the facets 124 to set them to the determined "on" or "off" state.

[0121] In step 1710, the controller 140 activates the identified light sources to output the image.

[0122] In step 1712, controller 140 provides SLM 304 with the determined distortion to be applied to the image before providing the image to LOE 114. The image is then projected onto a portion of pupil 182 with facet 124 of LOE 114 set to the "on" state, and the process returns to step 1700 and continues for each frame of the image. In this way, changes in the location of pupil 182 are taken into account, the effect of changes in accommodation of eye 180 on image sharpness is mitigated, the corresponding light source is activated, and the appropriate distortion is applied to produce an image with as little distortion as possible.

[0123] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0124] In the following claims, where applicable, corresponding structure, material, acts, and equivalents of all means-plus-function or step-plus-function elements are intended to include any structure, material, or act for performing that function in combination with other specifically claimed elements. The disclosed embodiments of the invention have been presented for purposes of illustration and description, but are not intended to be exhaustive or to limit the invention to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The present embodiments were chosen and described in order to best explain the principles and practical application of the invention and to enable others skilled in the art to understand the invention in various embodiments with various modifications as suited to the particular uses contemplated.

Claims

1. 1. An apparatus comprising at least one processor, the at least one processor comprising: selecting a light source from a plurality of light sources based at least in part on a location of the eye's pupil relative to an eye motion box, the selected light source being configured to illuminate with a light beam a portion of the eye motion box corresponding to the location of the pupil; determining individual states of the plurality of selectively activatable facets based at least in part on a projection distance corresponding to the image; causing the plurality of selectively activatable facets to be set to the respective states; and The apparatus is configured to perform the step of activating the selected light source to illuminate the portion of the eye motion box.

2. The apparatus of claim 1 , wherein the selected light source is configured to illuminate with the light beam the portion of the eye motion box corresponding to only a portion of the pupil.

3. the at least one processor: determining a distortion to be applied to the light beam based at least in part on the selected light source; and The apparatus of claim 1 , configured to cause a modification of the light beam based at least in part on the determined distortion.

4. 4. The apparatus of claim 3, wherein determining the distortion to be applied to the light beam based at least in part on the selected light source comprises determining a correction to the light beam for aberrations induced by a collimator.

5. 4. The apparatus of claim 3, wherein causing the modification of the light beam based at least in part on the determined distortion comprises causing a spatial light modulator to modify the light beam based at least in part on the determined distortion.

6. the light beam illuminates the portion of the eye motion box based at least in part on a plurality of elements in a coupling-out arrangement, at least one of a reflectivity and an intensity of each of the elements being selectively adjustable between at least two states; the at least one processor: determining a target state of a given element of the plurality of elements based at least in part on the selected light source; and The apparatus of claim 1 , further configured to cause the given element to be set to the target state.

7. the light source is a first light source, 2. The apparatus of claim 1, wherein the at least one processor is configured to select a second light source from the plurality of light sources, the second light source configured to illuminate the portion of the eye motion box.

8. 1. A method comprising: selecting a light source from a plurality of light sources based at least in part on a location of the eye's pupil relative to an eye motion box, the selected light source being configured to illuminate with a light beam a portion of the eye motion box corresponding to the location of the pupil; determining individual states of the plurality of selectively activatable facets based at least in part on a projection distance corresponding to the image; causing the plurality of selectively activatable facets to be set to the respective states; and activating the selected light source to illuminate the portion of the eye motion box.

9. The method of claim 8 , wherein the selected light source is configured to illuminate with the light beam the portion of the eye motion box corresponding to only a portion of the pupil.

10. determining a distortion to be applied to the light beam based at least in part on the selected light source; and The method of claim 8 , further comprising causing a modification of the light beam based at least in part on the determined distortion.

11. 11. The method of claim 10, wherein determining the distortion to be applied to the light beam based at least in part on the selected light source comprises determining a correction to the light beam for aberrations induced by a collimator.

12. 11. The method of claim 10, wherein causing the modification of the light beam based at least in part on the determined distortion comprises causing a spatial light modulator to modify the light beam based at least in part on the determined distortion.

13. the light beam illuminates the portion of the eye motion box based at least in part on a plurality of elements in a coupling-out arrangement, at least one of a reflectivity and an intensity of each of the elements being selectively adjustable between at least two states; The method comprises: determining a target state of a given element of the plurality of elements based at least in part on the selected light source; and The method of claim 8 further comprising causing the given element to be set to the target state.

14. the light source is a first light source, 9. The method of claim 8, wherein the method further comprises selecting a second light source from the plurality of light sources, the second light source configured to illuminate the portion of the eye motion box.

15. 1. An optical system comprising: Multiple light sources; a light-guiding optical element comprising a coupling-out arrangement configured to direct light beams received from the plurality of light sources towards an eye-motion box of the optical system; a controller, selecting a light source from the plurality of light sources based at least in part on a location of a pupil of an eye relative to the eye motion box, the selected light source being configured to emit a light beam that, when directed by the coupling-out arrangement, illuminates a portion of the eye motion box corresponding to the location of the pupil; determining individual states of the plurality of selectively activatable facets based at least in part on a projection distance corresponding to the image; causing the plurality of selectively activatable facets to be set to the respective states; and a controller configured to activate the selected light source to illuminate the portion of the eye motion box.

16. 16. The optical system of claim 15, further comprising an eye motion tracking system configured to determine the location of the pupil, wherein the controller is configured to determine the portion of the eye motion box that corresponds to the location of the pupil determined by the eye motion tracking system.

17. a spatial light modulator disposed between the plurality of light sources and the light-guiding optical element; the controller is configured to determine a distortion to be applied to the light beam based at least in part on the selected light source; The optical system of claim 15 , wherein the spatial light modulator is configured to modify the light beam based at least in part on the determined distortion.

18. The optical arrangement according to claim 1, further comprising: A first lens; A second lens; a first microlens array disposed between the first lens and the second lens; a second microlens array disposed between the first microlens array and the second lens; 18. The optical system of claim 17, wherein the first lens, the second lens, the first microlens array, and the second microlens array are configured to collectively direct the image from the selected light source toward the spatial light modulator.

19. 20. The optical system of claim 18, wherein the plurality of light sources are located at a focal plane of the first lens, the second microlens array is located at a focal plane of the first microlens array, and the spatial light modulator is located at the focal plane of the second lens.

20. 16. The optical system of claim 15, wherein the coupling-out arrangement comprises a plurality of elements, and the controller is configured to selectively adjust at least one of the reflectivity and intensity of each of the elements between at least two states.

Images