Optical system including selective illumination

Abstract

In an implementation, an apparatus including at least one processor is disclosed. The at least one processor is configured to select a light source from a plurality of light sources based at least in part on a location of a pupil of an eye relative to an eyebox. The selected light source is configured to illuminate a portion of the eyebox corresponding to the location of the pupil with a light beam. The at least one processor is further configured to activate the selected light source to illuminate the portion of the eyebox.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 130,957, filed December 28, 2020, entitled “DISPLAYS EMPLOYING SELECTIVE EYE PUPIL ILLUMINATION WITH OPTIONAL LIGHT FIELD PROJECTION,” and U.S. Provisional Application No. 63 / 121,937, filed December 6, 2020, entitled “DISPLAYS EMPLOYING SELECTIVE EYE MOTION BOX ILLUMINATION,” the entire contents of each of which are incorporated herein by reference.

[0003] BACKGROUND

[0004] The present disclosure relates to optical systems. More specifically, the present disclosure relates to optical systems with selective illumination that can be used in near-eye display systems in some embodiments.

[0005] Optical systems, such as near-eye display systems, often illuminate a user’s eye in a manner that can cause potential aberrations that result in reduced image quality. For example, an optical system can illuminate an entire pupil with a beam of light of an image. Due to aberrations in the beam (e.g., coma, astigmatism, or any other aberration), as portions of the image pass through the pupil and reach the retina, the portions can become blurred and possibly distorted. SUMMARY

[0006] In an embodiment, an apparatus comprising at least one processor is disclosed. The at least one processor is configured to select a light source from a plurality of light sources based at least in part on a location of a pupil of an eye relative to an eyebox. The selected light source is configured to illuminate a portion of the eyebox corresponding to the location of the pupil with a beam of light. The at least one processor is further configured to activate the selected light source to illuminate the portion of the eyebox.

[0007] In some embodiments, the selected light source is configured to illuminate a portion of the eyebox corresponding to only a portion of the pupil with the beam of light.

[0008] In an embodiment, the at least one processor is configured to determine a distortion to be applied to the beam of light based at least in part on the selected light source, and is configured to cause a modification to the beam of light based at least in part on the determined distortion.

[0009] In another embodiment, determining a distortion to be applied to the beam of light based at least in part on the selected light source comprises determining a correction to the beam of light for an aberration caused by a collimator.

[0010] In some embodiments, causing the modification of the light beam based at least in part on the determined deformation comprises causing a spatial light modulator to modify the light beam based at least in part on the determined deformation.

[0011] In embodiments, the light beam illuminates the portion of the eyebox based at least in part on a plurality of elements of the out-coupling device. At least one of a reflectivity and an intensity of each of the elements can be selectively adjusted between at least two states. The at least one processor is further configured to determine a target state of a given element of the plurality of elements based at least in part on the selected light source, and cause the given element to be set to the target state.

[0012] In another embodiment, the light source is a first light source, and the at least one processor is configured to select a second light source of a plurality of light sources. The second light source is configured to illuminate a portion of the eyebox.

[0013] In some embodiments, a method is disclosed, comprising selecting a light source from a plurality of light sources based at least in part on a position of a pupil of an eye relative to an eyebox. The selected light source is configured to illuminate a portion of the eyebox corresponding to the position of the pupil with a light beam. The method further comprises activating the selected light source to illuminate the portion of the eyebox.

[0014] In some embodiments, the selected light source is configured to illuminate a portion of the eyebox corresponding to only a portion of the pupil with the light beam.

[0015] In embodiments, wherein the method further comprises determining a deformation to be applied to the light beam based at least in part on the selected light source, and causing the modification of the light beam based at least in part on the determined deformation.

[0016] In another embodiment, determining the deformation to be applied to the light beam based at least in part on the selected light source comprises determining a correction to the light beam for aberrations caused by a collimator.

[0017] In some embodiments, causing the modification of the light beam based at least in part on the determined deformation comprises causing a spatial light modulator to modify the light beam based at least in part on the determined deformation.

[0018] In embodiments, the light beam illuminates the portion of the eyebox based at least in part on a plurality of elements of the out-coupling device, wherein at least one of a reflectivity and an intensity of each of the elements can be selectively adjusted between at least two states. The method further comprises determining a target state of a given element of the plurality of elements based at least in part on the selected light source, and causing the given element to be set to the target state.

[0019] In another implementation, the light source is a first light source, and the method further includes selecting a second light source of the plurality of light sources. The second light source is configured to illuminate a portion of the eyebox.

[0020] In an implementation, an optical system is disclosed. The optical system includes a plurality of light sources and a light guide optical element including a coupling-out device configured to direct light beams received from the plurality of light sources toward an eyebox of the optical system. The optical system further includes a controller configured to select a light source from the plurality of light sources based at least in part on a position of a pupil of an eye relative to the eyebox. The selected light source is configured to emit a light beam that, when directed by the coupling-out device, illuminates a portion of the eyebox corresponding to the position of the pupil. The controller is further configured to activate the selected light source to illuminate the portion of the eyebox.

[0021] In some implementations, the optical system further includes an eye tracking system configured to determine the position of the pupil. The controller is configured to determine the portion of the eyebox corresponding to the position of the pupil determined by the eye tracking system.

[0022] In an implementation, the optical system further includes a spatial light modulator disposed between the plurality of light sources and the light guide optical element. The controller is configured to determine a deformation to be applied to the light beam based at least in part on the selected light source, and the spatial light modulator is configured to modify the light beam based at least in part on the determined deformation.

[0023] In another implementation, the optical system further includes an optical device configured to direct the light beam from the selected light source toward the spatial light modulator. The optical device includes a first lens, a second lens, a first microlens array disposed between the first lens and the second lens, and a second microlens array disposed between the first microlens array and the second lens.

[0024] In some implementations, the plurality of light sources are located in a focal plane of the first lens, the second microlens array is located in a focal plane of the first microlens array, and the spatial light modulator is located in a focal plane of the second lens.

[0025] In an implementation, the coupling-out device includes a plurality of elements. The controller is configured to selectively adjust at least one of reflectivity and intensity of each of the elements between at least two states.

[0026] The foregoing overview is illustrative only and is not intended to be limiting in any way. Additional aspects, implementations, and features will become apparent from review of the disclosure, which is provided in conjunction with the following detailed description, in conjunction with the appended drawings. In the drawings, like reference numerals indicate like elements or functionally similar elements. BRIEF DESCRIPTION OF DRAWINGS

[0027] FIG. 1 is a schematic diagram of an example optical system according to an embodiment.

[0028] FIGS. 2A-2C is a schematic diagram of an example image projection assembly of the optical system of FIG. 1 for 2D pupil expansion according to an embodiment.

[0029] FIG. 3 is a schematic diagram of an example image projection assembly of the optical system of FIG. 1 for ID pupil expansion according to an embodiment.

[0030] FIG. 4 is a diagram showing a light beam projected onto a surface according to an embodiment.

[0031] FIG. 5 is a diagram showing a light beam projected onto a surface by a sub-aperture according to an embodiment. FIG. 4

[0032] FIG. 6A is a schematic diagram showing a light beam projected by a full-aperture of a projection optics device (POD) of an example image projection assembly of FIGS. 2A-2C according to an embodiment.

[0033] FIG. 6B is a diagram showing a light beam projected onto an eye according to an embodiment. FIG. 6A

[0034] FIG. 7A is a schematic diagram showing a light beam projected by a first sub-aperture of a POD of an example image projection assembly of FIGS. 2A-2C according to an embodiment.

[0035] FIG. 7B is a diagram showing a light beam projected onto an eye according to an embodiment. FIG. 7A

[0036] FIG. 8A is a schematic diagram showing a light beam projected by a second sub-aperture of a POD of an example image projection assembly of FIGS. 2A-2C according to an embodiment.

[0037] FIG. 8B is a diagram showing a light beam projected onto an eye according to an embodiment. FIG. 8A

[0038] FIGS. 9A-9C shows an example aberration diagram of a light beam of FIGS. 6A-8B projected onto an eye according to an embodiment.​​​​

[0039] FIG. 10 and FIG. 11 is a schematic diagram illustrating an example POD of an optical system according to an embodiment. FIG. 1

[0040] FIG. 12 is a schematic diagram illustrating a spatial light modulator (SLM) of a POD according to an embodiment. FIG. 10 and FIG. 11 is a schematic diagram illustrating an optical system according to an embodiment. FIG. 1

[0041] FIG. 13A is a diagram illustrating an example process for partial eye pupil illumination and selective eye motion box (EMB) illumination according to an embodiment.

[0042] FIG. 13B and FIG. 13C is a diagram illustrating an example aberration map of an example process according to an embodiment. FIG. 13A

[0043] FIG. 13D is a flowchart of an example process according to an embodiment. FIG. 13A

[0044] FIG. 14 and FIG. 15 is a schematic diagram illustrating an example POD of an optical system with a micro-lens array (MLA) according to an embodiment. FIG. 1

[0045] FIG. 16A and FIG. 16B is a schematic diagram illustrating an example image projection assembly of an optical system according to an embodiment. FIG. 1

[0046] FIG. 16C is a schematic diagram illustrating an example image projection assembly of an optical system according to an embodiment. FIG. 1

[0047] FIG. 17 is a schematic diagram illustrating a light beam projected by a POD of an optical system according to an embodiment into a light-guide optical element (LOE) of the optical system and guided to an EMB. FIG. 10 FIG. 1

[0048] FIGS. 18A-18C is a schematic diagram illustrating a light beam projected by a POD of an optical system according to an embodiment into a light-guide optical element (LOE) of the optical system and guided to an EMB. FIG. 10 ​​​​​​​​​a plot of the path of the light beams of different apertures and different fields of view (FOV) of the POD relative to the position of the pupil of the eye.

[0049] FIGS. 19A-19C is a schematic diagram illustrating an optical system with an MLA of FIG. 1 is a schematic diagram of a POD and an illumination element of the optical system with an MLA of

[0050] FIG. 20 and FIG. 21 is a plot illustrating the effect of accommodation changes in the eye on the light beams projected onto the pupil, according to an embodiment.

[0051] FIG. 22 is a schematic diagram illustrating an optical system with a dynamic facet structure of FIGS. 2A-2C is a schematic diagram of an image projection assembly of the optical system with a dynamic facet structure of

[0052] FIG. 23 is a schematic diagram illustrating an optical system with a dynamic facet structure of FIG. 3 is a schematic diagram of an image projection assembly of the optical system with a dynamic facet structure of

[0053] FIG. 24 is a plot illustrating a time-multiplexed light field image, according to an embodiment.

[0054] FIGS. 25A-25C is a schematic diagram illustrating an optical system with a switchable Bragg reflector (SBR) of FIG. 1 is a schematic diagram of an image projection assembly of the optical system with a switchable Bragg reflector (SBR) of

[0055] FIG. 26 is a schematic diagram illustrating the dynamic facet structure of FIG. 25, according to an embodiment.

[0056] FIG. 27A and FIG. 27B is a schematic diagram of an image projection assembly of the optical system with a switchable Bragg reflector (SBR) of FIG. 1 is a schematic diagram of an image projection assembly of the optical system with a switchable Bragg reflector (SBR) of

[0057] FIG. 28 is a schematic diagram illustrating an optical system of FIG. 12 is a schematic diagram of the optical system of

[0058] FIG. 29 is a flowchart of an example process for partial eye pupil illumination and selective eye movement box (EMB) illumination with time-multiplexed light field imaging, according to an embodiment. DETAILED DESCRIPTION

[0059] In optical systems such as near-eye display systems, a light beam is output from the display system to a target surface (e.g., a user’s eye) in close proximity to the display system. When projecting an image, such optical systems typically illuminate the entire eye or the entire pupil of the eye. In some cases, such carpet bombing of the eye or pupil can combine with aberrations of the optical projection system, degrading the quality of the image received by the user. For example, as the light beam passes through the lens of the eye and focuses onto the retina, some portions of the image can become blurry, distorted, or have other aberrations as seen by the user.

[0060] In some implementations, the disclosed optical systems are configured to reduce or suppress such aberrations by selectively illuminating only the portion of the pupil needed for the user to see a good quality image. Such selective illumination is also referred to herein as partial eye pupil illumination. For example, compared to full eye illumination, partial eye pupil illumination can be beneficial to achieve improved image quality and can utilize less complex optical systems. In some implementations, partial eye pupil illumination can be combined with displacement of the projected image to create a time-multiplexed light field image, which can provide a solution to the vergence-accommodation conflict (VAC) problem. The VAC occurs when the brain is presented with a mismatch between the distance (sometimes referred to as vergence) of a virtual three-dimensional (3D) object and the focus distance (sometimes referred to as accommodative convergence) needed for the eyes to focus on the virtual 3D object.

[0061] In some implementations, the disclosed optical systems are also or alternatively configured to illuminate only a portion of the eyebox (EMB) at a time (e.g., the portion of the EMB where the pupil of the eye is currently located), also referred to herein as selective EMB illumination. Compared to full EMB illumination, selective EMB illumination can provide increased power efficiency in the optical system, as the image illumination is distributed over a smaller area by the partial EMB illumination.

[0062] Partial eye pupil illumination, time-multiplexed light field imaging, and selective EMB illumination can be used individually or together, and provide the above-mentioned and other benefits to optical systems that can be configured as near-eye display systems.

[0063] Reference is now made to FIG. 1 and FIGS. 2A-2CAn example optical system 100 is described. The optical system 100 includes an image projection assembly 110, a controller 140, and an eye tracking system 160. The eye tracking system 160 can be optional and is configured to track a position of a pupil of an eye 180 of a user and provide corresponding position information to the controller 140. The image projection assembly 110 includes a projection optics (POD) 112 and a light-guide optical element (LOE) 114 and is configured to project an image onto the eye 180 of the user with two-dimensional (2D) pupil expansion.

[0064] The POD 112 includes an image generator, a spatial light modulator (SLM) 304 FIG. 10 ) or other components typically included in an image projection assembly. Some or all of these components can be arranged on a surface of one or more polarizing beamsplitter (PBS) cubes or other prismatic devices. The image generator includes an illumination source that provides an illumination such as a beam or a laser beam that corresponds to an image to be projected onto the eye 180 of the user. An example illumination source can include a light emitting diode (LED), a micro-LED, or other illumination source.

[0065] The SLM 304 can be implemented as a light-emitting SLM that includes components such as organic light emitting diode (OLED) display elements, a backlit liquid crystal display (LCD) panel, a micro-LED display, a digital light processing (DLP) chip, or another light-emitting component; or the SLM 304 can be implemented as a reflective SLM such as a liquid crystal on silicon (LCOS) chip. A beamsplitter cube block can be inserted between collimating optics and the SLM 304, enabling delivery of the illumination to a surface of the SLM 304.

[0066] The SLM 304 is configured to modulate a projected intensity of each pixel of the illumination to generate the image. In some implementations, the SLM 304 can provide a light beam from each pixel of the display that diverges in a plane of the LOE 114, such as a plane of the major outer surfaces 116 and 118 described below. After reflection from the reflective optical device 122 of the LOE 114, the light beam can be collimated in the plane of the LOE 114. In some implementations, the light beam can be collimated in the plane of the LOE 114 but can not be collimated in a plane orthogonal to the LOE 114.

[0067] Alternatively, the POD 112 can include a scanning device, such as a fast scanning mirror, that scans the illumination from the light source across the image plane of the POD 112 while the intensity of the illumination is varied in synchronization with the pixel-by-pixel motion to project the desired intensity for each pixel.

[0068] The POD 112 also includes an in-coupling device, such as an in-coupling reflector, an angled coupling prism, or any other in-coupling device, for injecting the illumination of the image into the LOE 114. In some implementations, the coupling between the POD 112 and the LOE 114 can include a direct coupling, e.g., the POD 112 can be in contact with a portion of the LOE 114, or can include a coupling via an additional aperture expansion device for expanding the size of the aperture at which the image is injected into the plane of the LOE 114.

[0069] The POD 112 also includes an aperture or other component that can be used to limit the size of the illumination. For example, as shown in FIG. 1A, the POD 112 can be configured to output a light beam 126 using a first aperture size such that the light beam 126, once collimated by the reflective optics 122 of the LOE 114, has a width D, and can be configured to output a second light beam 130 using a second aperture size that is smaller than the first aperture size such that the light beam 130, once collimated by the reflective optics 122 of the LOE 114, has a width d that is smaller than the width D. FIG. 2A

[0070] The LOE 114 includes a waveguide that includes edges that are not optically active and first and second parallel major external surfaces 116 and 118, e.g., as shown in FIG. 1A. The LOE 114 also includes an out-coupling device 120 and reflective optics 122, such as a lens. The reflective optics 122 are configured to redirect the illumination that passes through the LOE 114 back toward the out-coupling device 120 while also collimating the illumination, e.g., as shown in FIG. 1A. Although the reflective optics 122 are described above as a reflective lens, a wider range of other lens types and implementations can alternatively be used, including but not limited to, spherical, aspherical, or freeform refractive lenses formed from glass or plastic, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above. FIG. 2C FIG. 2A

[0071] ​​​The out-coupling means 120 is configured to direct the illumination coming out of the LOE 114 towards the EMB 128 for projection onto the user’s eye 180. In some embodiments, the out-coupling means is shown as a plurality of parallel partially reflective surfaces (also referred to herein as facets 124) arranged within the LOE 114 at an oblique angle to the major outer surfaces 116 and 118 of the LOE 114. The facets 124 comprise an angle-dependent coating that provides high transmission at certain angles and partial reflection at other angles.

[0072] For example, by reflecting off the major outer surfaces 116 and 118, the light beam 126 travels through the LOE 114 towards the reflective optical means 122, as shown. The light beam 126 travels through the facets 124 to the reflective optical means 122, for example due to high transmission at the angle of travel, and the reflective optical means 122 reflects the light beam 126, redirects the light beam 126 back towards the facets and collimates the light beam 126 to have a width D. FIG. 2C After being reflected by the reflective optical means 122, when the collimated light beam 126 encounters the facets 124, the light beam 126 is redirected by the facets 124 towards the EMB 128, for example due to partial reflection at the angle of travel of the light beam 126, with a width D EMB that is approximately the same as the width of the EMB 128. EMB

[0073] Although the description herein relates to a facet-based out-coupling means, any other out-coupling means can alternatively be utilized, including for example an out-coupling means with diffractive optical elements.

[0074] The reflective optical means 122 can have a cylindrical optical power that reflects at least a portion of the illumination back towards the facets 124 along an in-plane direction by internal reflection of the major outer surface 116 and the major outer surface 118. The illumination after reflection from the reflective optical means 122 is collimated in a plane perpendicular to the major outer surfaces 116 and 118 and in a plane parallel to the major outer surfaces 116 and 118. The reflective optical means 122 can be integrated with the edges of the LOE 114 and have a cylindrical axis perpendicular to the major outer surfaces 116 and 118 of the LOE 114. In some embodiments, the reflective optical means 122 can comprise diffractive optical elements with cylindrical power integrated into the LOE 114.

[0075] ​The reflective optics 122 can have high reflectivity within an angular range corresponding to the illumination when light propagates through LOE 114, and low reflectivity, such as transmission or absorption, at angles outside this range. In this way, the reflective optics 122 reflects light emitted by POD 112 that propagates through LOE 114 via total internal reflection, while suppressing reflection of light from any other light source. For example, light from an external light source (e.g., the sun) will reach the reflective optics 122 at an angle within the low reflectivity range and will be reflected away from LOE 114 or absorbed. In this way, the intensity of potential ghosting caused by external light sources is reduced. In some embodiments, the reflective optics 122 is formed with a reflectivity depending on the angle of incidence of light, for example, using a multilayer coating technique that provides selective reflectivity and desired angle. In another embodiment, the reflective optics 122 can be formed using one or more volumetric Bragg gratings with high diffraction efficiency over a relatively narrow angular range.

[0076] Now refer to FIG. 3 This describes an example image projection component 210 according to another embodiment. For example... FIG. 3 As shown, similar components have the same FIG. 1 and FIGS. 2A-2C Similar reference numerals to image projection assembly 110 are used. For example, image projection assembly 210 includes a POD 212, LOE 214, a main outer surface (not shown), a coupling device 220, a facet 224, and other components similar to those in image projection assembly 110 described above. Image projection assembly 210 is configured to project an image onto a user's eye 180 using one-dimensional (1D) pupil dilation. In an embodiment of image projection assembly 210, POD 212 is attached to LOE 214 on top, instead of using reflective optics 122 to redirect illumination back to facet 224. For example, illumination emitted from POD 212 propagates through LOE 214 and is gradually emitted toward EMB (not shown) via facet 224 of coupling device 220. In this embodiment, the illumination output from POD 212 is already collimated when it enters LOE 214.

[0077] Return to reference FIG. 1Controller 140 includes a computing device having one or more processing devices, memory, or other components. For example, controller 140 can include a central processing unit (CPU), a field-programmable gate array (FPGA), a microcontroller, a special-purpose circuit, or any other component. Controller 140 is configured to control POD 112 to generate images and output the images to LOE 114 for projection to user’s eye 180, as will be described in more detail below.

[0078] In some implementations, controller 140 can be integrated into image projection assembly 110 or into a device that includes image projection assembly 110, such as eyeglasses, a head-mounted display, or another device. In some implementations, controller 140 can be located remotely from image projection assembly 110. For example, image projection assembly can include a wired or wireless communication device configured to communicate with controller 140. As an example, controller 140 can be included as part of a mobile device or other computing device that is separate from image projection assembly 110 or a device that includes image projection assembly 110.

[0079] Eye tracking system 160 includes one or more eye tracking cameras, lasers, or other optical devices configured to determine a position of pupil 182 of user’s eye 180 and configured to generate position information, such as coordinates or other position information, corresponding to the position. The position information can be provided to controller 140 for use in controlling POD 112 to generate images and output the images to LOE 114.

[0080] Reference will now be made to FIGS. 2A-9C Portions of eye pupil illumination according to some implementations will now be described and illustrated in more detail.

[0081] Reference will now be made to FIG. 4 In an example scenario, ideal lens 400 has an aperture D0. Lens 400 is illuminated by a parallel light beam 402, where the wavefront is not perfectly planar, but contains one or more optical aberrations. In this example scenario, lens 400 produces an image at a point P0 on a screen 404 located at a focal plane of lens 400. Due to the presence of aberrations in the light beam, the image becomes blurred and has a size d0.

[0082] Reference will now be made to FIG. 5In another example scenario, the ideal lens 400 is illuminated by the same aberrated light beam 402. However, in this example scenario, the stop 406 is positioned in front of the lens 400 such that only a sub-aperture of diameter D1 is illuminated by the light beam 402, where D1 is smaller than D0. In this example scenario, the lens 400 produces an image at a point P1 on the screen 404, which can generally be different from the point P0 of the example scenario FIG. 4 The image is blurred and has a size d1 that is smaller than the image size d0 of the image produced in the example scenario FIG. 4

[0083] As seen in the example scenarios of FIG. 4 and FIG. 5 Reducing the diameter of the aperture through which the lens 400 is exposed to the light beam 402 can improve the image quality, as long as the resulting geometric image size d1 is larger than the diffraction limit of the aperture, as seen in the example scenarios of

[0084] FIG. 6A and FIG. 6B An example scenario using the image projection assembly 110 in which a full angular aperture of the POD 112 is illuminated by the light beam 134 is shown. The collimated light beam 134 has a width equal to D EMB , i.e. the width of the EMB 128. FIG. 6B An eye 180 of a user is shown, having a pupil 182 with a width D EP The collimated light beam 134 has a width equal to D EMB The collimated light beam 134 illuminates the eye 180 and projects an image J0 via the lens 184 of the eye 180 onto the retina 186 of the eye 180. In this example scenario, the resulting image J1 projected onto the retina 186 becomes blurred and has a size s1 due to aberrations of the light beam 134 collected by the pupil 182.

[0085] FIG. 7A and FIG. 7B An example scenario using the image projection assembly 110 according to an example embodiment is shown, in which only a portion of the angular aperture of the POD 112 in the plane of the LOE 114 is illuminated by the light beam 136. For example, a selective illumination system in the POD 112, such as will be described in more detail below, can be used to illuminate only a portion of the angular aperture of the POD 112. In this example scenario, the angular aperture of the POD 112 in the plane normal to the plane of the major outer surfaces 116 and 118 of the LOE 114 can be fully illuminated. The collimated light beam 136 has a width equal to D2 after reflection from the reflective optics 122, D2 being smaller than the width D​EP such that the collimated light beam 136 illuminates only a portion of the pupil 182, as shown in FIG. 7B The collimated light beam 136 projects the image J2 onto the retina 186 via the lens 184 of the eye 180. The image J2 is projected on the retina 186 at a different location than the location of the image J1, the image J2 is blurred and has a size s2 that is smaller than the size s1 of the image J1.

[0086] FIG. 8A and FIG. 8B An example scenario is shown using the image projection assembly 110 according to an example embodiment in which another portion of the angular aperture of the POD 112 in the plane of the LOE 114 is illuminated by the light beam 138. For example, the selective illumination system in the POD 112 can be used to illuminate only a portion of the angular aperture of the POD 112, in this example, a different portion of the angular aperture than the portion shown in FIG. 7A and FIG. 7B and illuminated by the light beam 136. Like the light beam 136, the angular aperture of the POD 112 in the plane normal to the plane of the major outer surfaces 116 and 118 of the LOE 114 can be fully illuminated by the light beam 138. The width of the collimated light beam 138 after reflection from the reflective optical device 122 is equal to D3, which is smaller than the width D EP such that the collimated light beam 138 illuminates only a portion of the pupil 182, as shown in FIG. 8B The collimated light beam 138 projects the image J3 onto the retina 186 via the lens 184 of the eye 180. The image J3 is projected on the retina 186 at a different location than the location of both the image J1 and the image J2, the image J3 is blurred and has a size s3 that is smaller than the size s1 of the image J1.

[0087] In each of the example scenarios, the location J1, J2 and J3 of the projected image on the retina 186 can be defined as the centroid of the illuminated location. In this way, FIGS. 6A-8B the locations J1, J2 and J3 of the projected image in

[0088] FIGS. 9A-9C Optical aberration maps of the optical system 100 according to each of the example scenarios described above for FIGS. 6A-8B are shown. In FIGS. 9A-9C the axis px refers to the pupil coordinate axis and is aligned with FIGS. 6A-8BIn the diagram, the X-axis coincides, the ey-axis refers to the lateral ray error as a function of the pupil incident radius, ac, bc and cc refer to the aberrations generated by the reflecting optics 122, af refers to the width of the beam with a fully illuminated angular aperture, and bf and cf refer to the sub-aperture widths of the POD 112.

[0089] FIG. 9A The image shows the entire area of ​​the pupil 182 illuminated by the beam 134. FIG. 6A and FIG. 6B Aberration map of an example scene. For example... FIG. 9A As shown, with FIG. 9B and FIG. 9C Compared to the aberrations shown, the aberration ac generated by the reflective optical device 122 is relatively large.

[0090] FIG. 9B The image shows the first part of the pupil 182 illuminated by the beam 136. FIG. 7A and FIG. 7B Aberration map of an example scene. For example... FIG. 9B As shown, aberration bc forms a first dashed rectangle on the aberration map, and aberration bc is smaller than aberration ac produced by complete illumination of pupil 182.

[0091] FIG. 9C This shows the second part of the pupil 182 illuminated by the beam 138. FIG. 8A and FIG. 8B Aberration map of an example scene. For example... FIG. 9C As shown, the aberration cc forms a second dashed rectangle on the aberration map, and the aberration cc is smaller than the aberration ac produced by complete illumination of the pupil 182. Additionally, as... FIG. 9B and FIG. 9C As shown, the types of aberrations formed by beams 136 and 138 are different, wherein, for example, beam 138 may have a reduced aberration cc due to the reflective optical device 122 compared to aberration bc of beam 136.

[0092] Reference FIGS. 10-15 This document describes example optical architectures and configurations of optical systems 100 and POD 112 according to various embodiments. Each embodiment of POD 112 includes an illumination system 300, projection optics 302, and an SLM 304. Selective pupil illumination or selective EMB illumination in these embodiments is achieved by the illumination system 300, and POD 112 uses the illumination system 300 as a LOE 114 of a 2D extended system. FIGS. 2A-2C ) or LOE 214 of the 1D extended system ( FIG. 3The image generator of FIG. 1 can be implemented as a light source array 306 and projection optics 302. The projection optics 302 are configured to collimate the light beams from the pixels of the SLM 304 such that each pixel generates a collimated light beam, and the collimated light beams from different pixels propagate in different directions. The projection optics 302 are further configured to inject the collimated light beam from each of the pixels into the LOE 114 or the LOE 214. For example, in some implementations, the projection optics 302 can include a single lens with the SLM 304 located in the focal plane of the lens. In other implementations, the projection optics 302 can include one or more additional or alternative optical elements including, for example, lenses, mirrors, waveplates, beam-splitting prisms, or other optical elements.

[0093] FIG. 10 and FIG. 11 An example configuration of the POD 112 in which selective eye pupil illumination or selective EMB illumination can be implemented is shown in accordance with an implementation. In this implementation, the illumination system 300 includes an array of light sources 306, such as LEDs or other selectively activatable light sources, located in the focal plane of an optical device 308, such as a collimating lens. Although the optical device 308 is described as a collimating lens, a wider range of lens types and implementations can be used including, but not limited to, spherical, aspherical, or freeform refractive lenses formed from glass or plastic, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above.

[0094] The array of light sources 306 can include red, green, and blue light sources or multi-color light sources configured to generate red, green, blue, or other colors. The light sources are configured to generate color images in a color sequential mode of operation of the SLM 304. Although the array 306 is shown in FIGS. 1-3 as having a particular number of light sources, the array 306 can alternatively include any other number of light sources. For example, additional light sources can be included in the array 306 to enable smoother EMB scanning. In some implementations, the aperture scanning can be performed in the YZ plane as shown in FIG. 4, while the entire aperture of the POD 112 can be illuminated in the XZ plane as shown in FIG. 5. In some implementations, the illumination system 300 can also include an optional diffuser 310 that expands the divergence of the light beams in the XZ plane as shown in FIG. 6. In other implementations, a cylindrical lens can be used in each of the light sources in place of the diffuser 310 to reduce the light beam divergence in the XY plane. FIG. 10 and FIG. 11 FIG. 10 FIG. 11 FIG. 11

[0095] ​​​​The output beam from each light source in the array 306 of the illumination system 300 is collimated or nearly collimated illumination provided to the SLM 304 via the projection optics 302. The angle of the collimated beam at the SLM 304 depends on which light source in the array 306 is activated, and the divergence of the illumination generated by the light source depends on the size of the light source and on the scattering angle range of the optional diffuser 310. Each light source in the array 306 corresponds to a different angular sub-aperture of the POD 112, where switching between the angular sub-apertures of the POD 112 can be achieved, for example, by turning on and off the respective light sources in the array 306. As FIG. 10 shown, for example, the light source 312 and the light source 314 generate the light beam 316 and the light beam 318, respectively, which are provided to the SLM 304 after being collimated by the optical arrangement 308 and optionally scattered by the diffuser 310. As FIG. 10 shown, the light beam 316 and the light beam 318 illuminate a field of view (FOV) region between the FOV A and the FOV B of the POD 112, respectively.

[0096] FIG. 12 Example embodiments of the optical system 100 are shown using the example configuration of the POD 112 described above for FIG. 10 and FIG. 11 The eye tracking system 160 is configured to measure the position of the pupil 182 of the eye 180 relative to the LOE 114 and to provide this measured position as position information to the controller 140. The controller 140 is configured to determine, based on the position information, the light source in the array 306 that can be turned on or otherwise activated to illuminate the aperture that will project the image onto the pupil 182 at the measured position or a sub-aperture of the pupil 182. In some embodiments, the controller 140 is also configured to compute or determine any warping to be applied to the image provided to the SLM 304 to compensate for any aberrations of the image caused by the POD 112 and the other components of the reflection optical arrangement 122 or the LOE 114.

[0097] The warping applied to the image provided to the SLM 304 can depend, for example, on which light source in the array 306 is activated, the position of the pupil 182, which part of the EMB is illuminated, or on any other criteria. The position of the image on the retina 186 can be different for the same FOV, for example, depending on which sub-aperture is observed by the pupil 182, for example, as FIGS. 7A-8BThe type and amount of warping applied by the controller 140 to the image at the SLM 304 can depend on which sub-aperture or corresponding light source of the POD 112 is activated, because the location of the projected image on the retina is different for each sub-aperture. By applying warping to the image based on which sub-aperture or light source is activated and, in some implementations, according to the location on the retina 186 where the image will be projected, the image projected onto the retina 186 from each sub-aperture or light source can be aligned such that the user sees the same or approximately the same image regardless of which sub-aperture or light source is activated to provide the image. For example, the control of the SLM 304 and the light source array 306 can be synchronized by the controller 140 to enable fast switching between light sources based on the location of the pupil 182 tracked by the eye tracking system 160. By synchronizing the control of the SLM 304 and the light source array 306, images that are corrected for aberrations can be projected regardless of changes in the location of the pupil 182 or the corresponding changes in the light source activated by the controller 140.

[0098] Reference will now be made to FIGS. 13A-13D An example process for determining the image warping to be applied to the light beams by the SLM 304 based on which light sources of the array 306 are activated will now be described. The example process can be used for a single FOV point or for a small local area of the FOV.

[0099] Reference will now be made to FIG. 13D An example process for operating the optical system 100 will now be described. The process can be performed, at least in part, by the controller 140, the eye tracking system 160, and the POD 112, or the process can be performed, at least in part, by other portions of the optical system 100. FIG. 13D The process of FIG. 5 includes steps 500-508. Although the process of FIG. 5 will be described with reference to the optical system 100 of FIG. 1, the process of FIG. 5 can be performed by other optical systems. FIG. 13D The process of FIG. 5 is described herein as having a certain order of steps or steps, but the process can alternatively perform the steps in any order, can include additional steps, can include fewer steps, or can perform only a portion of the steps described below in other implementations.

[0100] At step 500, the eye tracking system 160 locates the position of the pupil 182 within the EMB 128, e.g., as shown in FIG. 1. As an example, the eye tracking system 160 can utilize one or more eye tracking cameras or other optical elements to locate the position of the pupil 182. The eye tracking system 160 provides position information corresponding to the determined position, e.g., coordinates or other information, to the controller 140. FIG. 13A

[0101] ​At step 502, the controller 140 determines the light sources in the array 306 that can be activated to project an image onto a portion of the pupil 182. For example, the controller 140 can maintain a coordinate map of the EMB 128 that indicates which light source corresponds to each portion of the EMB 128. The controller 140 can select the light sources to activate based at least in part on a comparison between the location information and the coordinate map, e.g., by determining the location of the pupil 182 relative to the EMB 128 and identifying the corresponding light sources based on the coordinate map. In some implementations, the controller 140 is configured to identify, for each light source from the array of light sources 306, which region of the EMB 128 will be illuminated for each FOV point or small FOV local area. Given the location of the pupil 182 and the FOV point to project, the controller 140 can identify the light sources to turn on. For example, as shown in FIG. 13A , such light sources generate light beams 142 that illuminate a region that intersects the pupil 182. In FIG. 13A , the coordinates (x0, z0) correspond to the center of the illuminated region within the pupil 182.

[0102] At step 504, the controller 140 determines which deformations to apply to the image at the SLM 304 based at least in part on the selected light sources. In some implementations, the deformations can also or alternatively be determined based at least in part on the location information, e.g., in cases where multiple light sources can be utilized to illuminate the same location but where the light beams have different angles of collimation.

[0103] FIG. 13B and FIG. 13C shows the aberration curves of the optical system 100 with the reflective optical device 122, similar to the curves shown in FIGS. 9A-9C . FIG. 13A and 13B The aberration curves in FIG. 13B and FIG. 13C give the local displacement of the image along the y-axis and x-axis, respectively, dy and dx. Deformations equal to dx and dy can be applied to the image projected through the sub-aperture defined by the intersection of the light beam 142 and the pupil 182.

[0104] In some implementations, the controller 140 can determine the deformation to apply, e.g., using a lookup table having predefined deformation values based on the target position and the light source to activate. For example, the lookup table can be generated using a variety of techniques, including, e.g., using an inversion method to adjust the deformation based on the resulting image projected on the eye 180 or a representation of the eye, by simulating or modeling the aberrations and potential deformations, or in any other way. In some implementations, the illumination beam aperture can be slightly different for each of the RGB light sources. In this case, the controller 140 can also take these differences into account when applying the deformations to the red, green, and blue images to correct for such small differences in the aperture and correct for potential chromatic deformations of the projection optics, e.g., lateral color.

[0105] At step 506, the controller 140 activates the selected light source to output the image.

[0106] At step 508, the controller 140 provides the determined deformation to be applied to the image to the SLM 304 before providing the image to the LOE 114. The image is then projected by the LOE 114 onto a portion of the pupil 182, and the process returns to step 500 and continues for each frame of the image. In this way, changes in the position of the pupil 182 are taken into account, the corresponding light source is activated, and the appropriate deformation is applied to generate an image with as little deformation as possible.

[0107] In some implementations, the controller 140 is configured to sequentially activate each light source in the array 306 to perform a full EMB scan, where the deformation can be determined for each light source and applied to the image at the SLM 304. As an example, in the case where the eye tracking system 160 is not present or available and the position of the pupil 182 is not known, such sequential activation can be utilized to quickly present the image to each portion of the EMB 128 and ensure that at least one deformed image is projected onto the position of the pupil 182.

[0108] FIG. 14 and FIG. 15 An example configuration of a POD 112 in which selective eye pupil illumination or selective EMB illumination can be implemented is shown, according to another implementation. In this implementation, an illumination system 600 replaces the illumination system 300 in the POD 112. The illumination system 600 includes an array of light sources 606 similar to the array of light sources 306, a first optical arrangement 608, a first micro-lens array (MLA) 610, a second micro-lens array 612, and a second optical arrangement 614.

[0109] The first optical arrangement 608 and the second optical arrangement 614 can include lenses, such as Fresnel lenses or diffractive lenses, which can be used to collimate the light beams of the image. Although the first optical arrangement 608 and the second optical arrangement 614 are described above as including particular types of lenses or optical components, a wider range of other lens types or optical components and implementations can alternatively be used, including but not limited to spherical, aspherical, or freeform refractive lenses formed from glass or plastic, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above.

[0110] The first MLA 610 and the second MLA 612 each include an array of lenses that can act as a single element. In some implementations, the lenses of the first MLA 610 and the second MLA 612 can include refractive lenses. In some implementations, a baffle arrangement (not shown) can be inserted between the first MLA 610 and the second MLA 612 to reduce cross-talk between the collimating optics. Although the first MLA 610 and the second MLA 612 are described above as including particular types of lenses or optical components, a wider range of other lens types or optical components and implementations can alternatively be used, including but not limited to spherical, aspherical, or freeform refractive lenses formed from glass or plastic, diffractive lenses, Fresnel lenses, reflective lenses, and any combination of the above.

[0111] Each lens in the first optical arrangement 608 and the first MLA 610 is together configured to create an image of the light source array 606 at the plane of the second MLA 612. The second MLA 612 and the second optical arrangement 614 are together configured to generate an image of each lenslet element of the first MLA 610 at the plane of the SLM 304. In an example configuration, the light source array 606 is located in the focal plane of the optical arrangement 608, the second MLA 612 is located in the focal plane of the first MLA 610, and the SLM 304 is located in the focal plane of the optical arrangement 614. Selective illumination of the EMB 128 is achieved by turning on and off the light sources in the array 606 in a coordinated and timed manner, such that only the target portion of the EMB 128 and the corresponding portion of the pupil 182 are illuminated. For example, as shown in FIG. 6, the light source array 606 includes a first light source 606a and a second light source 606b. The first light source 606a is turned on to illuminate the first EMB 128a and the first portion of the pupil 182a, and the second light source 606b is turned off. The first light source 606a is then turned off, and the second light source 606b is turned on to illuminate the second EMB 128b and the second portion of the pupil 182b. The first light source 606a and the second light source 606b can be turned on and off in a coordinated and timed manner to selectively illuminate the EMB 128 and the pupil 182. FIG. 14 and FIG. 15As seen, light beams 616 and 618 of the image can be generated by selectively activating light source 620 and light source 622, for example, at different times. Light beams 616 and 618 travel through first optical arrangement 608, through one or more lenses of first MLA 610, through one or more lenses of second MLA 612, and through second optical arrangement 614, and are output from illumination system 600 as collimated light beams to FOV A and FOV B of SLM 304 via projection optics 302. SLM 304 then imparts a distortion to the images of light beams 616 and 618 and outputs light beams 616 and 618 to LOE 114 for projection to pupil 182 in a manner similar to that described above for the process of FIG. 13.

[0112] Referring to FIG. 16A and FIG. 16B Schematic diagrams in the YZ plane and the XZ plane of image projection assembly 710 according to some embodiments will now be described, respectively. Image projection assembly 710 can include similar components to those described above for image projection assembly 110, where such components have similar reference numerals. For example, image projection assembly 710 includes POD 712 and LOE 714. LOE 714 includes major external surface 716 and major external surface 718, out-coupling arrangement 720 including facets 724, for example, and reflective optical arrangement 722 similar to components of LOE 114.

[0113] Image projection assembly 710 also includes wedge 750 between POD 712 and LOE 714. Wedge 750 is configured to reduce FIGS. 9A-9C aberrations of the cylindrical mirror shown. Light beams 726 are coupled from POD 712 into LOE 714 via wedge 750, and a waveguide aperture stop is located after wedge 750. In some embodiments, surfaces 752 and 754 of wedge 750 can optionally have one- or two-dimensional optical power to compensate for optical aberrations of POD 712 and improve image quality. LOE 714 also includes mixer 756, for example, a semi-reflective plane, parallel to major external surfaces 716 and 718. In some scenarios, for example, the output light beams 726 of POD 712 can not fully fill LOE 714. Mixer 756 is used to spread light beams 726 over the entire aperture of LOE 714. In some embodiments, mixer 756 can be located within LOE 714 between wedge 750 and out-coupling arrangement 720, for example, as shown in FIG. 15. FIG. 16AIn other implementations, the mixer 756 can be located within the LOE 714 between the out-coupling device 720 and the reflective optical device 722. The reflective optical device 722 can also include a waveplate 723, such as a quarter waveplate, disposed between the out-coupling device 720 and the reflective optical device 722. In implementations where the mixer 756 is located between the out-coupling device 720 and the reflective optical device 722, the light beam will pass through the mixer 756 twice before being directed by the out-coupling device 720 from the LOE 714 toward the EMB or pupil. In such cases, the mixer 756 can have a shorter length along the z-direction compared to implementations where the mixer 756 is located between the wedge 750 and the out-coupling device 720.

[0114] The image projection assembly 710 can also include a polarizer 725 disposed between the LOE 714 and the wedge 750. In some implementations, the out-coupling device 720 includes a surface, such as facet 724, that is partially reflective to one polarization (e.g., polarization (s)) but substantially transparent to an orthogonal polarization (e.g., polarization (p)). If the input light beam 726 output from the POD 712 is p-polarized, the input light beam 726 will propagate toward the reflective optical device 722 without being coupled out by the out-coupling device 720. After reflecting from the reflective optical device 722 and passing through the waveplate 723, the light becomes s-polarized and is coupled out of the LOE 714 by the out-coupling device 720 when propagating back from the reflective optical device 722 toward the POD 712. The polarizer 725 is configured to block the back-propagating light beam from entering the POD 712.

[0115] In some implementations, the LOE 714 can also include optional cover plates 727 disposed on the major external surfaces 716 and 718 of the LOE 714. The cover plates 727 make the thickness of the out-coupling device 720 in a direction normal to the waveguide major surfaces less than the overall thickness of the LOE 714. The LOE 714 can also include optional polarizers (not shown) disposed parallel to the major external surfaces 716 and 718 of the LOE 714, the polarizers configured to suppress the passage of p-polarized light. For example, such an optional polarizer (not shown) can be disposed in front of the LOE 714, e.g., to the left of the LOE 714 in FIG. 16A Another optional polarizer (not shown) can be disposed behind the LOE 714, e.g., to the right of the LOE 714 in FIG. 16A Another optional polarizer (not shown) can be disposed behind the LOE 714, e.g., to the right of the LOE 714 in

[0116] The illumination system 800 of the POD 712 is shown in FIG. 16Bare shown in more detail in FIG. 16B The illumination system 800 includes an array of light sources 802, a polarization beam splitter disposed between prismatic devices 804 and 806, a quarter wave plate 808, and a collimating optical device 810 (e.g., a catadioptric lens or a mirror). Light beams emitted by the light sources in the array 802 become collimated after reflection from the optical device 810 and are injected into an LOE 812 (e.g., a waveguide) having a set of semi-reflective facets 814 for extracting light toward a prism 816 of the POD 712 and two parallel major planar surfaces. The POD 712 can also include an optional diffuser 818 between the LOE 812 and the prism 816 of the POD 712. The illumination system 800 can also include optional diffusers (not shown) between the prisms 804 and 806 and the LOE 812. Light beams that are coupled out of the prisms 804 and 806 into the LOE 812 propagate through the LOE 812 by means of total internal reflection and are coupled out of the LOE 812 toward the optional diffuser 818 and the prism 816 of the POD 712 by the facets 814. An example path of the light beams propagating in the illumination system 800 is shown as arrows in FIG. 16A

[0117] As shown in FIG. 16C The light beams received from the illumination system 800 enter the prism 816 and are redirected to an SLM 820. A polarization beam splitter 822 can be disposed between the prism 816 of the POD 712 and another prism 824. The SLM 820 can be similar to the SLM 304 and is configured to be controlled by the controller 140. In this implementation, the SLM 820 can be implemented as a reflective SLM or an emissive SLM described above with reference to the SLM 304.

[0118] The illumination light beams are modulated by the SLM 820, for example, in a manner similar to that described above for the SLM 304, and are directed through the polarization beam splitter 822 and the prism 824 toward a reflective optical device 826 of the POD 712. The light beams reflected by the reflective optical device 826 are then directed by the prism 824 toward the wedge 750.

[0119] FIG. 16B An illumination system 900 that can be used with the POD 712 and the LOE 714 according to another implementation is shown. In this implementation, the illumination system 900 replaces the illumination system 800. FIG. 16C ​The illumination system 800 includes a light source array 902 and an imaging system including an optical element 904. The optical element 904 may include a refractive lens, a Fresnel lens, a diffractive or phase lens, such as a Pancharatnam-Berry lens, or any other type of lens in any combination. A light beam emitted from the sources in array 902 is collimated after passing through the optical element 904 and injected into a LOE 906 via a prism 908. The light propagates in the LOE 906 by means of total internal reflection and is guided by a semi-reflective facet 910 into a prism 816 of a POD 712. FIG. 17 The arrows in the diagram indicate the light propagation path in the lighting system 900. Similar to the lighting system 800, optional diffusers (not shown) may also be included, for example, between LOE 906 and prism 816 and between optical element 904 and prism 908.

[0120] FIGS. 18A-18C and FIG. 10 The SLM 304 is shown. FIG. 10 The two pixels will come from the lighting system 300 ( FIGS. 18A-18C This embodiment of the image projection assembly 110 reflects a light beam into two beams, 1000 and 1002. Beams 1000 and 1002 correspond to two different points in the field of view (FOV), FOV A and FOV B respectively, but also to the same angular sub-aperture of the POD 112 defined by the active illumination system light source. Beams 1000 and 1002 are extended by a facet 124 of the coupling device 120 and projected onto the EMB 128.

[0121] FIG. 6B This illustrates how beams 1000 and 1002 with different FOVs are projected onto the EMB 128 and the eye 180. FIG. 6B ) pupil 182 ( FIGS. 18A-18C Example scenario on ). In FIG. 18A In the diagram, beams 1000 and 1002 are shown as dashed lines, and the position of pupil 182 is shown as a circle.

[0122] Reference FIG. 18A The example scenario describes two possible positions of the pupil, 1004 and 1006. At position 1004, both beams 1000 and 1002 illuminate the pupil, displaying FOV A and FOV B to the user. At position 1006, only beam 1000 illuminates the pupil, displaying only FOV A to the user. FIG. 18B The example scenario illustrates that different sub-apertures of POD 112 can be illuminated to make a particular FOV visible to the pupil. For example, to make FOV B visible to the pupil at position 1006, different sub-apertures of POD 112 can be illuminated.

[0123] Referring FIG. 18A , another example scenario is described in which projections of beams 1008 (FOV A) and 1010 (FOV B) at EMB 128 are shown, corresponding to different sub- apertures of the illuminated POD 112. In this example scenario, the same two possible positions 1004 and 1006 of the pupil are shown. In position 1004, both beams 1008 and 1010 illuminate the pupil, showing both FOV A and FOV B to the user, although beam 1008 only partially illuminates the pupil 182 at position 1004. In position 1006, only beam 1010 illuminates the pupil, showing FOV B to the user. In order for FOV B to be visible to the pupil in position 1006, a different sub-aperture of the POD 112 can be illuminated.

[0124] In FIG. 18B and FIG. 1 the example scenario, neither FOV A nor FOV B can be seen when the pupil is in position 1006. In some embodiments, controller 140 FIG. 18C is configured to address this issue by sequentially turning on and off one or more light sources in array 306 in a manner that ensures that the full set of FOVs will be visible to the user’s eye when the pupil is located at a particular location on EMB 128.

[0125] Referring FIGS. 19A-19C , another example scenario is described in which projections of beams 1012 (FOV A) and 1014 (FOV B) at EMB 128 are shown, corresponding to different sub- apertures of the illuminated POD 112. In this example scenario, the same two possible positions 1004 and 1006 of the pupil are shown. In position 1004, both beams 1012 and 1014 illuminate the pupil, showing both FOV A and FOV B to the user. In position 1006, the entire beam 1014 illuminates the pupil, showing FOV B to the user, but only a partial beam 1012 illuminates the pupil. Such partial illumination can result in a degraded image of FOV A due to, for example, diffraction at the edge of the pupil. In some embodiments, controller 140 can be configured to command SLM 304 to project FOV B without projecting FOV A while the pupil is in position 1006 and this particular sub-aperture of POD 112 is illuminated, to suppress the projection of the degraded image of FOV A. Alternatively, FOV A can be projected at the same position 1006 by sequentially activating another sub-aperture or light source of the POD 112.

[0126] ReferringFIG. 10 An illumination system 1100 according to another embodiment will now be described. The illumination system 1100 can be used, for example, in place of the illumination system 300 of the POD 112 as shown in FIG. 14 , in place of the illumination system 600 of the POD 112 as shown in FIG. 16A , in place of the illumination system 800 of the POD 712 as shown in FIG. 16B and FIG. 16C , in place of the illumination system 900 of the POD 712 as shown in FIG. 19A , or can be used with any other POD. The illumination system 1100 can include components similar to those found in any of the illumination system 300, the illumination system 600, the illumination system 800, and the illumination system 900.

[0127] In FIG. 16A , reference numerals corresponding to the POD 712 will be used in conjunction with the description of the illumination system 1100. For example, as mentioned above, the POD 712 includes the prisms 816 and 824, the SLM 820, the polarization beam splitter 822, the reflective optical arrangement 826, and the optional diffuser 828, all of which can function as described above with reference to FIG. 16B and FIGS. 19A-19C .

[0128] As shown in FIG. 19B , the illumination system 1100 includes an MLA 1102 and a matrix of light sources 1104, such as a micro-LED display or other light source arrangement. The light beams output from the illumination system 1100 are provided to the prisms 816. As shown in FIGS. 20-29 , each microlens in the MLA 1102 collimates light from its respective light source. The direction or angle of collimated illumination can vary from microlens to microlens depending on which sources in the matrix 1104 are activated. For example, when the light source 1106 is activated, the light beam 1108 is collimated by the microlens 1110 and output at a first direction or angle, while when the light source 1112 is activated, the light beam 1114 is collimated by the same microlens 1110 but output at a second direction or angle, different from the first direction or angle. The configuration of the MLA 1102 and the matrix of light sources 1104 enables the controller 140 to present different angles of illumination for different regions of the SLM 820. In another embodiment, a micromirror array (not shown) can be used in place of the MLA 1102.

[0129] Reference is now made to FIG. 20 , embodiments in which the various embodiments of the optical system 100 described above can be configured for projecting time-division multiplexed light field images are disclosed.

[0130] Reference is made toFIG. 21 and FIG. 20 The functionality of the eye 180 when the lens 184 is focused at infinity or a finite distance will now be described separately.

[0131] As FIG. 21 seen, the lens 184 of the eye 180 is focused at infinity and the pupil 182 is illuminated by two beams (beam 1200 and beam 1202) that produce images PI and P2 at the retina 186, respectively. FIG. 21 The lens 184 of the eye 180 is shown focused at a finite distance, rather than at infinity, where the lens 184 has a shorter focal length. As a result FIG. 22 of the shorter focal length of the lens 184 in the middle, the images PI and P2 converge into a single image. When the images PI and P2 are projected onto the pupil 182 through a small angular sub-aperture, the blurring of the images PI and P2 is less when the focal length of the lens 184 of the eye 180 is changed.

[0132] FIG. 23 and FIG. 20 Embodiments of the image projection assembly 110 and the image projection assembly 210 are shown that project the beams 1300 and 1302 corresponding to different points in the FOV and different angular sub-apertures of the POD 112 and the POD 212 onto the EMB 128, thereby illuminating different regions of the pupil 182 as FIG. 21 and FIG. 24 shown. Although the image projection assembly 110 and the image projection assembly 210 are shown with particular components, each of the image projection assembly 110 and the image projection assembly 210 can include the LOE, the POD, any components of the illumination system, or other portions of the optical system 100 found in the various embodiments described herein.

[0133] Referring now to FIG. 24 In some embodiments, the controller 140 is configured to divide the projection of the single image into a plurality of frames (e.g., frame 1, frame 2,... frame n) and is configured to project each frame to the eye 180 of the user in sequence. The controller 140 in this embodiment is configured to move the image 13001, 13002,... 1300 n in each successive frame 1, 2,... n so that the objects in the images 13001, 13002,... 1300 n of the successive frames 1 to n are slightly shifted relative to the previous frame, as FIG. 24The controller 140 is configured to project frames one at a time in rapid succession, and in some embodiments, one or more shifted frames can be projected using different sub-apertures of the POD 112 by activating different light sources in the array 306. As FIG. 22 seen in FIG. 13A, for example, the image 1300 for frame n is shifted by a distance e relative to the image 1300 for frame 1. In this way, the controller 140 can use time-division multiplexed projection of frames of images to simulate a one-dimensional light field. n

[0134] In some cases, the time-division multiplexed light field projection described above is created in only one dimension, for example, along the axis X in FIG. 13B. Along the axis Z, the beam of the image is wide and illuminates the entire aperture of the eye in the Z direction due to the pupil expansion of the facet 124. Thus, the image in the Z direction is sharp only when the eye is focused at infinity, and the image in the Z direction becomes blurred when the accommodation of the eye changes to a finite focus. FIGS. 25A-25C

[0135] Referring to FIGS. 25A-25C , an embodiment of an image projection assembly 210 is shown in which the blurring of time-division multiplexed light field projection in the Z direction, such as described above, can be overcome. For example, the aperture of the beam 1400 in the Z direction can be limited, for example, by dynamically increasing or decreasing at least one of the reflectivity and intensity of some of the facets 224, for example, dynamically making the facets more reflective or more transparent. As an example, in the embodiment shown in FIG. 14A, the reflectivity of the facets 1402, 1404, 1406, and 1408 of the LOE 214 is configured to be dynamically adjusted. Although described with reference to the image projection assembly 210, in other embodiments, the image projection assembly 110 can alternatively be used. FIG. 25B

[0136] Although the size of the beam 1400 in the X dimension is limited due to the illumination system (for example, any one of the illumination systems 300, 600, 800, and 900, or other components described herein), in the Z direction, the beam 1400 illuminates the entire EMB 128 by reflections from the facets 1402, 1404, 1406, and 1408, as shown in FIG. 14B. In the example shown in FIG. 14B, the reflections from the facets 1404 and 1406 illuminate the pupil 182 at a particular location in the EMB 128, while the reflections from the other facets (for example, facets 1402, 1408, and others) do not illuminate the pupil 182 at the particular location in the EMB 128. As FIG. 25B FIG. 25C FIG. 26 ​​​​​As shown, if facet 1406 becomes transparent (non-reflective), only a portion of pupil 182 is illuminated in the Z direction. By dynamically adjusting one or both of the reflectivity and intensity of each facet, the image of the point on the retina 186 can become sharp in the Z direction for any position of pupil 182 and any accommodation of the lens 184 of eye 180.

[0137] Reference is now made to FIG. 1 , showing a dynamic facet structure 1500 for controlling one or both of the reflectivity and intensity of a facet 1502, in accordance with some embodiments. Facet 1502 can be highly transmissive for p-polarization and partially reflective for s-polarization. Dynamic facet structure 1500 includes facet 1502 disposed between a first liquid crystal layer 1504 and a second liquid crystal layer 1506. In some embodiments, liquid crystal layers 1504 and 1506 are parallel and can be parallel to facet 1502. The state of the liquid crystals in each of liquid crystal layers 1504 and 1506 is controlled by a voltage applied to the layer, for example, by controller 140 FIG. 25C ) in the "on" state, the liquid crystals of each of liquid crystal layers 1504 and 1506 operate as half- wave plates that rotate the polarization of light beam 1400 by 90 degrees. In the "off state, the polarization state of light beam 1400 does not change after passing through the layer of liquid crystals.

[0138] In this embodiment, as mentioned above, light beam 1400 propagating in LOE 214 is s-polarized, and facet 1502 is highly transmissive for p-polarization and partially reflective for s-polarization. In other embodiments, light beam 1400 propagating in LOE 214 can be p-polarized, and facet 1502 can be highly transmissive for s-polarization and partially reflective for p-polarization.

[0139] When liquid crystal layers 1504 and 1506 are in the "off state, the polarization of light beam 1400 when it encounters facet 1502 is s-polarization, and facet 1502 is partially reflective for light beam 1400, for example, as shown by facet 1404 in FIG. 25C . When liquid crystal layers 1504 and 1506 are in the "on" state, the light beam polarization at facet 1502 is p-polarization, and facet 1502 is transparent for light beam 1400, for example, as shown by facet 1406 in FIG. 1 .

[0140] Note that because there is a liquid crystal layer disposed on either side of facet 1502, such as liquid crystal layer 1504 disposed on one side of facet 1502 and liquid crystal layer 1506 disposed on the other side of facet 1502, the polarization of light beam 1400 will change from s-polarization to p-polarization after passing through liquid crystal layer 1504 before encountering dynamic facet structure 1500 when in the “on” state. Light beam 1400 will encounter facet 1502 while having p-polarization and pass through due to the high transmission of facet 1502 at p-polarization. Light beam 1400 will then encounter liquid crystal layer 1506 on the other side of facet 1502 and change back to s-polarization from p-polarization. Light beam 1400 then exits dynamic facet structure 1500 while having s-polarization. In this way, each dynamic facet structure 1500 can be independently controlled by controller 140 FIG. 27A ) to reflect or transmit light beam 1400 without affecting the polarization of light beam 1400 with respect to any of the other facets.

[0141] In another implementation, an electrically switchable Bragg reflector can alternatively be used to dynamically control one or both of the reflectivity and intensity of facet 224. For example, in some implementations, each of the facets 224 of outcoupling device 220 can include an electrically switchable Bragg reflector, the reflectivity, intensity, or both of which can be electrically controlled by controller 140.

[0142] Referring now to FIG. 27B and FIG. 27A , an image projection assembly 1610 according to some implementations will now be described. The image projection assembly includes a POD 1612, which can include similar components and functionality as POD 112, POD 212, or any other POD disclosed herein. The image projection assembly includes an LOE 1614, which can include at least some similar components and functionality as LOE 114, LOE 214, or any other LOE disclosed herein, except as described in more detail below. POD 1612 is configured to output a light beam 1626 to LOE 1614, which is directed by outcoupling device 1620 to EMB 128 and pupil 182.

[0143] In implementations of FIG. 27B and FIG. 27A , outcoupling device 1620 includes a switchable Bragg reflector (SBR) 1624 disposed on one of the major external surfaces 1616 or 1618 of LOE 1614, such as illustrated in FIG. 27B and FIG. 27BThe SBR 1624 is configured to reflect the light beam 1626 toward the location of the pupil 182 in the EMB 128 when in the "on" state. When in the "off state, the SBR 1624 provides total internal reflection so that the light beam 1626 propagates within the waveguide. In some embodiments, the SBR 1624 is divided into multiple selectively activatable regions, for example including regions 1628 that can be independently controlled by the controller 140. By "turning on" selected regions of the SBR 1624, a target portion of the pupil 182 can be illuminated, for example as described above with respect to FIG. 16B. In another embodiment, a transmissive switchable grating can be used instead of the SBR 1624. FIG. 28

[0144] Referring to FIG. 16A, the SBR 1624 is configured to reflect the light beam 1626 toward the location of the pupil 182 in the EMB 128 when in the "on" state. When in the "off state, the SBR 1624 provides total internal reflection so that the light beam 1626 propagates within the waveguide. In some embodiments, the SBR 1624 is divided into multiple selectively activatable regions, for example including regions 1628 that can be independently controlled by the controller 140. By "turning on" selected regions of the SBR 1624, a target portion of the pupil 182 can be illuminated, for example as described above with respect to FIG. 16B. In another embodiment, a transmissive switchable grating can be used instead of the SBR 1624. FIGS. 2A-2C FIG. 3 FIGS. 20-24 In this embodiment, the controller 140 is also configured to determine which of the facets 124 in the LOE 114 to set to the "on" state (semi-reflective) and the "off state (transmissive) due to changes in accommodation of the lens 184 of the eye 180 to enhance image sharpness in the Z direction, for example as described above. For example, to project an image of an object located at infinity, only a single image needs to be projected. However, to project an image of an object located at a finite distance, multiple images need to be projected, for example as described above with respect to FIG. 15B.

[0145] FIGS. 13A-13C

[0146] ​​​​​In an example scenario, an image of an object located at a finite distance from the user is projected. For a given position of pupil 182, multiple images (e.g., 2 images, 3 images... 100 images or more) are projected through different sub-apertures of pupil 182. For each sub-aperture projection, the image is warped twice. The first warp is configured to compensate for aberrations (e.g., ...). FIG. 20 The deformation is caused by the aberration shown. The second deformation is configured to move the image to create a light field, for example, as... FIG. 21 and FIG. 29 As shown

[0147] Reference FIG. 28 The operation will now be described. FIG. 29 An example process of the optical system 100 includes control of a selectively activated facet 124. This process can be performed at least in part by the controller 140, the eye-tracking system 160, the POD 112, and the LOE 114, or at least in part by other parts of the optical system 100.

[0148] FIG. 29 The process includes steps 1700 to 1712. Although this document will... ​ The process is described as having specific steps or a specific order of steps, but in other implementations the process may alternatively perform the steps in any order, may include additional steps, may include fewer steps, or may perform only a portion of the steps described below.

[0149] At step 1700, the eye-tracking system 160 determines the position of the pupil 182, for example, using one or more eye-tracking cameras or other optical elements, and provides the controller 140 with position information corresponding to the determined position, such as coordinates or other information.

[0150] At step 1702, controller 140 determines light sources in array 306 that can be activated to project an image onto a portion of pupil 182. For example, controller 140 may maintain a coordinate map indicating which light source corresponds to each portion of EMB 128. Controller 140 may select the light source to activate based at least in part on a comparison between position information and the coordinate map—for example, by determining the position of pupil 182 relative to EMB 128 and identifying the corresponding light source based on the coordinate map.

[0151] At step 1704, the controller 140 determines which of the small planes 124 needs to be set to the "on" state and which of the small planes 124 needs to be set to the "off" state, for example, as described above.

[0152] At step 1706, the controller 140 determines which warps to apply to the image at the SLM 304 based at least in part on the identified light sources to activate. In some embodiments, the warps can also or alternatively be determined based at least in part on the position information, e.g. in cases where multiple light sources can be utilized to illuminate the same location but where the light beams have different angles of collimation. In some embodiments, the controller 140 can determine the warps to apply in a similar manner as described above for step 504 of FIG. 13, or in any other manner.

[0153] At step 1708, the controller 140 applies appropriate control signals to the facets 124 to set the facets 124 to the determined “on” or “off” state.

[0154] At step 1710, the controller 140 activates the identified light sources to output the image.

[0155] At step 1712, the controller 140 provides the determined warps to be applied to the image to the SLM 304 prior to providing the image to the LOE 114. The image is then projected by the facets 124 of the LOE 114 that are set to the “on” state onto a portion of the pupil 182, and the process returns to step 1700 and continues for each frame of the image. In this way, changes in the position of the pupil 182 are taken into account, the effects of changes in accommodation of the eye 180 on image clarity are mitigated, the corresponding light sources are activated, and appropriate warps are applied to generate an image with as little distortion as possible.

[0156] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0157] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements (if any) in the claims that follow, are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The disclosed embodiments of the present application are presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the form of the application to the precise forms disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments were chosen and described in order to best explain the principles of the application and its practical application, and to thereby enable others skilled in the art to best utilize the application in various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. An apparatus comprising at least one processor, said at least one processor being configured to: A light source is selected from a plurality of light sources based at least in part on the position of the pupil of the eye relative to the eye-tracking box, and the selected light source is configured to illuminate the portion of the eye-tracking box corresponding to the position of the pupil with a light beam; Activate the selected light source to illuminate the portion of the eye-tracking box; The deformation to be applied to the beam is determined at least in part based on the selected light source; And at least in part based on the determined deformation, to induce a modification of the beam. Determining the deformation to be applied to the beam, at least in part based on the selected light source, includes: determining corrections to the beam for aberrations caused by the collimator.

2. The apparatus according to claim 1, wherein, The selected light source is configured to illuminate only the portion of the eye-tracking box that corresponds to a portion of the pupil using the light beam.

3. The apparatus according to claim 1, wherein, Inducing a modification of the light beam at least in part based on the determined deformation includes: causing a spatial light modulator to modify the light beam at least in part based on the determined deformation.

4. The apparatus according to claim 1, wherein: The beam illuminates the portion of the eye-tracking box based at least in part on a plurality of elements of the coupling device, at least one of the reflectivity and intensity of each of the plurality of elements being selectively adjustable between at least two states; as well as The at least one processor is further configured to: The target state of a given element among the plurality of elements is determined at least in part based on the selected light source; and This enables the given element to be set to the target state.

5. The apparatus according to claim 1, wherein: The selected light source is the first light source; and The at least one processor is further configured to select a second light source from the plurality of light sources, the second light source being configured to illuminate the portion of the eye-tracking box.

6. A method for controlling an optical system (100), wherein, The optical system includes: a plurality of light sources (306); a light-guiding optical element (114) including a coupling device (120) configured to guide light beams received from the plurality of light sources toward an eye-tracking box (128) of the optical system; a spatial light modulator (304) disposed between the plurality of light sources (306) and the light-guiding optical element (114); and a controller (140). The method includes the following steps performed by the controller (140): A light source is selected from a plurality of light sources based at least in part on the position of the pupil (182) of the eye (180) relative to the eye-tracking box, and the selected light source is configured to illuminate the portion of the eye-tracking box corresponding to the position of the pupil with a light beam; Activate the selected light source to illuminate the portion of the eye-tracking box; The deformation to be applied to the light beam is determined at least in part based on the selected light source; and the modification of the light beam is caused at least in part based on the determined deformation. The method is characterized by: Determining the deformation to be applied to the beam, at least in part, based on the selected light source, includes: determining corrections to the beam for aberrations caused by the collimator.

7. The method according to claim 6, wherein, The selected light source is configured to illuminate only the portion of the eye-tracking box corresponding to a portion of the pupil using the light beam.

8. The method according to claim 6, wherein, Inducing a modification of the light beam at least in part based on the determined deformation includes causing the spatial light modulator (304) to modify the light beam at least in part based on the determined deformation.

9. The method according to claim 6, wherein: The beam illuminates the portion of the eye-tracking box based at least in part on a plurality of elements of the coupling device (120), at least one of the reflectivity and intensity of each of the plurality of elements being selectively adjustable between at least two states; as well as The method further includes: The target state of a given element among the plurality of elements is determined at least in part based on the selected light source; as well as This enables the given element to be set to the target state.

10. The method according to claim 6, wherein: The selected light source is the first light source; and The method further includes selecting a second light source from the plurality of light sources, the second light source being configured to illuminate the portion of the eye-tracking box.

11. An optical system (100), comprising: Multiple light sources (306); A light-guiding optical element (114) includes a coupling device (120) configured to guide light beams received from the plurality of light sources toward an eye-tracking box (128) of the optical system; Controller (140), which is configured to: A light source is selected from the plurality of light sources based at least in part on the position of the pupil (182) of the eye (180) relative to the eye-tracking box. The selected light source is configured to emit a beam (126) that, when guided by the coupling device, illuminates the portion of the eye-tracking box corresponding to the position of the pupil. Activate the selected light source to illuminate the portion of the eye-tracking box; as well as The deformation to be applied to the light beam is determined at least in part based on the selected light source, wherein determining the deformation to be applied to the light beam at least in part based on the selected light source includes: determining corrections to the light beam for aberrations caused by the collimator; and A spatial light modulator (304) is disposed between the plurality of light sources (306) and the light guide optical element (114), the spatial light modulator (304) being configured to modify the light beam at least in part based on a determined deformation.

12. The optical system of claim 11, further comprising an eye-tracking system (160) configured to determine the position of the pupil, wherein, The controller (140) is configured to determine the portion of the eye-tracking box corresponding to the position of the pupil determined by the eye-tracking system.

13. The optical system of claim 11, further comprising an optical device configured to direct a light beam from a selected light source toward the spatial light modulator, the optical device comprising: First lens; Second lens; A first microlens array is disposed between the first lens and the second lens; as well as The second microlens array is disposed between the first microlens array and the second lens.

14. The optical system according to claim 13, wherein, The plurality of light sources (608) are located in the focal plane of the first lens, the second microlens array (612) is located in the focal plane of the first microlens array (610), and the spatial light modulator (304) is located in the focal plane of the second lens.

15. The optical system according to claim 11, wherein, The coupling device (120) includes a plurality of elements, and the controller (140) is configured to selectively adjust at least one of the reflectivity and intensity of each of the elements between at least two states.

Images