Actuated pupil steering for head mounted display systems - Patents.com

Abstract

Some embodiments herein are directed to a head-mounted virtual retinal display (VRD) system with actuated reflective pupil steering. The display system includes a projection system for generating image content and an actuated reflective optical architecture, which may be part of an optical combiner that reflects light from the projection system into a user's eye. The display system is configured to track the position of the user's eye and actuate the reflective optical architecture to change the direction of the reflected light so that it is directed into the user's eye. The VRDs described herein may be highly efficient and may have improved size, weight, and brightness such that these VRDs are capable of all-day, everyday use.

Description

[Technical field]

[0001] The present disclosure relates to display systems, and more particularly to augmented reality display systems. [Background technology]

[0002] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or portions thereof are presented to the user as if they were real or could be perceived as real. Virtual reality, or "VR", scenarios usually involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, and augmented reality, or "AR", scenarios usually involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user. Mixed reality, or "MR", scenarios are a type of AR scenario, usually involving virtual objects that are integrated into and responsive to the natural world. For example, in MR scenarios, AR image content may be occluded by objects in the real world or may be perceived as otherwise interacting with objects in the real world.

[0003] 1, an augmented reality scene 10 is depicted in which a user of the AR technology views a real-world park-like setting 20 featuring people, trees, buildings, and a concrete platform 30 in the background. In addition to these items, the user of the AR technology also perceives that he is "seeing" "virtual content," such as a robotic figure 40 standing on the real-world platform 30, a cartoon-like avatar character 50 that appears to be an anthropomorphic bumblebee, even though these elements 40, 50 do not exist in the real world. Due to the complexity of the human visual perception system, it is a challenge to create AR technology that facilitates a pleasant, natural-feeling, and rich presentation of virtual image elements, among other virtual or real-world image elements.

[0004] The systems, devices, and methods disclosed herein address various challenges related to AR and VR technologies. Summary of the Invention [Means for solving the problem]

[0005] Some aspects include a head mounted display system comprising: a frame; an image light projector supported by the frame; a diffractive reflector supported by the frame and disposed in front of the image light projector, the diffractive reflector configured to receive image light projected by the image light projector and to reflect the image light into a user's eye when the display system is held on the user, the diffractive reflector comprising a plurality of diffractive layers; and an actuator supported by the frame and configured to change the relative orientation of the diffractive layers, wherein different relative orientations of the diffractive layers are configured to reflect light to different associated eye positions.

[0006] In some embodiments, the actuator is a mechanical actuator configured to change the relative physical positions of the diffractive layers. In some embodiments, the actuator is configured to rotate one or more of the diffractive layers. In some embodiments, the diffractive reflector comprises a Risley prism-style stack of diffractive layers. In some embodiments, the actuator is configured to laterally shift one or more of the diffractive layers. In some embodiments, the diffractive layer comprises a volume phase hologram. In some embodiments, the diffractive layer comprises a reflective geometric phase lens. In some embodiments, the image light projector has a fixed orientation relative to the diffractive reflector.

[0007] In some embodiments, the head mounted display system further comprises an eye tracking system for determining a position of a pupil of the user's eye, and the actuator is configured to cause different orientations corresponding to different pupil locations based on the determined position of the pupil.

[0008] Some aspects include an augmented reality display device comprising: an image light projector; a reflective diffractive combiner configured to receive a projection from the image light projector; the diffractive combiner configured to combine world light with the projection light from the image light projector and to direct the projection light to form an image at a variable, selectable, steered exit pupil location; the diffractive combiner comprises a switchable beam steering unit configured to diffract the projection light to form an image at the steered exit pupil location; the beam steering unit comprises two or more diffraction gratings configured to receive the projection light; and an actuation mechanism; at least one of the two or more diffraction gratings is coupled to the actuation mechanism; the actuation mechanism is configured to cause a change in orientation of at least one of the two or more diffraction gratings, the different orientations having different associated diffraction angles with respect to the incident projection light, the different associated diffraction angles corresponding to different steered exit pupil locations.

[0009] In some embodiments, the image light projector comprises one or more microelectromechanical systems scanning mirrors and a laser light source. In some embodiments, the image light projector comprises a variable focus lens. In some embodiments, the augmented reality display system further comprises an eye tracking assembly for tracking a position of an eye pupil of a user of the display system. In some embodiments, the eye tracking assembly includes an off-axis camera. In some embodiments, the two or more diffraction gratings form a Risley prism type steering mechanism, and the diffraction combiner further comprises a collimating holographic optical element behind the two or more diffraction gratings and a holographic optical element focusing lens in front of the two or more diffraction gratings.

[0010] In some embodiments, a first of the two or more diffraction gratings and a second of the two or more diffraction gratings are coupled to an actuation mechanism, where actuation of the first diffraction grating causes rotation of the first diffraction grating from a first grating orientation to a second grating orientation and actuation of the second diffraction grating causes rotation of the second diffraction grating from a third grating orientation to a fourth grating orientation, where the rotation of the first diffraction grating changes a diffraction angle of the projected light from the first diffraction grating and where the rotation of the second diffraction grating changes a diffraction angle of the projected light from the second diffraction grating. In some embodiments, the two or more diffraction gratings include transmission gratings and the projected light is diffracted in only a single pass through the two or more diffraction gratings of the beam steering unit.

[0011] In some embodiments, the two or more diffraction gratings include a transmitting volume phase holographic grating and a reflecting volume phase holographic grating, the transmitting volume phase holographic grating being an optical collimator, and the reflecting volume phase holographic grating being configured to focus the projection light to the steered exit pupil. In some embodiments, the reflecting volume phase holographic grating is mechanically coupled to an actuation mechanism. In some embodiments, the actuation mechanism is configured to laterally translate the reflecting volume phase holographic grating relative to the transmitting volume phase holographic grating, the lateral translation of the at least one reflecting volume phase holographic grating changing the diffraction angle of the projection light upon reflection from the at least one reflecting volume phase holographic grating. In some embodiments, the ratio of the movement distance of the steered exit pupil to the lateral translation distance of the at least one reflecting volume phase holographic grating is 1:1.

[0012] In some embodiments, the two or more gratings include at least two secondary phase gratings. In some embodiments, the at least two secondary phase gratings include a transmission volume phase holographic grating and a reflection volume phase holographic grating. In some embodiments, the reflection volume phase holographic grating is coupled to an actuation mechanism. In some embodiments, the actuation mechanism is configured to laterally translate the reflection volume phase holographic grating relative to the transmission volume phase holographic grating, the lateral translation causing the secondary phase grating to generate a linear grating function, the linear grating function changing a diffraction angle of the projected light upon reflection from the at least one reflection volume phase holographic grating. In some embodiments, a ratio of a travel distance of the steered exit pupil to a distance of lateral translation of the at least one reflection volume phase holographic grating is greater than 1:1. In some embodiments, the transmission volume phase holographic grating diverges the projected light at a predetermined focal length. In some embodiments, the actuation mechanism is configured to tilt the reflective volume phase holographic grating toward or away from the at least one transmissive volume phase holographic grating, with different tilts corresponding to different diffraction angles of the projected light upon reflection from the reflective volume phase holographic grating.

[0013] In some embodiments, the beam steering unit further comprises a polarization switch, a quarter wave plate in front of the polarization switch, and a reflector in front of the quarter wave plate. In some embodiments, the beam steering unit further comprises a first polarizing lens after the polarization switch and a volume phase holographic grating after the first polarizing lens. In some embodiments, the two or more gratings comprise a second polarizing lens in front of the polarization switch and a third polarizing lens in front of the second polarizing lens.

[0014] In some embodiments, the augmented reality display system further comprises a world compensator configured to counterbalance the redirection of world light by the beam steering unit. In some embodiments, the world compensator comprises substantially identical optical components as the beam steering unit, and the world compensator is driven in opposition to the beam steering unit such that the optical effects of the beam steering unit are cancelled by the world compensator. In some embodiments, the diffraction grating comprises a liquid crystal polarization grating, and the diffraction combiner further comprises a first holographic optical element configured to focus right-handed circularly polarized light and a second holographic optical element configured to collimate left-handed circularly polarized light, the second holographic optical element being behind the beam steering unit and the first holographic optical element being behind the second holographic optical element. In some embodiments, the beam steering unit further comprises a quarter-wave plate in front of the diffraction grating. In some embodiments, the actuation mechanism comprises an electrical switch for selectively applying a voltage to at least one of the one or more liquid crystal polarization gratings. In some embodiments, the applied voltage changes the grating structure of the at least one liquid crystal polarization grating, and the change in the grating structure changes the diffraction angle of the projected light upon transmission through the at least one liquid crystal polarization grating. In some embodiments, the liquid crystal polarization grating comprises a Bragg selective liquid crystal polarization grating. In some embodiments, the number of addressable steering positions of the steered exit pupil is equal to 2n, where n is the number of diffraction gratings in the beam steering unit.

[0015] Some aspects include a head mounted display system comprising: a frame; an image light projector supported by the frame; and a diffractive reflector supported by the frame and disposed in front of the image light projector, the diffractive reflector configured to receive image light projected by the image light projector and to reflect the image light into a user's eye when the display system is held on a user, the diffractive reflector comprising a plurality of diffractive layers comprising liquid crystal polarization gratings, a polarization switch configured to change the polarization of light propagating through the diffractive layers, different states of the switch configured to reflect light to different associated eye positions.

[0016] In some embodiments, the head mounted display system further comprises a dichroic mirror, and the diffractive layer comprises a first stack and a second stack of liquid crystal polarization gratings separated by the dichroic mirror.

[0017] In some embodiments, the head mounted display system further comprises a dichroic mirror and a polarizer, where the dichroic mirror, the polarizer, and the liquid crystal polarization grating form a stack, where the polarizer is in front of the dichroic mirror, and where the dichroic mirror is in front of the liquid crystal polarization grating. In some embodiments, the head mounted display system further comprises an eye tracking system for determining a position of a pupil of an eye of the user, where the actuator is configured to cause different orientations corresponding to different pupil locations based on the determined position of the pupil.

[0018] Some aspects include a method for displaying virtual content to a user from a head mounted display, the method including tracking a position of a pupil of an eye of a user, projecting image light from an image light projector onto a diffractive reflector comprising a plurality of diffractive layers, and varying a reflection angle of the image light from the diffractive reflector to track the position of the pupil. In some embodiments, varying the angle includes varying a relative orientation of the diffractive layers, where different relative orientations correspond to different pupil positions, and varying the relative orientation directs the light to the tracked position of the pupil. In some embodiments, varying the relative orientation includes rotating one or more of the diffractive layers relative to one or more others of the diffractive layers. In some embodiments, varying the relative orientation includes laterally shifting one or more of the diffractive layers relative to one or more others of the diffractive layers. In some embodiments, the diffractive layer includes a liquid crystal polarization grating and a polarization switch, and varying the reflection angle includes changing a polarization of light propagating through the polarization switch. [Brief description of the drawings]

[0019] These drawings are provided to illustrate example embodiments and are not intended to limit the scope of the disclosure.

[0020] [Figure 1] Illustrates a user's view of an augmented reality (AR) device. [Diagram 2] 1 illustrates an example of a wearable display system. [Diagram 3] 1 illustrates a conventional display system that simulates a three-dimensional image for a user. [Figure 4] 1 illustrates an aspect of an approach for simulating a three-dimensional image using multiple depth planes. [Figure 5A] 1 illustrates the relationship between the radius of curvature and the radius of focus. [Figure 5B] 1 illustrates the relationship between the radius of curvature and the radius of focus. [Figure 5C]1 illustrates the relationship between the radius of curvature and the radius of focus. [Figure 6] 1 illustrates an example of an optical system architecture for displaying virtual content by steering image light into a user's eye. [Figure 7] 1 illustrates an example of an optical architecture for an electronically actuated assembly for steering image light into a user's eye. [Figure 8A] 8 shows an example of the optical function of a holographic optical element for the optical architecture of FIG. 7. [Figure 8B] 8 shows an example of the optical function of a holographic optical element for the optical architecture of FIG. 7. [Figure 9] 8 illustrates an example of an electronically actuated beam steering unit (BSU) for the optical architecture of FIG. 7. [Figure 10] 1 shows an example of a liquid crystal polarization grating (LCPG) in the off and on states. [Figure 11] 1 shows an example of a diffraction angle θ that can be achieved using an LCPG BSU with n=3 LCPGs. [Figure 12] 1 illustrates another example of an optical architecture for an electronically actuated assembly for steering image light into a user's eye. [Figure 13] 1 shows another example of an electronically operated BSU. [Figure 14A] 13 shows an example of the optical function of the holographic optical element of the optical architecture of FIG. 12. [Figure 14B] 13 shows an example of the optical function of the holographic optical element of the optical architecture of FIG. 12. [Figure 15] 1 shows an example of a diffraction angle θ that can be achieved using a Bragg-selective LCPG BSU with n=3 LCPGs. [Figure 16] 1 shows an example of an optical architecture having a Risley prism type BSU. [Figure 17] FIG. 16 shows an example of the structure of a Risley prism type BSU. [Figure 18]FIG. 16 shows another example of the structure of a Risley prism type BSU. [Figure 19] 1 shows an example of the optical functionality of a single-pass Risley prism-based BSU. [Figure 20] 1 shows an example of the optical functionality of a double-pass Risley prism type BSU. [Figure 21] 1 shows an example of an optical architecture with a shift plate BSU. [Figure 22] 22A-22C show various exemplary pupil steering positions that can be achieved using the shift plate BSU of FIG. 21. [Diagram 23] 22 shows an example of a volume phase holographic grating structure of the shift plate BSU of FIG. 21. [Figure 24] 13 shows another example of an optical architecture with a shift plate BSU. [Diagram 25] 25 shows examples of pupil steering positions that can be achieved using the shift plate combiner of FIG. 24. [Figure 26] 25 shows an example of a volume phase holographic grating structure of the shift plate BSU of FIG. 24. [Figure 27A] 13 shows examples of pupil steering positions that can be achieved using a tilt and shift shift plate BSU. [Figure 27B] 13 shows examples of pupil steering positions that can be achieved using a tilt and shift shift plate BSU. [Figure 28] 1 illustrates an example optical architecture for a double pass powered shift plate BSU showing an example of light propagation through the optical structure. [Figure 29] 30 illustrates an example of a steered pupil position for the double pass shift plate combiner of FIG. 29. [Diagram 30] 1 illustrates an example flowchart of a process for generating a steered exit pupil image of a display device. [Diagram 31] 13 shows an example flowchart of another aspect of a process for generating a steered exit pupil image on a display device. [Diagram 32]FIG. 1 shows an example diagram of a parallel workflow process for generating a steered exit pupil image on a display device. [Diagram 33] 1 illustrates an example flowchart of a process for displaying a steered exit pupil image. [Diagram 34] 13 shows an example of a flowchart of another process for displaying a steered exit pupil image. [Diagram 35] 1 shows an example of a mechanically actuated BSU. [Diagram 36] 1 illustrates an example of a mechanically actuated BSU having linear actuators for moving layers of the BSU in two dimensions. [Figure 37] 13 shows another example of a mechanically actuated BSU having linear actuators for moving layers of the BSU in two dimensions. [Figure 38] 1 shows a diagram of an example of the integration of an upstream variable focus lens structure to provide image content on different depth planes. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Some embodiments herein are directed to a head mounted display system for displaying virtual content by actively steering image light into the eyes of a user or viewer. The display system includes a light projector that outputs light encoded with image information (which may be referred to as image light) to a reflector having actuators that change the relative orientation of the structures or the state of a light switch in the reflector. These changes change the reflection angle of the incoming image light, thereby steering the image light into the viewer's eyes.

[0022] In some embodiments, the reflector may be a diffractive reflector having a stack of diffractive layers. In some embodiments, these diffractive layers may form a beam steering unit (BSU), where different relative orientations of the diffractive layers and / or of the diffractive features in the diffractive layers may provide different angles of reflection of the incident light. In some embodiments, actuation of the reflector may be achieved using a mechanical actuator that changes the relative physical positions of the diffractive layers. For example, the mechanical actuator may be physically coupled to one or more of the diffractive layers. The diffractive layers may be a Risley prism-style stack of diffractive layers, where one or more of the layers may be rotated to provide image light reflection in a desired direction. In some other embodiments, the diffractive layers may be shifted laterally to provide image light reflection in a desired direction.

[0023] Actuation of the diffractive layer may also be achieved using an electrical actuator (e.g., one or more electrical switches) that provides a current difference and / or a voltage difference to change the state of an optical switch between the liquid crystal gratings of the reflector. For example, a change in the optical switch may change the polarization of the light, which may change the direction of propagation of the light through the liquid crystal grating.

[0024] The display system may include an eye tracking system that tracks the position of the pupil of the viewer's eye to determine the direction in which the image light should be steered. The diffractive structure may then be actuated to provide a desired reflection angle for steering the image light into the viewer's eye. The display system disclosed herein with pupil steering may be referred to as a virtual retinal display (VRD) system. It will be appreciated that the display system disclosed herein may be an augmented reality display system that may provide a view of the surrounding environment and overlay virtual content on the view, or may simply provide virtual content without providing a view of the surrounding environment. In an augmented reality display system, the diffractive reflector may be part of an optical combiner to combine light from the surrounding environment with the image light so that the viewer sees both the surrounding environment and the virtual content provided by the image light.

[0025] Advantageously, the VRD system disclosed herein may provide various benefits. For example, energy efficiency may be increased. It will be appreciated that in order to provide image light to a viewer as the viewer's eyes move, many conventional displays display output image light over a wide area encompassing various possible positions for the viewer's eyes. This is undesirably inefficient since when the viewer's eyes are in a given position, only a portion of the image light reaches that eye, and light destined for other possible positions is "wasted" in that it is not received by the eye. By steering the image light specifically within the viewer's eyes, the image light does not need to be provided over a wide area, and the VRD described herein may be ten times more efficient than conventional techniques. This improved efficiency may be leveraged to provide a reduction in device size and weight, and may increase brightness, thereby improving the perceived image quality and facilitating all-day, everyday use of the head mounted display system.

[0026] In some embodiments, the VRD may render an image directly onto the retina of the eye. For example, in a raster scan approach, the retina may be divided into a number of discrete picture elements or pixels arranged in an array. The display's light beam is swept across the retina (e.g., one row at a time, left and right across the array, and top to bottom). As the beam sweeps across the retina, which may be analogized to a projector screen for receiving images, the beam intensity is varied to create a pattern according to, for example, a frame buffer or refresh buffer that holds the corresponding intensity value of each pixel. If the beam is swept along a row or scan line, at the end of each row of pixels, the beam returns to the opposite side of the array with a horizontal retrace to begin displaying the next scan line. Similarly, after the beam has completed a single scan of the array, displaying an entire frame, the beam may return to the top of the array with a vertical retrace to begin scanning the next frame. Preferably, the scan is fast enough that the user perceives all of the pixels as present at the same time, thereby perceiving the entire image as a whole. In some embodiments, the display system may utilize a laser and the scanned beam may be a laser beam, which may advantageously create a small exit pupil at the eye. In some embodiments, the steered exit pupil of the display system herein may be understood to be where the image light directed by the actuated reflector enters the eye, with reflection of the image light at different angles by the actuated reflector providing different steered exit pupils by directing the image light to different locations. In some embodiments, the exit pupil may be advantageously small (e.g., 2 mm or less in diameter).

[0027] It will be appreciated that the diffractive layer that forms the diffractive reflector may take a variety of forms, in some embodiments the diffractive layer may include a volume phase hologram that changes relative orientation to steer the exit pupil to the user's eye.

[0028] In some embodiments, the VRDs herein may comprise one or more microelectromechanical systems (MEMS) devices for mechanical actuation of the structures of these VRDs. For example, an image light projector may include a MEMS mirror for directing light onto a diffractive reflector. Such MEMS may be useful for increasing the range of angles at which light reflects off the diffractive reflector, thereby increasing the number of possible exit pupil locations available to the display system.

[0029] Reference is now made to the drawings, wherein like reference numerals refer to like features throughout.

[0030] Exemplary Display System FIG. 2 illustrates an example of a wearable display system 80. The display system 80 includes a display 62 and various mechanical and electronic modules and systems that support the functionality of the display 62. The display 62 may be coupled to a frame 64 that is wearable by a user or viewer 60 of the display system and configured to position the display 62 in front of the user's 60 eyes. The display 62 may be considered eyewear in some embodiments. In some embodiments, a speaker 66 is coupled to the frame 64 and positioned adjacent to the ear canal of the user 60 (although not shown, another speaker may optionally be positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). The display system may also include one or more microphones 67 or other devices for detecting sound. In some embodiments, the microphone may be configured to allow a user to provide input or commands to system 80 (e.g., voice menu commands, natural language queries, etc.) and / or enable voice communication with others (e.g., with other users of similar display systems).

[0031] 2, the display 62 is operatively coupled by a communications link 68, such as by a wired lead or wireless connection, to a local data processing module 70, which may be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 60 (e.g., in a backpack-style configuration, a belt-connected style configuration), etc. The local processing and data module 70 may include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which may be utilized to assist in processing, caching, and storing data. The data may include a) data captured from sensors (which may be operatively coupled to the frame 64 or otherwise attached to the user 60), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyros, and / or other sensors disclosed herein, and / or b) data obtained and / or processed using a remote processing module 72 and / or a remote data repository 74 (including data related to virtual content), as the case may be, for passage to the display 62 after such processing or retrieval. The local processing and data module 70 may be operatively coupled to the remote processing module 72 and the remote data repository 74 by communications links 76, 78, such as via wired or wireless communication links, such that the remote modules 72, 74 are operatively coupled to each other and available as resources to the local processing and data module 70. In some embodiments, the local processing and data module 70 may include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyro.In some other embodiments, one or more of these sensors may be mounted to the frame 64 or may be stand-alone structures that communicate with a local processing and data module 70 via a wired or wireless communication path.

[0032] With continued reference to FIG. 2, in some embodiments, the remote processing module 72 may comprise one or more processors configured to analyze and process the data and / or image information. In some embodiments, the remote data repository 74 comprises a digital data storage facility, which may be available over the Internet or other networking configurations in a “cloud” resource configuration. In some embodiments, the remote data repository 74 may include one or more remote servers that provide information, such as information for generating augmented reality content, to the local processing and data module 70 and / or the remote processing module 72. In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing for fully autonomous use from the remote modules.

[0033] Referring now to FIG. 3, the perception of an image as "three-dimensional" or "3D" can be achieved by providing a slightly different image representation to each eye of a viewer. FIG. 3 shows a conventional display system for simulating a three-dimensional image to a user. Two distinct images 5, 7 for each eye 4, 6 are output to the user. The images 5, 7 are spaced apart from the eyes 4, 6 by a distance 10 along an optical axis or Z-axis parallel to the viewer's line of sight. The images 5, 7 are flat and the eyes 4, 6 may focus on the images by assuming a single accommodation state. Such a system relies on the human visual system to combine the images 5, 7 to provide a perception of depth and / or scale of the combined image.

[0034] However, it will be appreciated that the human visual system is more complex and providing a realistic perception of depth is more of a challenge. For example, many viewers of conventional "3D" display systems find such systems uncomfortable or may not perceive any sense of depth at all. Without being limited to theory, a viewer may perceive an object as being "three-dimensional" due to a combination of convergence and accommodation. Vergence of the two eyes relative to one another (i.e., rotation of the eyes such that the pupils move toward or away from one another, thereby converging and fixing the line of sight of the eyes on an object) is closely linked to the focusing of the eye lens and pupil (i.e., "accommodation"). Under normal conditions, changing the focus of the eye lens, i.e., accommodating the eye to change focus from one object to another at a different distance, together with pupil dilation or contraction, automatically changes the convergence of the eyes to the same distance under a relationship known as the "accommodative convergence reflex." Similarly, changes in congruence trigger matching changes in accommodation of the lens shape and pupil size under normal conditions. As described herein, many stereoscopic or "3D" display systems display a scene using a slightly different presentation (i.e., a slightly different image) for each eye, thereby causing the human visual system to perceive a three-dimensional perspective. However, such systems, among other things, merely provide different presentations of the scene, with the eyes viewing all image information in a single accommodation state, countering the "congruence of accommodation" and making them uncomfortable for many users. Display systems that provide a better match between accommodation and congruence may produce a more realistic and comfortable simulation of three-dimensional images.

[0035] FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 4, objects at various distances from the eyes 4, 6 on the z-axis are accommodated by the eyes 4, 6 to focus on these objects. The eyes (4 and 6) adopt a particular state of accommodation that focuses on objects at different distances along the z-axis. As a result, a particular state of accommodation may be associated with a particular one of the depth planes 14, which has an associated focal length such that an object or part of an object in the particular depth plane is in focus when the eye is in an accommodation state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing different image presentations to each of the eyes 4, 6, and further corresponding to each of the depth planes. Although shown separately for clarity of illustration, the fields of view of the eyes 4, 6 may overlap, for example, as the distance along the z-axis increases. Additionally, although shown flat for ease of illustration, it will be understood that the contour of the depth plane may be curved in physical space such that any feature in the depth plane is in focus for the eye in a particular state of accommodation.

[0036] The distance between an object and the eye 4 or 6 may also change the amount of divergence of light from the object as viewed by that eye. Figures 5A-5C show the relationship between distance and divergence of light rays. The distances between the object and the eye 4 are represented in descending order of distance: R1, R2, and R3. As shown in Figures 5A-5C, the shorter the distance to the object, the more divergent the light rays are. The longer the distance, the more collimated the light rays are. In other words, the bright field generated at a point (object or part of an object) may have a spherical wavefront curvature, which is a function of how far the point is from the user's eye. The curvature increases as the distance between the object and the eye 4 decreases. As a result, the degree of divergence of the light rays is different at different depth planes, and the degree of divergence increases as the distance between the depth plane and the viewer's eye 4 decreases. 5A-5C, only a single eye 4 is shown for clarity of illustration, however, it will be appreciated that discussion regarding eye 4 may apply to both eyes 4 and 6 of the viewer.

[0037] Without being limited by theory, it is believed that the human eye can typically interpret a finite number of depth planes to provide depth perception. As a result, a highly believable simulation of perceived depth can be achieved by providing the eye with different image presentations corresponding to each of these limited number of depth planes. The different presentations may be focused differently by the viewer's eyes, thereby helping to provide depth cues to the user based on the ocular accommodation required to focus on different image features for a scene located on different depth planes and / or based on observing that different image features on different depth planes are out of focus. As discussed below, the display systems and optical architectures described herein may be compatible with upstream variable focus lenses, which may facilitate accommodation-congruence matching.

[0038] In addition to accommodation-congruence mismatch, optical efficiency is a significant issue for current wearable display technology. Diffractive waveguides have been used due to their combination of eyebox size, overall volume, field of view, and image quality, but they can be inefficient in that typically only a few percent of the light injected into the waveguide is transmitted to the user's eye. This inefficiency can result from both diffraction losses and, as discussed herein, the fact that in most waveguide systems, the waveguide distributes the projected light over a large eyebox area. This inefficiency can adversely affect power consumption and display brightness, which in turn can require the display device to have a large battery, while also causing undesirable heat dissipation and low overall display brightness.

[0039] Compact and lightweight "all-day, every day" wearable displays may require higher efficiency than is readily available from current existing technologies. Various challenges are associated with achieving highly efficient optical systems and devices for use in wearable display technology. One strategy for increasing the efficiency of wearable displays is to output light with a small exit pupil, rather than distributing the light over a large eyebox. However, the image formed by such devices is only projected to a small area, so that the image may be lost or degraded when the user's eye moves. Thus, some optical systems and devices that utilize a small exit pupil use a movable mirror to direct light from the projection system onto another mirror, which then directs the light into the user's eye.

[0040] However, currently existing pupil steering methods and systems may be inadequate for continuous use devices due to lack of one or more of volume / bulk requirements, efficiency, exit pupil steering range, image quality over the steering range, color fidelity, quality of see-through vision, power consumption, and required eyeglass form factor. Furthermore, the power consumption, speed, accuracy, efficiency, brightness, and form factor of existing pupil steering devices may not be satisfactory for routine everyday use. Thus, existing head mounted display systems, both waveguide and pupil steering, may be inadequate for various applications, particularly for all-day, everyday wearable display device use.

[0041] Some embodiments herein are directed to head-mounted or near-eye virtual retinal display (VRD) systems with actuated pupil steering that may improve upon various deficiencies of existing systems. The VRD systems described herein may include optical architectures that may be highly efficient, providing improved size, weight, and brightness. Thus, the VRDs described herein may be used in optical systems and devices capable of all-day, everyday use.

[0042] Some embodiments described herein are directed to highly efficient optical architectures for see-through wearable displays. In some embodiments, instead of creating a large eyebox using a waveguide or conventional pupil expansion optics as discussed herein, the optical architecture may comprise a system that creates a relatively small exit pupil (e.g., less than the size of the eye's pupil, including 2 mm or less) at the eye that is actively steered to match the eye's position. Low etendue, which may be understood to describe the mismatch between the size of the exit pupil of the display device and the size of the entrance pupil of the viewer's eye, and high diffraction efficiency, such as from a volume phase hologram (VPH), may lead to very low optical losses from the laser light source to the exit pupil. Furthermore, in some embodiments, the exit pupil of the systems and devices herein is smaller than the eye's pupil and can be steered to follow the user's eye, so that losses associated with a large eyebox may be substantially reduced or eliminated. Advantageously, most of the light generated by the display light source may be transferred to the user's retina, rather than distributed over a larger area.

[0043] 6 illustrates an example of an optical system architecture for displaying virtual content by steering image light into a user's eye. The optical system architecture 600 may be a VRD architecture and may include an image light projector 602 for outputting image light (i.e., light encoded with image content, e.g., to provide a desired color and / or brightness to a pixel). The image light projector 602 may include a light source 604 for generating the image light. In some embodiments, the image light projector 602 may include a modulator for varying the intensity and / or color of the light beam output by the image light projector 602 to encode the output light with image information. In some other embodiments, the intensity and / or color of the output light may be controlled by directly controlling the light source 604.

[0044] In some embodiments, the image light projector 602 may be a scanning display that varies the light output based on the location and / or direction of the light output edge of the scanning display. In this way, as the output finishes scanning across an area, different display elements (e.g., pixels) may be formed and light is output at specific locations that correspond to desired pixels for the application. In some embodiments, scanning may be accomplished using a scanning mirror 606 that moves (e.g., rotates along one or more axes) to direct the light output from the light source 604 in a desired direction for light scanning.

[0045] In some embodiments, the image light projector 602 may be a scanning display in the form of a laser scanning projector that outputs laser light. For example, the image light projector 602 may include a laser light source 604 that directs light to a MEMS mirror 606. It will be appreciated that the laser scanning projector 602 may include one or more additional lasers, MEMS, mirrors, galvanometer scanners, collimators, upstream variable focus components, or other optical components 609, 611. It will be appreciated that the optical systems 609, 611 may be projection optics configured to modify the wavefront divergence of the output light to provide different adjustment stimuli for locating objects at different depth planes, and / or may be variable focus lens elements. The laser scanning projector 602 may include one laser light source for monochromatic projection, or multiple light sources that output different wavelengths of light for different component colors to form a full-color image (e.g., three light sources for RGB (red, green, and blue) full-color projection). Additionally, laser scanning projector 602 may include a controller and digital-to-analog converter (DAC) for converting digital control signals into analog signals that control scanning components in laser scanning projector 602. In some embodiments, one or more processors, such as processing modules 70 and / or 72 of display system 62, may include a controller for laser scanning projector 602.

[0046] As discussed herein, in some embodiments, the MEMS 606 may comprise an oscillating scanning mirror for scanning and redirecting light along one or more axes to a diffractive reflector 608. As discussed herein, scanning may be used to render an image on a user's retina.

[0047] It will be understood that MEMS may refer to an integrated device or system that combines mechanical and electrical components. MEMS may be fabricated using integrated circuit (IC) batch processing techniques and may be in the micrometer or millimeter scale size range. MEMS may comprise mechanical microstructures, microsensors, microactuators, and microelectronics, each integrated on the same substrate. Micromechanical components may be fabricated by manipulation of silicon and other substrates using micromachining processes known in the art. Processes such as bulk and surface micromachining, as well as high aspect ratio micromachining (HARM), selectively remove portions of silicon or add additional structural layers to form mechanical and electromechanical components. MEMS have the ability to sense, control, and act on a microscale, which may produce effects on a macroscale. The small size, low cost, low power consumption, mechanical durability, high precision, and low cost batch processing of MEMS provide various advantages for use in display devices.

[0048] In some embodiments, the MEMS described herein may include a scanning mirror projector, which is a type of micromirror actuator for dynamic light modulation. In some embodiments, the scanning mirror projector may include a micro-sized motor for moving one or more mirrors in one or more dimensions. A laser beam directed at one or more mirrors may be precisely deflected and steered by the scanning mirror, which may be translated and / or rotated by the motor, to reach a target point. Scanning mirror projectors may be advantageous because they operate in an ultra-low power environment, offer exceptional repeatability and reliability, and may provide rapid optical beam scanning in one or more axes.

[0049] A MEMS scanning mirror as used herein may include, for example, a cylindrical, rectangular, or square micromirror adapted to move and deflect light over time. The micromirror may be connected to a fixed part by a torsion arm and may tilt and oscillate along one or more axes. Different actuation principles may be used, including, for example, electrostatic, thermal, electromagnetic, or piezoelectric. The MEMS scanning mirror may include a single micromirror configured to oscillate along one or more oscillation axes for scanning light along either or both of the horizontal and vertical axes. Alternatively, the MEMS device may include a first MEMS scanning mirror configured to oscillate along a first oscillation axis and a second MEMS scanning mirror configured to oscillate along a second oscillation axis. In some embodiments, in a two-mirror arrangement, the first and second MEMS scanning mirrors may be precisely positioned such that the oscillation axes are orthogonal.

[0050] 6, the redirected light may include one or more light rays 607. If architecture 600 is part of an augmented reality display system, a diffractive reflector 608 may be an optical combiner and may be provided to redirect light rays 607 to a steered exit pupil 610 while transmitting world light 612. Preferably, the location of the steered exit pupil 610 matches the location of the pupil of eye 614. Although one eye and architecture may be shown in various figures herein, it will be understood that a viewer may have another eye and the display system may be provided with another architecture, such as architecture 600, to provide image light to the other eye.

[0051] The diffractive reflector 608 may include one or more pupil steering mechanisms along with additional optical elements as described in more detail herein. For example, the diffractive reflector 608 may include one or more VPH gratings or liquid crystal gratings and functional beam steering optics to steer the exit pupil 610 in the visual plane of the eye 614 of the user 60 to match the position of the eye's entrance pupil. In some embodiments, the architecture 600 may be part of a virtual reality display system that is not a transmitted light to the eye 614. In such embodiments, the diffractive reflector 608 may be understood to be simply a reflective optics for beam steering.

[0052] It will be appreciated that a VPH may include an optical grating that includes a periodic phase structure for diffracting different wavelengths of light from a common input path into output paths at different angles. A VPH grating is formed in a layer of a transparent material, such as dichromated gelatin, sealed between two layers of clear glass or fused silica. The phase of incident light is modulated as it passes through an optically thick film that has a periodic differential hardness or refractive index. A VPH grating differs from a conventional grating in that the depth of the surface relief pattern modulates the phase of the incident light. However, similar to a conventional reflective grating, the spatial frequency of the periodic structure determines the spectral dispersion, or angular separation of the wavelength components, in the diffracted light. The average refractive index, the refractive index differential or modulation, and the thickness of the film determine the efficiency, polarization, and bandwidth performance characteristics of the VPH. A VPH grating may be advantageous in that the optical structure of the VPH is durable, cleanable, customizable, stable, and has excellent optical properties. A VPH grating may include a reflective or transmissive VPH grating.

[0053] Unlike conventional gratings, VPH gratings offer enhanced strength, robustness, and environmental durability because the grating structure is protected by an external glass and / or surface coating. VPH gratings may be combined with other optical elements to enable additional or enhanced optical properties. Classical geometric optical elements, such as lenses in VR and AR head-mounted displays (HMDs), cannot meet the weight or size requirements of displays that require a corrective eyeglass / sunglasses form factor. However, the geometry of VPH gratings enables compact, high-performance system geometries that cannot be realized using conventional reflective gratings. Conventional gratings typically work in a Littrow geometry where the diffracted / dispersed light is reflected back towards the incident light source. In contrast, due to the transmission geometry of VPH gratings, the incident and diffracted light may be located on opposite sides of the grating plane. As a result, VPH gratings allow the incident and diffracted light to be operated by independent optics and components, such as detectors, fibers, or MEMS. Furthermore, imaging optics may be placed directly adjacent to the grating, which may minimize system size and improve performance in some applications. In some embodiments, the use of polarization insensitive VPH grating structures may eliminate the need for polarization splitting system configurations, further reducing parts count and device size. VPH gratings can be fabricated with line frequencies from less than 300 lines / mm to more than 6000 lines / mm. VPH gratings may also be referred to herein as holographic optical elements (HOEs).

[0054] Continuing to refer to FIG. 6, the exemplary VRD architecture 600 may have various advantages over existing architectures in that the architecture may be extremely compact and power efficient, and may project images that are substantially free of coherent artifacts. The VRD architecture 600 may also create a very small exit pupil for a waveguide-based display, such that a majority of the projected light enters the user's pupil. In some embodiments, substantially all of the projected light from the exemplary optical architecture 600 enters the user's pupil. In contrast, other display systems, such as waveguide-based display systems, may create a large eyebox, where the power is widely distributed and only a small portion of the projected light enters the user's pupil.

[0055] Additionally, the exemplary optical architecture 600 may be used to generate a compact display system into a device with a large field of view (FOV), high brightness, high efficiency, and controlled continuous contra-accommodative matching via upstream variable focus. The FOV indicates the vertical and horizontal angles subtended by the user's eyes at which the diffractive reflector 608 displays an image relative to the appearance. A wide FOV allows for a wider view, while a narrower FOV would result in a limited view near the center of the user's field of view.

[0056] Advantageously, as discussed herein, the exemplary VRD architecture 600 may be smaller than comparable devices because the VRD architecture 600 does not require pupil expansion optics and may be folded into a wearable eyewear device. In addition, the efficiency of the exemplary VRD architecture 600 may be relatively high because only a small amount of light may be wasted (i.e., only a small amount of light generated by the light source 604 does not reach the pupil of the eye 614) due to light steering and the small size of the exit pupil 614 created by the architecture. Finally, high brightness is achieved because most of the light generated by the image light projector 602 is actively directed through the small exit pupil 610 to the active eyebox, rather than being distributed throughout a larger eyebox.

[0057] In some embodiments, the exemplary VRD architecture 600 can generate a steered exit pupil 610 that accounts for rotation and lateral movement of the user's eyes over a portion or the entire FOV. In some embodiments, the VRD architecture 600 can account for both horizontal and vertical rotation. In some embodiments, the VRD architecture 600 can account for both horizontal and vertical rotation over the entire range of physically possible rotation of the eye. For example, in some embodiments, the VRD architecture 600 can be capable of accounting for vertical eye rotation of approximately 50 degrees (e.g., 25 degrees downward and 25 degrees upward from the horizontal line of sight). In some embodiments, the VRD architecture 600 may be configured to rotate at any angle between about 1 degree, about 1.5 degrees, about 2 degrees, about 2.5 degrees, about 3 degrees, about 3.5 degrees, about 4 degrees, about 4.5 degrees, about 5 degrees, about 5.5 degrees, about 6 degrees, about 6.5 degrees, about 7 degrees, about 7.5 degrees, about 8 degrees, about 8.5 degrees, about 9 degrees, about 9.5 degrees, about 10 degrees, about 10.5 degrees, about 11 degrees, about 11.5 degrees, about 12 degrees, about 12.5 degrees, about 13 degrees, about 13.5 degrees, about 14 degrees, about 14.5 degrees, about 15 degrees, about 15.5 degrees, about 16 degrees, about 16.5 degrees, about It may be possible to consider upward vertical eye rotation of 17 degrees, about 17.5 degrees, about 18 degrees, about 18.5 degrees, about 19 degrees, about 19.5 degrees, about 20 degrees, about 20.5 degrees, about 21 degrees, about 21.5 degrees, about 22 degrees, about 22.5 degrees, about 23 degrees, about 23.5 degrees, about 24 degrees, about 24.5 degrees, about 25 degrees, about 25.5 degrees, about 26 degrees, about 26.5 degrees, about 27 degrees, about 27.5 degrees, about 28 degrees, about 28.5 degrees, about 29 degrees, about 29.5 degrees, about 30 degrees, or values ​​between any of the aforementioned values.

[0058] In some embodiments, the VRD architecture 600 may be configured to rotate at any angle between about 1 degree, about 1.5 degrees, about 2 degrees, about 2.5 degrees, about 3 degrees, about 3.5 degrees, about 4 degrees, about 4.5 degrees, about 5 degrees, about 5.5 degrees, about 6 degrees, about 6.5 degrees, about 7 degrees, about 7.5 degrees, about 8 degrees, about 8.5 degrees, about 9 degrees, about 9.5 degrees, about 10 degrees, about 10.5 degrees, about 11 degrees, about 11.5 degrees, about 12 degrees, about 12.5 degrees, about 13 degrees, about 13.5 degrees, about 14 degrees, about 14.5 degrees, about 15 degrees, about 15.5 degrees, about 16 degrees, about 16.5 degrees, about It may be possible to consider a downward vertical eye rotation of 17 degrees, about 17.5 degrees, about 18 degrees, about 18.5 degrees, about 19 degrees, about 19.5 degrees, about 20 degrees, about 20.5 degrees, about 21 degrees, about 21.5 degrees, about 22 degrees, about 22.5 degrees, about 23 degrees, about 23.5 degrees, about 24 degrees, about 24.5 degrees, about 25 degrees, about 25.5 degrees, about 26 degrees, about 26.5 degrees, about 27 degrees, about 27.5 degrees, about 28 degrees, about 28.5 degrees, about 29 degrees, about 29.5 degrees, about 30 degrees, or values ​​between any of the aforementioned values.

[0059] In some embodiments, the VRD architecture 600 may be capable of accounting for horizontal eye rotation of approximately 70 degrees (eg, 35 degrees to the left and 35 degrees to the right from the vertical central gaze). In some embodiments, the VRD architecture 600 may have a lattice angle of about 1 degree, about 1.5 degrees, about 2 degrees, about 2.5 degrees, about 3 degrees, about 3.5 degrees, about 4 degrees, about 4.5 degrees, about 5 degrees, about 5.5 degrees, about 6 degrees, about 6.5 degrees, about 7 degrees, about 7.5 degrees, about 8 degrees, about 8.5 degrees, about 9 degrees, about 9.5 degrees, about 10 degrees, about 10.5 degrees, about 11 degrees, about 11.5 degrees, about 12 degrees, about 12.5 degrees, about 13 degrees, about 13.5 degrees, about 14 degrees, about 14.5 degrees, about 15 degrees, about 15.5 degrees, about 16 degrees, about 16.5 degrees, about 17 degrees, about 17.5 degrees, about 18 degrees, about 18.5 degrees, about 19 degrees, about It may be possible to consider a left horizontal eye rotation of 19.5 degrees, about 20 degrees, about 20.5 degrees, about 21 degrees, about 21.5 degrees, about 22 degrees, about 22.5 degrees, about 23 degrees, about 23.5 degrees, about 24 degrees, about 24.5 degrees, about 25 degrees, about 25.5 degrees, about 26 degrees, about 26.5 degrees, about 27 degrees, about 27.5 degrees, about 28 degrees, about 28.5 degrees, about 29 degrees, about 29.5 degrees, about 30 degrees, about 30.5 degrees, about 31 degrees, about 31.5 degrees, about 32 degrees, about 32.5 degrees, about 33 degrees, about 33.5 degrees, about 34 degrees, about 34.5 degrees, about 35 degrees, or a value between any of the aforementioned values.

[0060] In some embodiments, the VRD architecture 600 may have a lattice angle of about 1 degree, about 1.5 degrees, about 2 degrees, about 2.5 degrees, about 3 degrees, about 3.5 degrees, about 4 degrees, about 4.5 degrees, about 5 degrees, about 5.5 degrees, about 6 degrees, about 6.5 degrees, about 7 degrees, about 7.5 degrees, about 8 degrees, about 8.5 degrees, about 9 degrees, about 9.5 degrees, about 10 degrees, about 10.5 degrees, about 11 degrees, about 11.5 degrees, about 12 degrees, about 12.5 degrees, about 13 degrees, about 13.5 degrees, about 14 degrees, about 14.5 degrees, about 15 degrees, about 15.5 degrees, about 16 degrees, about 16.5 degrees, about 17 degrees, about 17.5 degrees, about 18 degrees, about 18.5 degrees, about 19 degrees, about It may be possible to consider right horizontal eye rotations of 19.5 degrees, about 20 degrees, about 20.5 degrees, about 21 degrees, about 21.5 degrees, about 22 degrees, about 22.5 degrees, about 23 degrees, about 23.5 degrees, about 24 degrees, about 24.5 degrees, about 25 degrees, about 25.5 degrees, about 26 degrees, about 26.5 degrees, about 27 degrees, about 27.5 degrees, about 28 degrees, about 28.5 degrees, about 29 degrees, about 29.5 degrees, about 30 degrees, about 30.5 degrees, about 31 degrees, about 31.5 degrees, about 32 degrees, about 32.5 degrees, about 33 degrees, about 33.5 degrees, about 34 degrees, about 34.5 degrees, about 35 degrees, or values ​​between any of the aforementioned values.

[0061] With continued reference to FIG. 6, in some embodiments, a camera assembly 82 (e.g., a digital camera including visible light and infrared cameras) may be provided to capture images of the eye to, for example, track the position of the eye's pupil. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 82 may include an image capture device and a light source for projecting light (e.g., near-infrared light) onto the eye, which may then be reflected by the eye and detected by the image capture device. In some embodiments, the camera assembly 82 may be mounted on the frame 64 (FIG. 2) and may be in electrical communication with processing modules 70 and / or 72 (FIG. 2) that may process image information from the camera assembly 82. In some embodiments, one camera assembly 82 may be utilized for each eye to monitor each eye separately.

[0062] The camera assembly 82 may be integrated into an inner facing imaging system that may track the movement of the user's eyes. The inner facing imaging system may track either one eye movement or both eye movements. The inner facing imaging system may be mounted to the frame 64 and may be in electrical communication with the processing module 70 or 72, which may process image information acquired by the inner facing imaging system to determine, for example, the pupil diameter or orientation of the user's 60 eyes, eye movement, or eye pose. The inner facing imaging system may include one or more cameras as part of the camera assembly 82. For example, at least one camera may be used to image each eye. Images acquired by the cameras may be used to determine the pupil size or eye pose of each eye separately, thereby enabling the presentation of image information to each eye to be dynamically adapted to that eye.

[0063] The camera assembly 82 may include, for example, a video-based eye tracking system including near-eye tracking (on-axis or off-axis), remote tracking, model-based tracking, or regression-based tracking, with, for example, a single camera input or multiple camera inputs. Near-eye tracking may be split into on-axis and off-axis configurations depending on the camera position viewing the eye. Off-axis arrangements may take up less space, possibly at the expense of uneven accuracy in gaze estimation.

[0064] In some embodiments, a display system incorporating the exemplary VRD architecture 600 may also need to account for the varying interpupillary distance (IPD) of users. IPD is an important design consideration for binocular viewing systems, including head-mounted displays, where the pupils of both eyes are desirably positioned within the exit pupils of the viewing system. IPD measurements may be used in the design of such systems to specify the range of lateral adjustment of the exit optics or eyepieces. IPD may also be used to describe the distance between the exit pupils of a binocular optical system. In some embodiments, the varying IPD between users may be accounted for during fitting of a display device to each user.

[0065] One potential challenge in utilizing the VRD architecture 600 is in creating a temporal visual effect and, relatedly, maintaining a realistic and comfortable user experience by continuously projecting a high-quality image to the retina, regardless of eye movement. To maintain high image quality for a user's eye with a small exit pupil, an accurate and fast user pupil sensing, exit pupil steering, and image rendering loop is preferably implemented. Therefore, appropriate pupil steering hardware and software can be employed to generate a satisfactory loop.

[0066] Some previous approaches to exit pupil steering involve passively enlarging the VRD's exit pupil using various pupil duplication methods. However, in some embodiments herein, the example VRD architecture 600 does not duplicate the pupil. Instead, the exit pupil 614 of the VRD herein can be dynamically steered to track the eye.

[0067] Several variations of exit pupil steering are described herein, including mechanical and non-mechanical pupil steering for VRDs. As discussed herein, the underlying VRD architecture may, for example, comprise an image light projector that projects light onto a diffractive reflector that is worn near the user's temple and reflects the projected light to the eye. In some embodiments, the exit pupil steering mechanism is integrated into the diffractive reflector and may involve a stack of diffractive layers, such as a combination of one or more VPHs, and / or a liquid crystal grating. The pupil steering mechanism described herein may also be equivalent to an upstream variable focus lens structure to provide a figurative accommodation stimulus, which may be beneficial for mitigating congruence-accommodation conflict (VAC). As discussed herein, VAC can be problematic in near-eye displays and difficult to address in waveguide-based displays. In contrast, a variable focus VRD system may utilize a small, low-power variable focus lens to provide varying levels of wavefront divergence for varying accommodation stimuli. Using the pupil steering mechanism herein in combination with a variable focus structure upstream of a diffractive combiner may advantageously provide an adjustable variable focus display.

[0068] Non-mechanical exit pupil steering mechanism Some of the exit pupil steering mechanisms disclosed herein may be non-mechanical in their operation. In some embodiments, non-mechanical beam steering may advantageously provide high efficiency, low wear, high speed, and / or low failure rate. A wide variety of approaches have been explored to achieve non-mechanical beam control, including microlens arrays, electro-optic prisms, and diffractive acousto-optic techniques. Unfortunately, these approaches may suffer from one or more of the following limitations: low throughput, scattering, small steering angles / apertures, and large physical size.

[0069] Some non-mechanical exit pupil steering mechanisms herein may include electronically actuated pupil steering mechanisms. Some of these pupil steering mechanisms may be based on the polarization-sensitive properties of liquid crystal polarization gratings (LCPGs). For example, some electronically actuated pupil steering mechanisms may include a combiner that includes one or more LCPGs combined with an electronically addressable polarization switch to actively steer one or more exit pupils. Devices incorporating electronic pupil steering may be adapted to operate at nearly any wavelength, including at least wavelengths from visible to infrared wavelengths.

[0070] An electronically actuated pupil steering mechanism may be used to build an efficient optical architecture for a see-through wearable display. As discussed herein, instead of creating a large eyebox using a waveguide or conventional pupil expansion optics, an optical system with an electronically actuated pupil steering mechanism may create a small exit pupil at the eye that is actively steered to match the eye's position. Stacking LCPG, other diffractive layers (such as VPH), and / or electronically addressable polarization switches may allow the construction of a transparent combiner with a modest amount of switchable exit pupil location.

[0071] In some embodiments, electronically actuated exit pupil steering may be used, for example with off-axis diffractive VRDs. The electronically actuated pupil steering mechanisms herein may also be equivalent to upstream variable focus structures to mitigate congruent-accommodative conflict (VAC).

[0072] In some embodiments, an electronically actuated exit pupil steering mechanism is integrated into the diffractive combiner and comprises a combination of layers of polarization selective and non-selective diffractive elements (including, for example, LCPG, HOE, and other volume phase gratings). Pupil steering can be achieved by placing a polarization switch between the polarization selective gratings to allow for different ordering of the gratings depending on whether the polarization switch is switched on or off. In some embodiments, the number of addressable steering positions can be increased to 2. n where n is the number of stacked switches.

[0073] LCPG with Polarization Switch Some optical combiners described herein may include an electronically actuated pupil steering mechanism that includes a non-Bragg selective LCPG. In some embodiments, the non-Bragg selective LCPG may be paired with a world light compensation unit to allow world light to pass through the combiner. The non-Bragg selective LCPG is generally opaque to world light. Thus, a combiner that includes a non-Bragg selective LCPG may require an additional LCPG stack to compensate for world light. The additional stack of LCPGs may constrain the number of layers that can be used for pupil steering if a lightweight and compact display device is desired. As a result, the number of layers (i.e., stacked switches) may be a limiting factor for a combiner that utilizes a non-Bragg selective LCPG for electronically actuated pupil steering, since increasing the number of layers reduces the transparency of the combiner.

[0074] FIG. 7 shows an example of an optical architecture 700 for an electronically actuated assembly for steering image light into a user's eye. In some embodiments, the optical architecture 700 can be used to form a diffractive reflector for use in a VRD display, such as the diffractive reflector 608 for the display system of FIG. 6. The optical architecture 700 comprises a first HOE 720, a second HOE 722, a beam steering unit (BSU) 724, a dichroic mirror 726, and a global compensator 728. In the embodiment shown, the polarized beam 607 projected from the image light projector 602 passes through the first polarization-selective HOE 720 unaffected and is collimated by the second polarization-selective HOE 722 towards the BSU 724. The BSU 724 selectively steers the light based on the polarization properties of the light's polarization and the on / off status of the LCPG in the BSU 724 in single-pass or double-pass mode. In the double-pass embodiment shown in Figure 7, the incoming light beam 607 from the second HOE 722 is steered by the BSU 724 at a first time to the dichroic mirror 726, which reflects the light beam 607 back to the BSU 724 toward the user's eye 716. After reflection from the dichroic mirror 726, the BSU 724 dynamically steers the light beam 607 at a second time back to the second HOE 722, through which the light beam 607 passes unaffected, and to the first HOE 720, which focuses the light to the steered exit pupil 610.

[0075] 8A and 8B show the optical functions of the first HOE 720 and the second HOE 722, respectively. The first HOE 720 may have an optical function including focusing the right-handed circularly polarized light 802 and leaving the left-handed circularly polarized light 804 unaffected. The second HOE 704 may have an optical function including collimating the left-handed circularly polarized light 804 and leaving the right-handed circularly polarized light 802 unaffected. Thus, in the illustrated embodiment of FIG. 7, the light beam 607 is left-handed circularly polarized when projected from the image light projector 602. The left-handed circularly polarized beam 607 is unaffected by the first HOE and is collimated by the second HOE 722. However, after passing through the BSU 724 ( FIG. 7 ) on a first pass away from the eye 716 and a second pass toward the eye 716, the light beams 607 may switch their polarization to be right-handed circularly polarized at the time the second pass passes through the first HOE 720 and the second HOE 722 toward the eye 716. Thus, during the second pass, the light beams 607 are unaffected by the second HOE 722 and may be focused by the first HOE 720 to the exit pupil 610.

[0076] In some embodiments, the optical functions of the first HOE 720 and the second HOE 722 may be interchangeable such that the first HOE 720 performs the above-mentioned optical functions of the second HOE 722, and vice versa. In the illustrated embodiment of FIG. 7, the light beam 607 is initially left-handed circularly polarized so as to be unaffected by the first HOE 720 and collimated by the second HOE 722. However, after reflecting from the dichroic mirror 726 and leaving the BSU 724 toward the eye 716, the light beam 607 may be right-handed circularly polarized so as to pass through the second HOE 722 unaffected and be focused by the first HOE 720 to the exit pupil 610.

[0077] FIG. 9 illustrates an example of an electronically actuated beam steering unit (BSU) for the optical architecture of FIG. 7. The BSU 724 may comprise one or more LCPGs 902, each forming a diffractive layer. In some embodiments, each of the LCPGs may include a switchable nematic LCPG having a continuous in-plane bend splay pattern. Without being limited by theory, the theoretical diffraction efficiency of each of the LCPGs may be expressed as η=cos 2 (Γ / 2), η ±1 =(1 / 2)(1±S'3)sin 2 (Γ / 2), where η m is the mth order diffraction efficiency, Γ=2πΔnd / λ is the retardation of the LCPG layer, λ is the wavelength of the incident light, and S′3=S3 / S0 is the normalized Stokes parameter corresponding to the ellipticity of the incident light. In some embodiments, the incident light is circularly polarized (S ’ 3=±1) and the retardation of the LC layer is half-wave (Δnd=λ / 2), it can be diffracted into only one of the first orders.

[0078] When the input light beam 607 is circularly polarized, the first HOE 720 and / or the second HOE 722 ensure that the input light to the LCPGs of the BSU 724 is in one of two orthogonal (left-handed / right-handed) circular polarization states. Depending on the polarization properties of the polarization, each LCPG diffracts the beam into one of the first orders (+θ or −θ) and reverses its polarization properties. LCPG (V>>V threshold ), the grating profile is effectively erased (i.e., Δn=0) and the incident beam passes through the LCPG, preserving the polarization state of the incident beam. The diffraction angle is given by the classical grating equation θ out =sin -1 (mλ / Λ+sinθ in ), where the order m depends on the incident polarization and Λ is the grating period.

[0079] FIG. 10 shows an example of a liquid crystal polarization grating (LCPG) in the off and on states. Without being limited by theory, LCPGs can be understood to be similar to traditional diffraction gratings that utilize a periodic structure to steer light. However, LCPGs use polarization modulation instead of pure phase or amplitude modulation, and in some cases can provide high first-order efficiency, exceeding 99.8%. Liquid crystals are well suited for polarization grating media because they are composed of highly anisotropic molecules whose orientation can be manipulated by an electric field. The high efficiency and compact size make LCPGs suitable for coherent beam steering and active image scanning systems. When an active LCPG is switched to the on state (i.e., a voltage above a certain voltage threshold is applied between the electrodes), the LCPG's lattice structure disappears, resulting in a third, unpolarized and unpolarized light path. Because each element in an LCPG stack can be switched off, added to, or subtracted from the net deflection, a relatively small stack can provide a large set of deflection angles, allowing a wide range of angles in two dimensions to be achieved with a small number of stack elements.

[0080] 9, the BSU 724 may comprise one or more LCPGs 902. The example BSU 724 of FIG. 9 comprises three LCPGs 902 (n=3). However, in some embodiments, the BSU 724 may comprise 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 24, or 25 LCPGs 902. However, in some embodiments, the number of LCPGs 902 in the BSU 724 may be lower to increase transparency and reduce the size of a display device utilizing the optical architecture 700, since each LCPG layer may introduce some optical loss.

[0081] Without being limited by theory, when the spatial frequency of grating i+1 of LCPG 902 is different from the spatial frequency of grating i of LCPG 902, the number of possible diffraction angles θ is 2 nwhere n is the number of LCPGs 902 in the BSU 724. FIG. 11 shows an example diffraction angle θ that can be achieved using a BSU 724 with n=3, and the spatial frequency σ i+1 =2σ i As shown, a beam steering unit with n=3 LCPGs has 2 3 = 8 equally spaced diffraction angles can be achieved. In the double pass configuration, the diffraction angle is doubled to 2θ.

[0082] Referring again to FIG. 9, in some embodiments, the BSU 724 may also comprise one or more wave plates 904. The wave plates 904 may be composed of a birefringent material, such as quartz, mica, or plastic, that has a different refractive index depending on the polarization of the incident light. The behavior of the wave plate may depend on the thickness of the crystal, the wavelength of the light, and the variation in the refractive index. By appropriately choosing the relationship between these parameters, it may be possible to introduce a controlled phase shift between the two polarization components of the light wave, thereby modifying the polarization of the light wave. In some embodiments, the one or more wave plates 904 include at least one quarter-wave plate that converts linearly polarized light to circularly polarized light and vice versa. Additionally, in some embodiments, the BSU 724 may comprise one or more VPH gratings 906.

[0083] The optical function of the BSU 724, in combination with the first HOE 720, the second HOE 722, and the dichroic mirror 726, may be to direct the light beam 607 from the image light projector 602 to a specific steered exit pupil 610 to generate an image on the retina of the eye 716. One or more of the LCPGs 902 in the BSU 724 may be continuously switched to an on or off state to direct the light beam 607 in two or more directions based on the detected movement of the eye 716. n The diffraction angle can be steered to different angles.

[0084] 7, the optical architecture 700 may also include a world compensator 728, which may comprise an optical structure identical to the BSU 724 but driven inversely. The world compensator may function by performing the inverse optical function of the BSU 724 on the world light 612, such that it has minimal or no net optical effect on the world light 612. As mentioned above, non-Bragg selective LCPGs generally include a grating that is opaque to the world light, so the world compensator 728 may be utilized to allow the world light 612 to reach the eye 716.

[0085] Bragg-selective LCPG Some electronically actuated pupil steering mechanisms may comprise Bragg-selective LCPGs. In some embodiments, the Bragg-selective LCPGs may be more transparent than comparable non-Bragg-selective LCPGs. Thus, in some embodiments, combiners and optical architectures comprising Bragg-selective LCPGs may comprise more layers than combiners and optical architectures comprising non-Bragg-selective LCPGs while maintaining a similar or even higher level of transparency. Increasing the number of layers in the optical architecture may allow for an increase in the amount of exit pupil steering positions that may be achieved by the combiner, which may increase the precision, accuracy, and / or speed of pupil steering. Furthermore, because Bragg-selective LCPGs may be transparent, world light may pass through combiners utilizing these LCPGs without the need for additional world adaptive optics. Thus, in some embodiments, combiners with Bragg-selective LCPGs may provide an enhanced user experience and image projection compared to display systems employing other pupil steering mechanisms.

[0086] FIG. 12 shows another example of an optical architecture 1200 for an electronically actuated assembly for steering image light into a user's eye. In some embodiments, the optical architecture 1200 can be used to form a diffractive reflector for use in a VRD display, such as the diffractive reflector 608 of FIG. 6. The optical architecture 1200 comprises a first HOE 720, a second HOE 722, a beam steering unit (BSU) 724, a dichroic mirror 726, and a polarizer 728. In the embodiment shown, the polarized beam 607 projected from the image light projector 602 passes through the first polarization-selective HOE 720 unaffected and is collimated by the second polarization-selective HOE 722 towards the BSU 724. The BSU 724 selectively steers the light based on the polarization properties of the light's polarization and the on / off status of the Bragg-selective LCPG in the BSU 724 in single-pass or double-pass mode. 12, the incoming light beam 607 from the second HOE 722 is steered at a first time by the BSU 724 to the dichroic mirror 726, which reflects the light beam 607 back to the BSU 724 toward the user's eye 716. After reflection from the dichroic mirror 726, the BSU 724 dynamically steers the light beam 607 at a second time back to the second HOE 722, through which the light beam 607 passes unaffected, and to the first HOE 720, which focuses the light to the steered exit pupil 610.

[0087] 14A and 14B show examples of optical functions of the holographic optical elements of the optical architecture of FIG. 12. The first HOE 720 may have an optical function including focusing the right-handed circularly polarized incident light 802 and leaving the left-handed circularly polarized incident light 804 unaffected. The second HOE 704 may have an optical function including collimating the left-handed circularly polarized light 804 and leaving the right-handed circularly polarized light 802 unaffected. Thus, in the illustrated embodiment of FIG. 12, the light beam 607 is left-handed circularly polarized when projected from the image light projector 602. The left-handed circularly polarized beam 607 is unaffected by the first HOE and is collimated by the second HOE 722. However, after passing through the BSU 724 on a first pass away from the eye 716 and a second pass towards the eye 716, the polarization of the light beam 607 may be switched to be right-handed circularly polarized at the time the second pass passes through the first HOE 720 and the second HOE 722 towards the eye 716. Thus, during the second pass, the light beam 607 is unaffected by the second HOE 722 and may be focused by the first HOE 720 to the exit pupil 610.

[0088] In some embodiments, the optical functions of the first HOE 720 and the second HOE 722 may be interchangeable such that the first HOE 720 performs the above-mentioned optical functions of the second HOE 722, and vice versa. In the illustrated embodiment of FIG. 12, the light beam 607 is initially left-handed circularly polarized so as to be unaffected by the first HOE 720 and collimated by the second HOE 722. However, after reflecting from the dichroic mirror 726 and leaving the BSU 724 toward the eye 716, the light beam 607 may be right-handed circularly polarized so as to pass through the second HOE 722 unaffected and be focused by the first HOE 720 to the exit pupil 610.

[0089] FIG. 13 illustrates an exemplary BSU 724 according to some embodiments herein. The BSU 724 may comprise one or more Bragg-selective LCPGs 1302. Each of the LCPGs may include a switchable nematic Bragg-selective LCPG 1302. In an exemplary BSU 724 using Bragg-selective LCPGs 1302, all of the one or more Bragg-selective LCPGs 1302 are configured to act on light having the same polarization. For example, in some embodiments, each of the one or more Bragg-selective LCPGs 1302 may act on light having right-handed circular polarization. In some other embodiments, each of the one or more Bragg-selective LCPGs 1302 may act on light having left-handed circular polarization. As a result of this configuration of the Bragg-selective LCPGs 1302, world light having a polarization opposite to that on which the Bragg-selective LCPGs act may pass through the BSU 724 unaffected. Thus, in this configuration, a world compensator may not be necessary.

[0090] Without being limited by theory, when the spatial frequency of grating i+1 of the Bragg-selective LCPG 1302 is different from the spatial frequency of grating i of the Bragg-selective LCPG 1302, the number of possible diffraction angles θ is reduced to 2 n where n is the number of LCPGs 1302 in the BSU 724. FIG. 15 shows an example diffraction angle θ that can be achieved using a BSU 724 with n=3 Bragg-selective LCPGs 1302, and the spatial frequency σ i+1 =2σ i As shown, the beam steering unit with n=3 LCPG1302 has 2 3 = 8 spaced diffraction angles can be achieved. In a double pass configuration, the diffraction angle can be doubled to 2θ.

[0091] Mechanically actuated pupil steering mechanism In some embodiments, the head mounted display system may include a diffractive reflector with an exit pupil steering mechanism that is mechanically actuated to provide lateral translation, rotation, and / or tilt of the optical element. In some embodiments, the diffractive reflector and associated image light projector may provide a single exit pupil. The diffractive reflector may include multiple diffractive layers, at least one of which may be mechanically actuated to steer the exit pupil of the display system to the location of the eye's pupil. In some other embodiments, the diffractive reflector may include two or more exit pupils. As discussed herein, the diffractive reflector may be part of an optical combiner for an augmented reality display system, etc.

[0092] Risley prism type diffractive reflector With reference to FIG. 16, in some embodiments, the mechanically actuated exit pupil steering mechanism may comprise a Risley prism-based architecture 1600. In some embodiments, the Risley prism-based architecture 1600 may comprise multiple diffractive layers, e.g., four diffractive layers, where two of the diffractive layers are mechanically rotated to shift the exit pupil 612 of the display device. In some embodiments, the Risley prism-based architecture 1600 is structured based on the principle of a Risley prism, also called a Risley prism pair, which includes a pair of wedge prisms that may be used for beam steering. For example, rotating one wedge relative to the other wedge changes the direction of the light beam. When the wedges are angled in the same direction, the angle of the refracted beam increases. When the wedges are rotated to an angle in the opposite direction, they cancel each other out, allowing the beam to pass straight through. It will be appreciated that diffractive layers function in a similar manner. In some embodiments, the optical architecture 1600 may be used to form a diffractive reflector for use in a VRD display, such as the diffractive reflector 608 for the display system of FIG.

[0093] In some embodiments, the Risley prism architecture 1600 may comprise four diffractive layers, with each of the two middle layers rotating relative to the other, and the two outer layers remaining stationary. The Risley prism architecture 1600 may comprise reflective or transmissive diffractive layers. In some embodiments, the outer layer 1620 closest to the incident light beam 607 from the image light projector 602 may comprise a first HOE. In some embodiments, the active layers (e.g., the two middle layers) of the Risley prism architecture 1600 may provide the desired optical functionality based on manipulation by an actuator (not shown). In some embodiments, the active layers may act on the collimated light from the first HOE 1620. In some embodiments, the outer layer 1622 furthest from the incident light may comprise a second HOE. In some embodiments, the second HOE of layer 1622 may comprise a focusing lens or a focusing plate. In some embodiments, the Risley prism architecture may be part of a combiner. In some embodiments, the layers of the Risley prism architecture 1600 may be angled with respect to each other to reduce artifacts or other image defects in the projected image from the image light projector 602.

[0094] In some embodiments, the Risley prism-based architecture 1600 comprises four or more layers of diffractive structures. In some embodiments, the layers may comprise VPHs, liquid crystal-based phase gratings, or a combination of both. Depending on the optical architecture and device requirements, such as eyebox size, resolution, and FOV, among others, different types of gratings may be utilized. Depending on the optical architecture selected (e.g., single-pass, double-pass, reflective, transmissive), additional layers may be required to compensate for see-through transparency. For example, single-pass and double-pass architectures may be formed using either transmission or reflective gratings, or a combination of both.

[0095] 16, the architecture 1600 may include an optical combiner comprising an active Risley prism BSU 1624, a first diffractive layer 1620, and a second diffractive 1622. The Risley prism architecture 1600 may also include an image light projector 602 comprising a laser light source 604 that directs light to the MEMS 606. The image light projector 602 may include one or more additional lasers, MEMS, mirrors, galvanometer scanners, collimators, upstream variable focus components, or other optical components 609, 611.

[0096] In some embodiments, the first HOE 1620 may act as a collimator for the light beam 607 from the image light projector 602 so that the light beam 607 is directed to the BSU 1624. The BSU 1624 may be mechanically actuated to diffract the light beam 607 so that the light beam 607 is steered to the exit pupil 610 in response to the movement of the user's pupil 614. The BSU 1624 may be formed in a single pass configuration, where the light beam 607 is only diffracted when entering the BSU 1624 from the image light projector 602, the first diffractive layer 1620, and the eye side, or in a double pass configuration, where the light beam 607 is diffracted both in a first pass through the BSU 1624 and in a second pass through the BSU 1624 after interacting with the second diffractive layer 1622. In some embodiments, the second diffractive layer 1622 may act as a focusing lens to reflect the light beam 607 back to the BSU 1624 and toward the user's eye. In some embodiments, the light beam 607 is unaffected by the first diffractive layer 1620 on the second pass after returning from the BSU 1624 and the second diffractive layer 1622. In some embodiments, the optical architecture 1600 may include additional layers, such as additional diffractive layers (e.g., additional HOEs), and / or dichroic mirrors, among others.

[0097] FIG. 17 illustrates an example of the structure of the Risley prism-based BSU 1624 of FIG. 16. The exemplary BSU 1624 of FIG. 17 includes a transmissive Risley prism-based pupil steering mechanism. In the illustrated embodiment, the BSU 1624 includes at least two transmission gratings in a single-pass or double-pass configuration as discussed herein. The light beam 607 incident on the first transmission grating 1702 and the second transmission grating 1704 may be diffracted upon transmission through each of the gratings. In some embodiments, the diffraction angle may be changed by rotation of the first transmission grating 1702 and / or the second transmission grating 1704. For example, as shown, mechanical rotation of the first grating 1702 from a first orientation 1702A to a second orientation 1702B may change the diffraction angle of the light beam 607 upon transmission through the first grating 702. Similarly, mechanical rotation of the second grating 1704 from the first orientation 1704A to the second orientation 1704B may change the diffraction angle of the light beam 607 upon transmission through the second grating 1704. As a result of the change in diffraction angle caused by the rotation of the first transmission grating 1702 and / or the second transmission grating 1704, the location of the exit pupil 610 may be steered in at least two dimensions. In some embodiments, the plane of rotation of the first transmission grating 1702 and / or the second transmission grating 1704 is oriented perpendicular to the direction of light propagation through the first grating 1702 and the second grating 1704. In a single pass configuration, the light beam 607 will be diffracted only once by the BSU 1624, while in a double pass configuration, the light beam 607 may be diffracted on both the first and second passes through the BSU 1624.

[0098] FIG. 18 illustrates another example of the structure of the Risley prism-based BSU of FIG. 16. The BSU 1624 of FIG. 18 includes a reflective Risley prism-based pupil steering mechanism. In the illustrated embodiment, the BSU 1624 includes at least two reflective gratings in a single-pass or double-pass configuration as discussed herein. The light beam 607 incident on the first reflective grating 1802 and / or the second reflective grating 1804 may be diffracted upon reflection from the inner side or surface 1806 of each grating. In some embodiments, the inner side 1806 of the grating refers to the side of the first reflective grating 1802 and the second reflective grating that face each other. In some embodiments, the diffraction angle of the light beam 607 may be changed by rotation of the first reflective grating 1802 and / or the second transmission grating 1804. For example, as shown, mechanical rotation of the first grating 1802 from a first orientation 1802A to a second orientation 1802B can change the diffraction angle of the light beam 607 upon reflection from an internal side 1806 of the first grating 1802. Similarly, mechanical rotation of the second grating 1804 from a first orientation 1804A to a second orientation 1804B can change the diffraction angle of the light beam 607 upon reflection from an internal side 1806 of the second grating 1804. As a result of the change in diffraction angle caused by the rotation of the first reflective grating 1802 and / or the second reflective grating 1804, the location of the exit pupil 610 can be steered in at least two dimensions. In some embodiments, the plane of rotation is oriented perpendicular to the direction of light propagation through the first grating 1802 and the second grating 1804.

[0099] The net optical functions of Risley prism BSUs formed with transmission and reflection gratings can be similar depending on the orientation of the gratings. However, in a reflection grating BSU, light can be diffracted upon reflection from the facing surfaces of adjacent gratings rather than upon transmission of light through each grating, as shown in FIG.

[0100] FIG. 19 illustrates an example of the optical functionality of a single-pass Risley prism-based BSU 1624. The combiner 1900 may include a first HOE 1604 and a second HOE 1606 as discussed with respect to the optical architecture 1600 of FIG. 16. Additionally, the combiner 1900 may include a Risley prism BSU 1624 in a single-pass configuration. As shown, the light beam 607 may be diffracted once on the first transmission through the BSU 1624 from the first HOE 1620. However, the light 607 may be unaffected by the BSU 1624 on the second transmission through the BSU 1624 from the opposite side, returning from the second HOE 1622. It will be understood that the diffraction of the light beam 607 is illustrated in FIG. 19 as a net diffraction. As discussed in connection with Figures 17 and 18, the light beam 607 may be diffracted multiple times within the BSU 1624 and may be diffracted upon transmission through or reflection from one or more layers of a diffraction grating within the BSU 1624.

[0101] FIG. 20 illustrates an example of the optical functionality of a single-pass Risley prism-based BSU 1624. The diffractive reflector 2000 may comprise a first diffractive layer 1620 and a second diffractive layer 1622, as discussed in connection with the optical architecture 1600 of FIG. 16. As discussed herein, each of the diffractive layers 1620, 1622 may comprise an HOE. In some embodiments, the diffractive reflector 2000 may comprise a Risley prism BSU 1624 in a double-pass configuration. As shown, the light beam 607 may be diffracted once on a first transmission through the BSU 1624 from the first diffractive layer 1620, and diffracted again on a second transmission through the BSU 1624 from the opposite side, returning from the second diffractive layer 1624.

[0102] Both the transmissive and reflective Risley prism BSUs can be oriented in single-pass or double-pass configurations, depending on the optical requirements of a particular display device. As discussed in connection with Figures 17 and 18, the light beam 607 can be diffracted multiple times within the BSU 1624 and upon transmission through or reflection from one or more diffraction gratings within the BSU 1624.

[0103] Shift plate type operation In some embodiments, the diffractive reflector may include an exit pupil steering mechanism that includes a shift plate type BSU, in some embodiments, the shift plate type BSU may include two or more diffractive layers, where a lateral shift of one or more of the diffractive layers results in a shift of the exit pupil 610 ( FIG. 6 ) of the display system.

[0104] In some embodiments, the shift plate BSU may function by laterally displacing one or more optical elements (e.g., diffractive layers) of a diffractive reflector in the X and / or Y directions on a plane intersecting the user's line of sight. It will be understood that the X and Y directions may also be understood to refer to directions parallel to the horizontal and vertical axes in the user's FOV (such as directions along the horizontal and vertical axes of the user's field of view). In some embodiments, the shift plate BSU may require two or more layers, with one or more of the layers moving relative to the other layers. The layers may include VPH or reflective geometric phase lenses, or a combination of both. In some embodiments, an additional layer for world light correction is not required.

[0105] FIG. 21 illustrates an example optical architecture 2100 with a collimating shift plate combiner. The architecture 2100 may include an optical combiner with a BSU 2124 with a first grating layer 2120 and a second grating layer 2122. The optical architecture 2100 may also include an image light projector 602 with a light source 604 (e.g., a laser light source) that directs light to a MEMS 606 for scanning the light across a digitally windowed pupil steering range. It will be appreciated that the image light projector 602 may include one or more additional light sources (e.g., lasers), MEMS, mirrors, galvanometer scanners, collimators, upstream variable focus components, or other optical components 609, 611. In some embodiments, the optical architecture 2100 may be used to form a diffractive reflector for use in a VRD display, such as the diffractive reflector 608 for the display system of FIG. 6.

[0106] In some embodiments, the first grating layer 2120 includes a first HOE that functions by collimating all the chief rays of the light beam 607 at a steep angle to avoid crosstalk between the first grating layer 2120 and the second grating layer 2122. As used herein, crosstalk refers to the interference or crossing of the optical signals between the gratings. Collimating all the chief rays before focusing ensures that the second grating layer 2122 is insensitive to positional variations such that all the incident light beams 607 to the second grating layer 2122 are Bragg matched, thus forming a small exit pupil 610 that is highly efficiently focused. In some embodiments, the second grating layer 2122 includes a second HOE that functions to focus the collimated chief rays to the exit pupil 610. In some embodiments, mechanical translation of the second grating layer 2122 in the X and Y directions is used to steer the light beam 607 to the exit pupil 610 based on the detected location of the user's eye. In some embodiments, mechanical translation of the second grating layer 2122 in the X and Y directions moves the exit pupil 610 in a 1:1 relationship. In other words, the exit pupil 610 moves in the X and Y directions a distance equal to the corresponding movement of the second grating layer 2122 in the X and Y directions. In some embodiments, the first grating layer 2120 may include a transmissive HOE such that diffraction of the light beam 607 occurs upon transmission of the light beam 607 through the first grating layer 2120. In some embodiments, the second grating layer 2122 may include a reflective HOE such that diffraction of the light beam 607 occurs upon reflection of the light beam 607 from the second grating layer 2120.

[0107] In some embodiments, in a collimated shift plate arrangement, the displacement of the exit pupil 610 is proportional to the displacement of the second grating 2122 in a 1:1 ratio. As a result of the simple structure of the collimated shift plate arrangement, the collimated shift plate arrangement can be used to form a highly efficient and simple display system. In some embodiments, when using three-color projection (RGB), the pupil steering of the collimated shift plate can result in a total power efficiency of over 90%. Furthermore, as discussed herein, the collimated shift plate arrangement can avoid optical crosstalk, which reduces artifacts and improves the overall visual quality of the optical system. Finally, in some embodiments, the collimated shift plate arrangement can minimize or eliminate pupil aberrations that can lead to a poor user experience.

[0108] FIG. 22 shows various pupil steering positions that can be achieved using a collimated shift plate diffractive reflector such as the one shown in FIG. 21. In some embodiments, a combiner with a collimated shift plate beam steering mechanism may be capable of steering the exit pupil over an area of ​​about 8 mm×8 mm in the X and Y directions. However, the steering range may be expanded or limited based on the working range of the shift plate. FIG. 22 shows the chief rays of the light beam 607 from the image light projector 602 being steered to three different exemplary Y-axis exit pupil positions 610A, 610B, and 610C by the BSU 2124 with the first grating 2120 and the second movable grating 2122.

[0109] FIG. 23 shows an example of a volume phase holographic grating structure of the shift plate BSU of FIG. 21. The volume phase holographic grating structure may form a collimated shift plate diffractive reflector. In some embodiments, the first grating layer 2120 may include a transmission VPH. In some embodiments, as shown in FIG. 23, the light beam 607 may be split into two beams, one known as the object beam 2302 and the other as the reference beam 2304, for example by a beam splitter. A transmission VPH is a VPH in which the object beam 2302 and the reference beam 2304 are incident on the VPH from the same side, as shown in FIG. 23 on the first grating layer 2120. In some embodiments, the second grating layer 2122 may include a reflection VPH. In a reflection VPH, the object beam 2302 and the reference beam 2304 may be incident on the VPH from opposite sides, as shown in FIG. 23 on the second grating 2122. In some embodiments, the first grating 2120 and / or the second grating 2122 include additional optical elements, such as a focusing lens 2306, to provide a desired optical function.

[0110] Power shift plate type operation In some embodiments, the diffractive reflector may include an exit pupil steering mechanism that includes a powered shift plate type BSU, hi some embodiments, the powered shift plate type BSU may include two or more diffractive layers, where lateral shifting of one or more of the diffractive layers results in a shift of the exit pupil 610 ( FIG. 6 ) of the display device.

[0111] In some embodiments, the powered shift plate BSU may comprise at least two transmissive refractive plates or gratings in a modified Alvarez lens arrangement. In the conventional Alvarez lens, each grating has a flat surface and an opposing surface shaped with a two-dimensional cubic profile. The cubic surfaces of adjacent gratings are shaped inversely to each other so that the induced phase variation of each grating is cancelled by the adjacent grating when both gratings have their vertices on the optical axis. However, when the two plates undergo a relative lateral translation, a phase variation is introduced that is the derivative of the cubic surface profile, resulting in a quadratic phase profile (e.g., optical power). Thus, the relative translation in the X or Y direction induces cylindrical power independently in the orthogonal directions, whereas the combined movement may produce a circular, elliptical, or cylindrical phase profile.

[0112] In some embodiments, the powered shift plate BSU may comprise a variation of a standard Alvarez lens. For example, the BSU may comprise two or more diffractive lenses, each with a quadratic phase. When adjacent gratings are translated or shifted in the X or Y directions relative to each other, the gratings generate a linear grating function. This function may be used to provide two-dimensional beam steering. In some embodiments, in contrast to the collimated shift plate architecture, the powered shift plate BSU does not collimate the beam between adjacent gratings. However, the powered shift plate BSU may change the wavefront divergence of the light beam 607 (FIG. 6).

[0113] FIG. 24 illustrates an example optical architecture with a powered shift plate combiner. The architecture 2400 may include a diffractive reflector that may be part of the optical combiner, including a BSU 2424 with a first grating layer 2420 and a second grating layer 2422. The optical architecture 2400 may also include an image light projector 602 that includes a light source 604 (e.g., a laser light source) configured to direct image light to a MEMS 606 arranged to reflect the image light to a different BSU 2424. In some embodiments, the MEMS 606 may be movable, e.g., configured to rotate to scan the light across a digitally windowed pupil steering range. The image light projector 602 may include one or more additional lasers, MEMS, mirrors, galvanometer scanners, collimators, upstream variable focus, beam splitters, or other optical components 609, 611. In some embodiments, the light beam 607 from the image light projector 602 is Bragg matched with a ray emanating from the center of the pupil of the image light projector 607. In some embodiments, the optical architecture 2400 may be used to form a diffractive reflector for use in a VRD display, such as the diffractive reflector 608 for the display system of FIG.

[0114] In some embodiments, the first secondary phase grating 2420 may be stationary and may diverge all the chief rays of the light beam 607 with a particular focal length F, the chosen focal length F depending on the desired optical properties of the diffractive reflector. Diverging all the chief rays of the focal length F before focusing may reduce crosstalk and the second secondary phase grating 2422 may be insensitive to position variations such that all the incident light beams 607 to the second secondary phase grating 2422 are Bragg matched and thus form a small exit pupil 610 that is highly focused. In some embodiments, the second secondary phase grating 2422 focuses the divergent chief rays to the exit pupil 610. In some embodiments, mechanical translation of the second grating layer 2422 in the X and Y directions is used to steer the light beam 607 to the exit pupil 610 based on the detected location of the user's eye. In some embodiments, mechanical translation of the second secondary phase grating 2422 in the X and Y directions moves the exit pupil 610 in a 1:1 or 1:>1 relationship. For example, the exit pupil 610 may move an equal or greater distance in the X and Y directions as the corresponding movement of the second secondary phase grating 2422 in the X and Y directions. In some embodiments, the first secondary phase grating 2420 may include a transmissive VPH such that diffraction of the light beam 607 occurs upon transmission of the light beam 607 through the first secondary phase grating 2420. In some embodiments, the second grating 2422 may include a reflective VPH such that diffraction of the light beam 607 occurs upon reflection of the light beam 607 from the second secondary phase grating layer 2422.

[0115] In some embodiments, in a powered shift plate arrangement, the displacement of the exit pupil 610 is proportional to the displacement of the second secondary phase grating 2422 in a ratio of 1:1 or >1:1. In some embodiments, the ratio of the movement of the exit pupil 610 to the movement of the second secondary phase grating 2422 is about 1.00:1, about 1.05:1, about 1.10:1, about 1.15:1, about 1.20:1, about 1.25:1, about 1.30:1, about 1.35:1, about 1.40:1, about 1.45:1, about 1.50:1, about 1.55:1, about 1.60:1, about 1.65:1, about 1.70:1, about 1.75:1, about 1.80:1, about 1.85:1, about 1.90:1, The ratio may be about 1.95:1, about 2.00:1, about 2.05:1, about 2.10:1, about 2.15:1, about 2.20:1, about 2.25:1, about 2.30:1, about 2.35:1, about 2.40:1, about 2.45:1, about 2.50:1, about 2.55:1, about 2.60:1, about 2.65:1, about 2.70:1, about 2.75:1, about 2.80:1, about 2.85:1, about 2.90:1, about 2.95:1, about 3.00:1, or values ​​between any of the foregoing values.

[0116] Compared to a collimated shift plate BSU, a powered shift plate BSU may be advantageous due to a >1:1 ratio between the movement of the exit pupil 610 and the movement of the second secondary phase grating 2422. As a result, a relatively small movement of the second secondary phase grating 2422 may induce a movement of the exit pupil 610 a certain distance, whereas moving the exit pupil 610 the same distance with a collimated shift plate BSU would require a larger movement of the second grating 2144. Furthermore, as discussed herein, a powered shift plate arrangement may advantageously reduce optical crosstalk, thereby reducing artifacts and improving the overall visual quality of the optical system. The occurrence of pupil aberrations may be reduced or eliminated with a powered shift plate BSU. However, a powered shift plate BSU has a more complex structure than a collimated shift plate BSU and generally has a lower efficiency of about 60% when using three-color projection (RGB).

[0117] FIG. 25 shows various examples of pupil steering positions that can be achieved using a powered shift plate combiner such as the one shown in FIG. 24. In some embodiments, a combiner with a powered shift plate beam steering mechanism may be capable of steering the exit pupil over an area of ​​about 8 mm×8 mm in the X and Y directions. However, the steering range may be expanded or constrained based on the working range of the shift plate. FIG. 25 shows the chief rays of the light beam 607 from the image light projector 602 being steered to three different exemplary Y-axis exit pupil positions 610A, 610B, and 610C by the BSU 2424 with the first grating 2420 and the second movable grating 2422.

[0118] FIG. 26 shows an exemplary volume phase holographic grating structure of the shift plate BSU of FIG. 24. In some embodiments, the light beam 607 can be split into an object beam 2302 and a reference beam 2304, for example by a beam splitter, as shown in FIG. 23. In some embodiments, the first grating layer 2420 can include a transmission VPH. In some embodiments, the second grating layer 2422 can include a reflection VPH. In some embodiments, the first grating layer 2420 and / or the second grating layer 2422 include additional optical elements, such as a focusing lens 2306, to provide the desired optical function. As shown in FIG. 26, a small off-axis offset in the focus of the two object beams is necessary to avoid crosstalk on the second pass interaction with the first grating layer 2420 after reflection diffraction from the second grating layer 2422.

[0119] As an alternative to a powered shift plate where the second grating layer 2422 may only be shifted laterally to steer the exit pupil 610, FIGS. 27A and 27B show various exemplary pupil steering positions that may be achieved using a tilt-and-shift powered shift plate combiner, according to some embodiments herein. The tilt-and-shift powered shift plate architecture may have the same architecture as the optical architecture 2400 of FIG. 24. However, in a tilt-and-shift powered shift plate BSU, the second secondary phase grating layer 2422 may be actuated to shift or tilt laterally toward or away from the user's or viewer's eye. In some embodiments, lateral shift refers to movement of the grating layer in the X and Y dimensions of the FOV, while tilt refers to movement of the grating in the optical Z axis, where one portion of the grating tilts toward the user or viewer and another portion of the grating tilts away from the viewer's user.

[0120] FIG. 27A shows the pupil steering position of the tilt and shift powered shift plate BSU 2424. In some embodiments, the second secondary phase grating layer 2422 has a lateral shift range of about 1.0 mm in the X direction, or about 0.5 mm to the left and about 0.5 mm to the right, and a tilt shift range of about 4 degrees, or about 2 degrees, from the viewer and about 2 degrees toward the viewer. FIG. 27B shows the pupil steering position of the tilt and shift powered shift plate BSU 2424, where the second secondary phase grating layer 2422 has a tilt shift range of about 6 degrees, or about 3 degrees, from the viewer and about 3 degrees toward the viewer.

[0121] In some embodiments, a diffractive reflector with tilt and shift powered shift plate beam steering mechanism may be capable of steering the exit pupil over an area of ​​approximately 8 mm x 8 mm in the X and Y directions. However, the steering range may be expanded or reduced based on the shift and tilt actuation range of the shift plate. Figures 27A and 27B show the chief rays of the light beam 607 from the image light projector 602 being steered to different exemplary Y-axis exit pupil positions 610A, 610B, 610C, 610D, 610E, and 610F by the BSU 2424 with the first grating layer 2420 and the second movable grating layer 2422.

[0122] The exemplary powered shift plate architecture 2400 of FIG. 24 and the tilt-and-shift powered shift plate architectures of FIGS. 27A and 27B utilize two grating layers to steer the exit pupil 610, but may utilize more complex configurations of diffractive elements to reduce the frequency and / or distance of mechanical translation for a desired exit pupil transformation. An exemplary configuration is shown in FIG. 28 illustrating the optical structure of a double-pass powered shift plate combiner. The double-pass powered shift plate combiner may provide an optical architecture with low RGB crosstalk, low pupil aberrations, and an advantageous exit pupil steering mechanical translation ratio.

[0123] 28, an example optical architecture 2800 of a double pass powered shift plate BSU is shown illustrating an example of light propagation through an optical structure. The architecture 2800 may operate on a projected light beam 607 from an image light projector 602 (FIG. 6), which may include, for example, a laser light source 604 that directs light to a MEMS 606. The image light projector 602 may include one or more additional light sources (e.g., lasers), MEMS, mirrors, galvanometer scanners, collimators, upstream variable focus components, or other optical components 609, 611. In some embodiments, the light beam 607 may include left-handed circularly polarized light 802.

[0124] The architecture 2800 may include a first VPH layer 2802 and a first polarizing lens layer 2804, which function in combination to collimate the light beam 607 towards a polarization switch 2806. The polarization switch 2806 functions by selectively converting the light beam 607 to light having a character polarization or simply transmitting the light without changing the polarization. For example, the polarization switch 2806 may simply transmit light having left-handed circular polarization 802 or may convert the light to that having right-handed circular polarization 804. The left-handed circular polarization 802 or right-handed circular polarization 804 is then directed from the polarization switch 2806 to a second polarizing lens 2808 and a third polarizing lens 2810. In some embodiments, the second polarizing lens 2808 has the optical function of converging the collimated left-handed circular polarization 802 or diverging the collimated right-handed circular polarization 804.

[0125] Light from the second polarizing lens 2808 is directed to a third polarizing lens 2810 that is configured to be mechanically shifted and / or tilted relative to the second polarizing lens 2808 to dynamically steer the light to the exit pupil 610. The third polarizing lens 2810 may be shifted and / or tilted using any of the mechanical steering mechanisms described herein, but is preferably shifted and / or tilted using a powered shift plate approach. Light from the third polarizing lens 2810 is directed to a wave plate 2814, which may include a quarter wave plate, and a reflector 2812, which may reflect the light back through the optical architecture 2800.

[0126] On the second pass through the polarization switch 2806, the light is switched to left-handed circular polarization 802 and directed to a first polarizing lens 2804. The first polarizing lens 2804 focuses the light beam 607 to the steered exit pupil 610 of the eye. In some embodiments, each of the first polarizing lens 2804, second polarizing lens 2808, and third polarizing lens 2810 can include an HOE.

[0127] Figure 29 illustrates various pupil steering positions of a double pass power type shift plate combiner according to some embodiments herein. Figure 29 illustrates the chief rays of a light beam 607 from an image light projector 602 being steered by an optical architecture 2800 to three different exemplary Y-axis exit pupil positions 610A, 610B, 610C.

[0128] The illustrated embodiment of Figure 29 may be able to induce an exit pupil shift range of about 8 mm in the X direction, or about 4 mm to the left and about 4 mm to the right, using a mechanical translation range of only about 0.50 mm. Thus, the double pass powered shift plate combiner may have a ratio of mechanical grating movement to mechanical grating movement of about 16:1. In some embodiments, the double pass power type shift plate combiner is about 1:1, about 2:1, about 3:1, about 4:1, about 5:1, about 6:1, about 7:1, about 8:1, about 9:1, about 10:1, about 11:1, about 12:1, about 13:1, about 14:1, about 15:1, about 16:1, about 17:1, about 18:1, about 19:1, about 20:1, about 21:1, about 22:1, about 23:1, about 24:1, about 25:1, about 26:1, The ratio of ejection motion to mechanical lattice motion may be about 27:1, about 28:1, about 29:1, about 30:1, about 31:1, about 32:1, about 33:1, about 34:1, about 35:1, about 36:1, about 37:1, about 38:1, about 39:1, about 40:1, about 41:1, about 42:1, about 43:1, about 44:1, about 45:1, about 46:1, about 47:1, about 48:1, about 49:1, about 50:1, or between any of the foregoing values.

[0129] Software Integration As discussed herein with reference to FIG. 2, the optical architecture discussed herein may be integrated into an optical combiner as part of a display system 62. The display system 62 may include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or hard disk drive), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) data captured from sensors (e.g., which may be operatively coupled to the frame 64 or otherwise attached to the user 60), such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, gyros, and / or other sensors disclosed herein, and / or b) data acquired and / or processed using a remote processing module 72 and / or a remote data repository 74 (including data related to virtual content), possibly for passage to the display 62 after such processing or retrieval. The local processing and data module 70 may be operably coupled to a remote processing module 72 and a remote data repository 74 by communication links 76, 78, such as via wired or wireless communication links, such that the remote modules 72, 74 are operably coupled to each other and available as resources to the local processing and data module 70.

[0130] FIG. 30 shows a flow chart of an exemplary high-level process for generating an image on a steered exit pupil in a display device according to some embodiments herein. In 3002, one or both eyes of a user may be tracked, for example, by a video-based eye tracking system of the camera assembly 82 (FIG. 60 for generating eye tracking data). The eye tracking data may be transmitted to a processing module 70 or 72, which may process image information acquired by an inner facing imaging system to determine, for example, the diameter or orientation of the pupil of the eye of the user 60, eye movement, or eye posture. In some embodiments, the processing module 70 or 72 may include a controller, for example, of a pupil steering actuation mechanism and an image light projector as discussed herein, for performing exit pupil steering in 3004 to steer the exit pupil according to the eye tracking data. Finally, the controller may perform rendering and / or image correction in 3006 to provide an image to the user's eye according to the location of the steered exit pupil. For example, exit pupil shifts can induce changes in the angular content of the projected image, and therefore in some embodiments it may be necessary to perform geometric distortions and shifts of the image in a coordinated manner with pupil steering.

[0131] In some embodiments, the display system 62 may include one or more controllers (e.g., in the processing modules 70 or 72) for monitoring, initiating, and completing multiple processes in parallel to rapidly and accurately deliver an image to the steered exit pupil. FIG. 31 illustrates a flowchart of an exemplary workflow process for generating an image on a steered exit pupil in a display device, according to some embodiments herein. In some embodiments, the one or more controllers may be configured to control a master process 3100 that manages various other processes and the overall functionality of the display device 62. In some embodiments, the master process 3100 may include loading an image or images at 3102, for example, from the data module 70 or the remote data depository 74. The master process 3100 may also generate various other processes at 3104, including, for example, a motor process 3112 for providing mechanical or electronic actuation for pupil steering, a projection process 3120 for projecting an image and providing image correction, and / or an eye tracking process 3130 for tracking the eye of a user or viewer 60. The master process 3100 may also include displaying the image at 3106 and / or terminating or stopping other processes at 3108. The master process may terminate or stop at 3110.

[0132] The motor process 3112 may control various mechanisms within the display device 62, including, for example, MEMS, mechanical actuators, and / or electrical circuits, to provide proper image projection and pupil steering functionality. In some embodiments, the motor process may include activating mechanical actuators to move one or more stages within the display device 62 to provide movement of a diffractive layer of a diffractive reflector. For example, the motor process 3112 may include initiating a stage control mechanism at 3114 based on received eye tracking data and / or instructions from the master process 3100. At 3116, the motor process may move one or more stages to a desired position to enable proper exit pupil steering. Finally, at 3118, the motor process 3112 may complete the stage control, for example, when the stage has been shifted to direct the image to an exit pupil that matches the location of the user's 60 eye.

[0133] Independently or in parallel with the motor process 3112, the projection process 3120 may include correcting and projecting an image based on eye tracking data and / or instructions received from the master process 3100. For example, the projection process 3120 may include applying color correction to the image at 3122. In optical see-through displays, light coming from background objects mixes with light emanating from the display, resulting in what is known as the color blending problem. Color blending adversely affects the usability of such displays, as it affects the readability and color encoding of digital content. Thus, color correction may be desirable for head-mounted display systems, as color blending affects the usability of such devices, especially when used outdoors. Color correction aims to mitigate the effects of color blending by finding alternative display colors that, when mixed with the background, result in the originally intended color. Another potential problem in augmented reality display systems is image distortion, which may occur depending on the particular display architecture incorporated in the display. Thus, in some embodiments, the projection process 3120 may include applying image distortions at 3124 that may offset or neutralize image distortions caused by the optical architecture of the display device 62. In some embodiments, after providing color correction and image distortions, the correct image or images may be provided or projected at 3126. Finally, at 3128, the correct image or images may be displayed to a user at 3128.

[0134] As discussed herein, a small exit pupil head mounted display may require accurate and fast pupil tracking so that the exit pupil can be steered to the exact location of the user's pupil. Thus, independently of or in parallel with the motor process 3112 and the projection process 3120, an eye tracking process 3130 may include locating the pupil position of the user or viewer 60 at 3132. The pupil position may be located by an inner facing imaging system comprising, for example, the camera assembly 82 (FIG. 6).

[0135] FIG. 32 shows a diagram of exemplary parallel workflow processes and their connections for generating an image in a steered exit pupil in a display system, according to some embodiments herein. In some embodiments, data and / or instructions may be transferred between various processes to enable proper image generation and projection, eye tracking, and pupil steering. For example, a master process 3100 may generate a motor process 3112, a projection process 3120, and / or an eye tracking process 3130. As described above, the eye tracking process 3130 may include locating the pupil of the eye of the user or viewer 60. As shown in FIG. 32, data including the determined location of the pupil may be transmitted to the master process at 3134. In response to receiving the pupil position, the master process may generate and / or issue instructions to the motor process 3112 at 3136, the instructions including speed, positioning, and / or electrical actuation specifications for initiating and completing actuation of the beam steering mechanism of the display device 62. In some embodiments, simultaneously or sequentially with issuing instructions to the motor process 3112 at 3136, the master process 3100 may also transmit data and instructions to generate and / or facilitate the projection process 3120 at 3138, the instructions including the predicted position of the steered exit pupil. In some embodiments, in response to receiving the instructions, the projection process 3120 may apply color correction and image distortion based on the predicted exit pupil position to provide a proper image for display to the user or viewer 60 in the display device 62.

[0136] In some embodiments, once the motor process 3112 has completed actuation of the beam steering mechanism of the display device 62, data including the actual steered position of the exit pupil may be transmitted to the master process 3100 at 3140. This data may be transmitted to the projection process 3120 so that further color correction and image distortion may be applied based on the actual position of the exit pupil. It will be understood by those skilled in the art that the above processes and connections are examples and that the processes described herein may be controlled via one or more of the processing modules 70 or 72.

[0137] FIG. 33 illustrates an example of a flow chart of a process for displaying a steered exit pupil image. In particular, FIG. 33 illustrates a steered exit pupil image display process 3300 in which the image is pre-calibrated prior to displaying the image. In some embodiments, a still image may be loaded at 3302, for example, from the local processing and data module 70 or the remote data depository 74. In some embodiments, hardware such as, for example, the camera assembly 82 (FIG. 6), MEMS, mechanical actuators, image light projectors, and / or electrical circuitry may be activated at 3304. In some embodiments, image distortion may be applied at 3306. In some embodiments, the image distortion may be applied as a function of pupil position. Additionally, in some embodiments, after the image distortion is applied, region of interest (ROI) image copies for various pupil positions may be created and stored. In some embodiments, color correction may be applied at 3308. In some embodiments, color correction may be applied as a function of pupil position. Additionally, in some embodiments, ROI image copies for various pupil positions may be created and stored after color correction has been applied.

[0138] In some embodiments, at 3310, a pupil position of the user or viewer 60 may be determined. As an example, the camera assembly 82 (FIG. 6) may be configured to capture an image of the user or viewer's eye and transmit the image to one or more processing modules 70 or 72, which may determine the pupil position based on the image. In some embodiments, a bin position to the pupil radius may be determined. In some embodiments, once the pupil position is determined at 3312, a pupil steering mechanism may be actuated to move the exit pupil to the determined pupil position of the user or viewer 60. In some embodiments, one or more grids of the beam steering unit may be mechanically translated in two dimensions to move the pupil position, as described in detail above. In some embodiments, at 3314, an appropriate image may be selected from the stored ROI image copies based on the pupil position of the user or viewer 60. Finally, at 3316, an image may be projected and displayed to the user or viewer 60 at the steered exit pupil. The process may end at 3318.

[0139] The above process may require image distortion, color correction, and mechanical translation of beam steering components determined as a function of pupil position. In some embodiments, the above process may allow for a simpler and faster display loop, which refers to pupil tracking, exit pupil steering, and image display loops. However, in some embodiments, the above process may require extensive storage of calibrated ROI image copies from which the correct image can be selected based on pupil position. Although the illustrated embodiment describes loading a single still image, it will be understood that sequences of still or moving images may also be calibrated and displayed, although the amount of images stored would increase linearly with the number of images for display.

[0140] 34 illustrates a flowchart of another exemplary process for displaying an image on a steered exit pupil, according to some embodiments herein. In particular, the process 3400 illustrated in FIG. 34 is directed to automatic and responsive image calibration, in contrast to the pre-calibration process 3300.

[0141] In some embodiments, at 3402, an image, sequence of images, or video may be loaded, for example, from the local processing and data module 70 or the remote data depository 74. In some embodiments, at 3404, hardware such as, for example, the camera assembly 82, MEMS, linear actuators, laser projectors, and / or electrical circuitry may be initiated. In some embodiments, at 3206, a pupil position of the user or observer 60 may be determined. By way of example, the camera assembly 82 may be configured to acquire an image of the user or observer's eye and transmit the image to one or more processing modules 70 or 72 that may determine the pupil position based on the image. In some embodiments, a bin position to the pupil radius may be determined. In some embodiments, once the pupil position is determined, at 3408, a pupil steering mechanism may be actuated to move the exit pupil to the determined pupil position of the user or observer 60. In some embodiments, one or more gratings of the beam steering unit may be mechanically translated in two dimensions to move the pupil position, as described in detail above. In some embodiments, an image ROI may be selected as a function of pupil position at 3410. Once the image ROI is selected, image distortion and color correction may be applied to the image ROI as a function of pupil position at 3412 and 3414, respectively. Finally, at 3416, the image may be displayed to the user or viewer 60 at the exit pupil. The process ends at 3418.

[0142] The above process may require image distortion, color correction, and mechanical translation of the beam steering components to be determined as a function of pupil position. In contrast to process 3300, process 3400 may be easily adapted to accept various types of image or video inputs. However, because image calibration is completed in response to tracked pupil movement rather than during a pre-processing step, display process 3400 may require relatively fast motion tracking, pupil steering, and image correction to provide a proper image to the user or view 60. On the other hand, multiple pre-calibrated ROI images do not generally need to be stored using process 3400, and therefore do not require large amounts of storage.

[0143] Integration of mechanical and optical systems As mentioned above, the display systems described herein may include a diffractive reflector that includes a beam steering unit that includes one or more of the pupil steering mechanisms. In some embodiments, the pupil steering mechanism may include one or more mechanically actuated diffraction gratings. In some embodiments, the mechanically actuated diffraction grating includes a VPH.

[0144] FIG. 35 illustrates an exemplary mechanically actuated diffractive reflector assembly according to some embodiments herein. The diffractive reflector assembly 3500 may include an actuator assembly 3502 coupled to a diffractive reflector 3504 (e.g., an optical combiner). The diffractive reflector 3504 may include any of the optical and / or diffractive reflector architectures described herein. In some embodiments, the actuator assembly may include one or more linear actuators for actuating a pupil steering mechanism within the diffractive reflector 3504. In some embodiments, the one or more linear actuators may include an electric linear actuator. It will be appreciated that the electric linear actuator may convert the rotary motion of an AC or DC motor into linear motion. The linear motion may lift, drop, slide, adjust, tilt, push, or pull a grating within the diffractive reflector 3504 to achieve a desired beam steering position. For example, the linear motion may cause a translation, rotation, and / or tilt of a diffractive grating within the diffractive reflector 3504 such that light from the projector is steered to the exit pupil at a desired location. The linear actuator assembly 3502 may be controlled via one or more controllers. For example, the linear actuator assembly may be controlled by the processing modules 70 and / or 72 of the display device 62.

[0145] In some embodiments, the diffractive reflector 3604 may comprise a mounting portion 3506 that couples to the actuator assembly 3502 and a display portion 3508, where most of the optical architecture is housed and where the optical functions of the diffractive reflector 3504 are generally performed.

[0146] In some embodiments, the display portion 3508 may include a length of approximately 50 mm and a height of 33 mm, where in FIG. 35 the length includes the horizontal dimension or X-direction of the display portion 3508 and the height includes the vertical dimension or Y-direction of the display portion 3508.

[0147] In some embodiments, the length of the display portion 3508 is about 10 mm, about 11 mm, about 12 mm, about 13 mm, about 14 mm, about 15 mm, about 16 mm, about 17 mm, about 18 mm, about 19 mm, about 20 mm, about 21 mm, about 22 mm, about 23 mm, about 24 mm, about 25 mm, about 26 mm, about 27 mm, about 28 mm, about 29 mm, about 30 mm, about 31 mm, about 32 mm, about 33 mm, about 34 mm, about 35 mm, about 36 mm, about 37 mm, about 38 mm, about 39 mm, about 40 mm, about 41 mm, about 42 mm, about 43 mm, about 44 mm, about 45 mm, about 46 mm, about 47 mm, about 48 mm, about 49 mm, about 50 mm, about 51 mm, about 52 mm, about 53 mm, about 54 mm, about 55 mm, about 56 mm, about 57 mm, about 58 mm, about 59 mm, about 60 mm, about 61 mm, about 62 mm, about 63 mm, about 64 mm, about 65 mm, about 66 mm, about 67 mm, about 68 mm, about 69 mm, about 70 mm, about 71 mm, about 72 mm, about 73 mm, about 74 mm, about 75 mm, about 76 mm, about 77 mm, about 78 mm, about 79 mm, about 80 mm, about 81 mm, about 82 mm, about 83 mm, about 84 mm, about 85 mm, about 86 mm, about 87 mm, about 88 mm, about 89 mm, about 90 mm, about 91 mm, about 9 , about 55 mm, about 56 mm, about 57 mm, about 58 mm, about 59 mm, about 60 mm, about 61 mm, about 62 mm, about 63 mm, about 64 mm, about 65 mm, about 66 mm, about 67 mm, about 68 mm, about 69 mm, about 70 mm, about 71 mm, about 72 mm, about 73 mm, about 74 mm, about 75 mm, about 76 mm, about 77 mm, about 78 mm, about 79 mm, about 80 mm, about 81 mm, about 82 mm, about 83 mm, about 84 mm, about 85 mm, about 86 mm, about 87 mm, about 88 mm, about 89 mm, about 90 mm, about 91 mm, about 92 mm, about 93 mm, about 94 mm, about 95 mm, about 96 mm, about 97 mm, about 98 mm, about 99 mm, about 100 mm, or a value between any of the aforesaid values.

[0148] In some embodiments, the height of the display portion 3508 is about 10 mm, about 11 mm, about 12 mm, about 13 mm, about 14 mm, about 15 mm, about 16 mm, about 17 mm, about 18 mm, about 19 mm, about 20 mm, about 21 mm, about 22 mm, about 23 mm, about 24 mm, about 25 mm, about 26 mm, about 27 mm, about 28 mm, about 29 mm, about 30 mm, about 31 mm, about 32 mm, about 33 mm, about 34 mm, about 35 mm, about 36 mm, about 37 mm, about 38 mm, about 39 mm, about 40 mm, about 41 mm, about 42 mm, about 43 mm, about 44 mm, about 45 mm, about 46 mm, about 47 mm, about 48 mm, about 49 mm, about 50 mm, about 51 mm, about 52 mm, about 53 mm, about 54 mm , about 55 mm, about 56 mm, about 57 mm, about 58 mm, about 59 mm, about 60 mm, about 61 mm, about 62 mm, about 63 mm, about 64 mm, about 65 mm, about 66 mm, about 67 mm, about 68 mm, about 69 mm, about 70 mm, about 71 mm, about 72 mm, about 73 mm, about 74 mm, about 75 mm, about 76 mm, about 77 mm, about 78 mm, about 79 mm, about 80 mm, about 81 mm, about 82 mm, about 83 mm, about 84 mm, about 85 mm, about 86 mm, about 87 mm, about 88 mm, about 89 mm, about 90 mm, about 91 mm, about 92 mm, about 93 mm, about 94 mm, about 95 mm, about 96 mm, about 97 mm, about 98 mm, about 99 mm, about 100 mm, or a value between any of the aforesaid values.

[0149] FIG. 36 illustrates another exemplary actuated diffractive reflector assembly according to some embodiments herein. The diffractive reflector assembly 3600 includes a diffractive reflector 3504 including a display portion 3508 and a connector portion 3506. The diffractive reflector 3504 may include any of the optical and / or diffractive reflector architectures described herein. The diffractive reflector assembly 3600 may also include an actuator assembly 3502 including linear actuators 3602A and 3602B for actuating a beam steering mechanism in the diffractive reflector 3504. In some embodiments, the actuator assembly 3502 may enable a 360 degree range of motion of the beam steering mechanism in the horizontal and vertical axes of FIG. 36. In some embodiments, the actuator assembly 3502 may also enable rotation and / or tilt of the beam steering mechanism of the diffractive reflector 3504.

[0150] 37 illustrates another exemplary actuated diffractive reflector according to some embodiments herein. The diffractive reflector assembly 3700 includes a diffractive reflector 3504 including a display portion 3508 and a connector portion 3506. The diffractive reflector assembly 3700 may also include an actuator assembly 3502 including linear actuators 3702A and 3702B for actuating a beam steering mechanism in the diffractive reflector 3504. In some embodiments, the actuator assembly 3502 may enable a 360 degree range of motion of the beam steering mechanism in the horizontal and vertical axes of FIG. 37. In some embodiments, the actuator assembly 3502 may also enable rotation and / or tilt of the beam steering mechanism of the diffractive reflector 3504.

[0151] The illustrated embodiments of Figures 35-37 are exemplary in nature, and it will be appreciated by those skilled in the art that any actuation mechanism may be used to perform mechanical actuation of the beam steering mechanism in a display that includes a diffractive reflector that may form part of an optical combiner.

[0152] As mentioned above, the optical architecture herein may provide accommodation-congruence matching in combination with an upstream variable focus. Without utilizing a variable focus lens, the virtual location of the displayed image may be at optical infinity and the optical architecture may only be configured to display the image on a single depth plane. However, the integrated upstream variable focus may allow the image to be displayed at multiple focal planes by moving the virtual image source. FIG. 38 shows an example diagram of the functionality of an upstream variable focus according to some embodiments herein. In some embodiments, the variable focus lens 3802 may be located in a projection system such as the image light projector 602. All of the VRD systems and architectures described herein are equivalent to a variable focus element located in the projection system. For example, as shown in FIG. 38, placing the variable focus element 3802 adjacent to the laser collimator 3804 allows the display to be focused at various distances to provide appropriate accommodation stimuli for the user's eyes 716 and avoid congruence-accommodation conflicts. This configuration is further equivalent to the MEMS 606 and additional optical elements 609, 611 as described herein. Those skilled in the art will appreciate that the placement of the variable focus within the optical architecture may vary and is not limited to the embodiment shown. For example, the variable focus may be provided upstream or downstream of the MEMS 606.

[0153] Other embodiments In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

[0154] Indeed, it will be understood that the systems and methods of the present disclosure each have several novel aspects, none of which alone serve or are required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to be within the scope of the present disclosure.

[0155] Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. In addition, although features may be described above as acting in a particular combination, and may even initially be claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. No feature or group of features is required or essential to each and every embodiment.

[0156] In particular, it will be understood that conditional terms used herein, such as "can," "could," "might," "may," "e,g.," and the like, unless otherwise specified or understood otherwise within the context in which they are used, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include them. Thus, such conditional terms are generally not intended to imply that features, elements, and / or steps are in any way required by one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without the provision or prompting of the author, whether those features, elements, and / or steps should be included or performed in any particular embodiment. The terms "comprising," "including," "having," and the like, are synonymous and are used in an inclusive, open-ended manner and do not exclude additional elements, features, acts, operations, and the like. Also, the term "or" is used in its inclusive sense (not its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, "a," "an," and "the" as used in this application and the appended claims are to be construed to mean "one or more" or "at least one" unless expressly stated otherwise. Similarly, although the figures depict operations in a particular order, it is to be understood that such operations need not be performed in the particular order depicted, or sequentially, or that not all of the depicted operations need to be performed to achieve desirable results. Additionally, the figures may generally depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may be incorporated into the generally depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. Additionally, operations may be rearranged or reordered in other embodiments.In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and further, it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0157] Thus, the scope of the claims is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the present disclosure, the principles and novel features disclosed herein.

Claims

1. A head mounted display system, A frame, an image light projector supported by the frame; a diffractive reflector supported by the frame and disposed in front of the image light projector, the diffractive reflector configured to receive image light projected by the image light projector and to reflect the image light into an eye of the user when the display system is held on a user, the diffractive reflector comprising: a diffractive reflector comprising a plurality of diffractive layers; an actuator supported by the frame and configured to change a relative orientation of the diffractive layer, the different relative orientations of the diffractive layer configured to reflect light to different associated eye positions; A head mounted display system comprising:

2. The head mounted display system of claim 1 , wherein the actuator is a mechanical actuator configured to change the relative physical position of the diffractive layer.

3. The head mounted display system of claim 2 , wherein the actuator is configured to rotate one or more of the diffractive layers.

4. The head mounted display system of claim 3 , wherein the diffractive reflector comprises a Risley prism-style stack of diffractive layers.

5. The head mounted display system of claim 1 , wherein the actuator is configured to laterally shift one or more of the diffractive layers.

6. The head mounted display system of claim 5 , wherein the diffractive layer comprises a volume phase hologram.

7. The head mounted display system of claim 5 , wherein the diffractive layer comprises a reflective geometric phase lens.

8. 2. The head mounted display system of claim 1, further comprising an eye tracking system for determining a position of a pupil of the eye of the user, the actuator being configured to cause different orientations corresponding to different pupil locations based on the determined position of the pupil.

9. The head mounted display system of claim 1 , wherein the image light projector has a fixed orientation relative to the diffractive reflector.

10. 1. An augmented reality display system, comprising: Image light projector, a reflective diffractive combiner configured to receive projection light from the image light projector, the diffractive combiner configured to combine world light with the projection light from the image light projector and to direct the projection light to form an image at a variable, selectable, steered exit pupil location, the diffractive combiner comprising: a switchable beam steering unit configured to diffract the projection light to form the image at the steered exit pupil location, the beam steering unit comprising: two or more diffraction gratings configured to receive the projection light; an actuation mechanism; At least one of the two or more diffraction gratings is coupled to the actuation mechanism; An augmented reality display system, wherein the actuation mechanism is configured to cause a change in orientation of at least one of the two or more diffraction gratings, wherein different orientations have different associated diffraction angles with respect to incident projection light, and the different associated diffraction angles correspond to different steered exit pupil locations.

11. The image light projector includes: The augmented reality display system of claim 10 comprising one or more microelectromechanical systems scanning mirrors and a laser light source.

12. The augmented reality display system of claim 10 , wherein the image light projector comprises a variable focus lens.

13. The augmented reality display system of claim 10 , further comprising an eye tracking assembly for tracking the position of an eye pupil of a user of the display system.

14. the two or more diffraction gratings form a Risley prism type steering mechanism; The diffraction combiner comprises: a collimating holographic optical element behind the two or more diffraction gratings; a holographic optical element focusing lens in front of the two or more diffraction gratings; The augmented reality display system of claim 10 further comprising:

15. the two or more diffraction gratings include a transmission volume phase holographic grating and a reflection volume phase holographic grating; the transmission volume phase holographic grating is an optical collimator; The augmented reality display system of claim 10 , wherein the reflective volume phase holographic grating is configured to focus the projection light to the steered exit pupil.

16. The augmented reality display system of claim 10 , wherein the two or more gratings include at least two second-order phase gratings.

17. The beam steering unit comprises: A polarization switch; a quarter-wave plate in front of the polarization switch; and a reflector in front of the quarter-wave plate. The augmented reality display system of claim 10 further comprising:

18. The augmented reality display system of claim 10 , further comprising a world compensator configured to offset redirection of the world light by the beam steering unit.

19. the diffraction grating comprises a liquid crystal polarization grating, and the diffraction combiner comprises: a first holographic optical element configured to focus right-handed circularly polarized light; a second holographic optical element configured to collimate left-handed circularly polarized light; The augmented reality display system of claim 10 , wherein the second holographic optical element is behind the beam steering unit and the first holographic optical element is behind the second holographic optical element.

20. The number of addressable steering positions of the steered exit pupil is 2 ｎ 11. The augmented reality display system of claim 10, wherein n is equal to a number of diffraction gratings in the beam steering unit.

Images