Augmented reality display with waveguide configured to capture images of eye and / or environment

Abstract

A head-mounted display system may include a camera, at least one waveguide, at least one coupling optical element configured such that light is coupled into and guided in the waveguide, and at least one decoupling element. At least one out-coupling element may be configured to couple out light guided within the waveguide from the waveguide and to direct the light to the camera. A camera may be disposed in an optical path relative to the at least one out-coupling optical element to receive at least a portion of light that is coupled into and guided in the waveguide via a coupling element and coupled out of the waveguide through the out-coupling element such that an image may be captured by the camera.

Description

[0001] This application is a divisional application of the application with the filing date of September 21, 2018, PCT international application number PCT / US2018 / 052241, Chinese national phase application number 201880072523.3, and invention title "Augmented Reality Display with Waveguides Configured to Capture Images of the Eye and / or the Environment".

[0002] Cross - reference to related applications

[0003] This application claims the benefit of priority under 35 U.S.C § 119(e) to U.S. Provisional Application No. 62 / 561645, filed on September 21, 2017, entitled "AUGMENTED REALITY DISPLAY WITH EYEPIECE CONFIGURED TO CAPTURE IMAGES OF EYE AND ENVIRONMENT", the disclosure of which is hereby incorporated by reference in its entirety. Technical field

[0004] The present disclosure relates to optical devices including augmented reality imaging and visualization systems. Background art

[0005] Modern computing and display technologies have facilitated the development of systems for so - called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that they appear to be real or can be perceived as real. Virtual reality (or "VR") scenarios typically involve the presentation of digital or virtual image information and are opaque to other actual real - world visual inputs; augmented reality (or "AR") scenarios typically involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user. Mixed reality (or "MR") scenarios are a type of AR scenario and typically involve virtual objects integrated into and responsive to the natural world. For example, an MR scenario can include AR image content that appears to be blocked by an object in the real world or otherwise perceived as interacting with that object.

[0006] Reference Figure 1, shows an augmented reality scene 10. A user of AR technology sees a real-world park-like setting 20 featuring people, trees, buildings in the background, and a concrete platform 30. The user also perceives that he / she "sees" "virtual content" such as a robotic statue 40 standing on the real-world platform 30, as well as a flying cartoonish avatar character 50 that appears to be an avatar of a bumblebee. These elements 50, 40 are "virtual" because they do not exist in the real world. Since the human visual perception system is complex, and AR technology that produces a comfortable, natural, and rich presentation of virtual image elements together with other virtual or real-world image elements is challenging.

[0007] The systems and methods disclosed herein address various challenges associated with AR and VR technologies.

[0008] A polarization beam splitter can be used in a display system to direct polarized light to a light modulator and then direct the light to a viewer. Generally, there is a continuing requirement to reduce the size of the display system, and thus there is also a continuing requirement to reduce the size of the components of the display system, including the components that utilize a polarization beam splitter. Summary of the Invention

[0009] Various implementations described herein include a display system configured to provide illumination to the eyes and / or map image projections. Additionally or alternatively, the display system can image the eyes and / or the environment.

[0010] In some embodiments, a head-mounted display system is configured to project light into a user's eyes to display augmented reality image content in the user's field of view. The head-mounted display system can include a frame configured to be supported on the user's head. The display system can also include an image projector configured to project an image into the user's eyes to display image content in the user's field of view. The display system can include a camera, at least one waveguide, at least one coupling optical element configured to couple light into the waveguide and direct it therein, and at least one out-coupling element. The at least one out-coupling element can be configured to couple out light guided within the waveguide from the waveguide and direct the light to the camera. The camera can be disposed in the optical path relative to the at least one out-coupling optical element to receive at least a portion of the light that is coupled into the waveguide via the coupling element, directed therein, and coupled out of the waveguide by the out-coupling element, so that an image can be captured by the camera. Brief Description of the Drawings

[0011] Figure 1 Shows a user view of augmented reality (AR) through an AR device.

[0012] Figure 2 Shows an example of a wearable display system.

[0013] Figure 3 Shows a conventional display system for simulating three-dimensional images for a user.

[0014] Figure 4 Shows aspects of a method of simulating three-dimensional images using multiple depth planes.

[0015] Figures 5A - 5C Shows the relationship between the radius of curvature and the radius of focus.

[0016] Figure 6 Shows an example of a waveguide stack for outputting image information to a user.

[0017] Figure 7 Shows an example of an outgoing light beam output from a waveguide.

[0018] Figure 8 Shows an example of a stacked waveguide assembly, where each depth plane includes an image formed using multiple different component colors.

[0019] Figure 9A Shows a cross-sectional side view of an example of a stacked waveguide group, each stacked waveguide including an incoupling optical element. As discussed herein, the waveguide stack may include an eyepiece.

[0020] Figure 9B Shows Figure 9A a perspective view of an example of multiple stacked waveguides.

[0021] Figure 9C Shows Figure 9A and 9B a top plan view of an example of multiple stacked waveguides.

[0022] Figure 10 Schematically shows a cross-sectional side view of an example imaging system that includes an eyepiece, an image projector, a light source for illuminating an eye, and a camera for capturing an eye image.

[0023] Figure 11A Schematically shows a light source for illuminating an eye and an image projector for injecting an image into the eye, both of which emit light towards an incoupling optical element on a waveguide of the eyepiece.

[0024] Figure 11B Schematically shows the projected light from the light source and from the image projector coupled into the waveguide.

[0025] Figure 11CSchematically shows how the coupled-in light propagates through the waveguide by total internal reflection.

[0026] Figure 11D Schematically shows the light from the light source and from the image projector coupled out from the eyepiece.

[0027] Figure 11E Schematically shows the waveguide and the coupling optical element, which is configured to propagate the coupled-in light at least along the full size of the coupling optical element (e.g., along the x direction). The light entering the eye is shown as coming from an extended light source (e.g., the area where the imaging light will capture the retina).

[0028] Figure 12A Is a cross-sectional view schematically showing the light reflected from the retina and incident on the eyepiece exiting the eye.

[0029] Figure 12B Schematically shows an example of light coupled into the waveguide of the eyepiece.

[0030] Figure 12C Schematically shows the collimated coupled-in light from the eye, which propagates through the waveguide towards the imaging device.

[0031] Figure 12D Schematically shows the coupled-in light from the eye propagating to one or more coupled-out optical elements.

[0032] Figure 12E Schematically shows the light from the eye, which is coupled out of the waveguide by the coupled-out optical element and guided to the camera so that an image of the eye (e.g., the retina) can be captured by the camera.

[0033] Figure 13A Schematically shows how the imaging system can image various parts of the eye (e.g., the retina), which can enable the orientation of the eye and the eye position to be tracked to be determined.

[0034] Figure 13B Shows a pattern of fixation targets displayed sequentially, which is used to orient the eye in various different directions for imaging the retina. The resulting images correspond to different parts of the retina. For example, when the eye is oriented in various directions to view fixation targets located differently on the display, the images captured by the camera include different parts of the retina. These images can be combined to form a larger map or composite image of the retina.

[0035] Figure 14AA cross-sectional view of an imaging system is schematically shown. The imaging system includes an eyepiece and a camera for collecting light from the environment in front of the eyepiece. Light from the environment is shown as being reflected or emitted from one or more physical objects in the environment. Collection of light from objects in the environment in front of the eyepiece can enable an environmental image to be captured.

[0036] Figure 14B Light from the environment is schematically shown being coupled by a coupling optical element into the waveguide of the eyepiece.

[0037] Figure 14C An imaging system is schematically shown that uses a powered optical element, such as a refractive optical element (e.g., a lens such as a wide field of view lens), in front of the eyepiece to collect light from the environment.

[0038] Figure 15A An exemplary imaging system is schematically shown that includes a polarization-selective coupling-in optical element for receiving light from an illumination source and coupling the light into the waveguide of the eyepiece. The eyepiece also includes a polarization-selective optical coupling element for coupling light out of the waveguide. A polarizer can be used to polarize light from the illumination source, and a half-wave retarder can be used to rotate the orientation of linearly polarized light so that the light is transferred into the waveguide through the polarization-selective coupling-in optical element.

[0039] Figure 15B Light from the eye (e.g., light from the retina illuminated by infrared light from an illumination source) is schematically shown being coupled back into the waveguide and directed to the camera for image capture.

[0040] Figure 16 An imaging system configured to image the front part of the eye (e.g., the cornea) is schematically shown. The imaging system includes an eyepiece as described above. The imaging system also includes a positive lens for collimating light collected from the front part of the eye to be coupled into the waveguide via an optical coupling element and propagated to the camera for image capture. The system also includes a negative lens to cancel the positive power introduced by the positive lens and prevent otherwise inverting the environmental image in front of the eyepiece caused by the positive lens.

[0041] Figure 17 Another example imaging system configured to image the front part of the eye (e.g., the cornea) is schematically shown. The imaging system includes a curved wavelength-selective reflector for collimating light from the front part of the eye to be coupled into the waveguide via an optical coupling element and propagated to the camera for image capture. The wavelength-selective reflector can operate in a reflection mode for infrared light reflected from the eye and in a transmission mode for visible light from the environment in front of the user.

[0042] Figure 18 An example imaging system is schematically shown, which also includes a curved wavelength-selective reflector that collimates light from the front of the eye for coupling into a waveguide via an optical coupling element and propagating to a camera for image capture. Polarization selectivity can be employed to assist in controlling the path of light reflected from the eye. Illumination of the eye is provided via the waveguide rather than via a plurality of light sources located between the waveguide and the eye, as Figure 18 shown.

[0043] Figure 19 An imaging system is schematically shown that includes a shutter to assist in a process for subtracting noise.

[0044] Figures 20A - 20E An alternative process for subtracting noise using wavelength modulation in combination with a curved wavelength-selective reflector is schematically shown.

[0045] Figure 21 An example eyepiece is shown that can be used to simultaneously project light into a user's eye to provide image content thereto and simultaneously receive image data of the user's eye or the environment in front of the user.

[0046] Figure 22 A cross-sectional side view of an example of a cholesteric liquid crystal diffraction grating (CLCG) having a plurality of uniform chiral structures is shown.

[0047] Figure 23 An example of an imaging system is shown that includes a forward-facing camera configured to image a wearer's eye using a cholesteric liquid crystal (CLC) off-axis mirror.

[0048] The accompanying drawings are provided to illustrate example embodiments and are not intended to limit the scope of the present disclosure. Throughout the specification, like reference numerals represent like parts. DETAILED DESCRIPTION

[0049] Reference will now be made to the accompanying drawings, where like reference numerals always represent like components.

[0050] Figure 2An example of a wearable display system 60 is shown. The display system 60 includes a display 70, as well as various mechanical and electronic modules and systems that support the functionality of the display 70. The display 70 may be coupled to a frame 80 that can be worn by a user or viewer 90 of the display system and is configured to position the display 70 in front of the eyes of the user 90. In some embodiments, the display 70 may be considered glasses. In some embodiments, a speaker 100 is coupled to the frame 80 and is configured to be located near the ear canal of the user 90 (in some embodiments, another speaker (not shown) may optionally be located near the other ear canal of the user to provide stereo / plastic sound control). The display system may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphones are configured to allow the user to provide input or commands to the system 60 (e.g., selection of voice menu commands, natural language questions, etc.) and / or may allow audio communication with other people (e.g., with users of other similar display systems). The microphones may also be configured as peripheral sensors to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system may also include a peripheral sensor 120a that may be separated from the frame 80 and attached to the body of the user 90 (e.g., on the head, torso, limbs, etc. of the user 90). In some embodiments, the peripheral sensor 120a may be configured to acquire data characterizing the physiological state of the user 90. For example, the sensor 120a may be an electrode.

[0051] Continuing to refer to Figure 2, the display 70 is operatively coupled to the local data processing and module 140 via a communication link 130 (such as via a wired lead or a wireless connection). The local data processing and module 140 can be installed in various configurations, such as being fixedly attached to the frame 80, being fixedly attached to a helmet or hat worn by the user, being embedded within a headset, or otherwise detachably attached to the user 90 (e.g., in a backpack configuration, in a belt-coupled configuration). Similarly, the sensor 120a can be operatively coupled to the local processing and data module 140 via a communication link 120b (e.g., via a wired lead or a wireless connection). The local processing and data module 140 can include a hardware processor and a digital memory such as a non-volatile memory (e.g., flash memory or hard disk drive), both of which can be used to assist in processing, caching, and storing data. The data includes: a) data captured from sensors (which can, for example, be operatively coupled to the frame 80 or otherwise operatively attached to the user 90), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, gyroscopes, and / or other sensors disclosed herein; and / or b) data obtained and / or processed using the remote processing module 150 and / or the remote data repository 160 (which includes data related to virtual content), and these data can be transmitted to the display 70 after such processing or retrieval. The local processing and data module 140 can be operatively coupled to the remote processing module 150 and the remote data repository 160 via communication links 170, 180, such as via wired or wireless communication links, such that these remote modules 150, 160 are operatively coupled to each other and can be used as resources for the local processing and data module 140. In some embodiments, the local processing and data module 140 can include one or more of an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio, and / or a gyroscope. In some other embodiments, one or more of these sensors can be attached to the frame 80 or can be independent structures that communicate with the local processing and data module 140 via a wired or wireless communication path.

[0052] Continuing to refer to Figure 2, in some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and / or map imagery information. In some embodiments, the remote data repository 160 may include a digital data storage facility that may be made available via the Internet or other network configurations in a “cloud” resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers that provide information to the local processing and data module 140 and / or the remote processing module 150, e.g., information for generating augmented reality content. In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing for fully autonomous use from the remote modules.

[0053] Now referring Figure 3 , an image can be perceived as “three-dimensional” or “3-D” by providing slightly different image presentations to each eye of a viewer. Figure 3 A conventional display system for simulating three-dimensional images for a user is shown. Two different images 190, 200 - one for each of the eyes 210, 220 - are output to the user. The images 190, 200 are separated from the eyes 210, 220 by a distance 230 along an optical axis or z-axis parallel to the viewer's eye gaze. The images 190, 200 are flat, and the eyes 210, 220 can focus on the images by presenting a single accommodate state. Such a 3D display system relies on the human visual system to combine the images 190, 200 to provide depth perception and / or scale of the combined image.

[0054] However, it should be understood that the human visual system is more complex and providing a true perception of depth is more challenging. For example, many viewers of conventional "3-D" display systems find such systems uncomfortable or simply unable to perceive a sense of depth. Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of vergence and accommodation. The vergence movement of the two eyes relative to each other (i.e., the rotation of the eyes such that the pupils move towards or away from each other to converge the eyes' gaze to fixate on an object) is closely related to the focusing (or "accommodation") of the eyes' lenses and pupils. Under normal circumstances, changing the focusing of the eyes' lenses or accommodating the eyes to change the focus when switching from one object to another object located at a different distance will automatically result in a matching change in vergence to the same distance under a relationship called the "accommodation-vergence reflex" and pupil dilation or constriction. Similarly, under normal circumstances, a change in vergence will trigger a matching change in accommodation of the lens shape and pupil size. As described herein, many stereoscopic or "3-D" display systems use slightly different presentations (and thus slightly different images) to display a scene to each eye such that the human visual system perceives a three-dimensional perspective. However, such systems are uncomfortable for many viewers because, among other things, such systems simply provide different presentations of the scene, and the eyes view all image information in a single state of accommodation and work in violation of the "accommodation-vergence reflex". A display system that provides a better match between accommodation and vergence can form a more realistic and comfortable three-dimensional image simulation.

[0055] Figure 4 Aspects of a method of simulating a three-dimensional image using multiple depth planes are shown. Continuing reference Figure 4, objects at different distances of 210 and 220 from the eyes along the z-axis are adapted by the eyes 210 and 220 so that those objects are in focus. The eyes 210 and 220 assume a specific adaptation state to bring objects at different distances along the z-axis into focus. Thus, it can be said that a specific adaptation state is associated with a specific depth plane in the depth plane 240, and the specific depth plane has an associated focal length such that when the eyes are in the adaptation state of that depth plane, the objects or parts of the objects in the specific depth plane are in focus. In some embodiments, a three-dimensional image can be simulated by providing different presentations of an image for each of the eyes 210 and 220, and a three-dimensional image can also be simulated by providing different presentations of an image corresponding to each depth plane in the depth plane. Although shown as separate for clarity of illustration, it should be understood that, for example, as the distance along the z-axis increases, the fields of view of the eyes 210 and 220 can overlap. Additionally, although shown as flat for ease of illustration, it should be understood that the profile of the depth plane can be curved in physical space, for example such that all features in the depth plane are in focus with the eyes in a specific adaptation state.

[0056] The distance between the object and the eye 210 or 220 can also change the amount of divergence of the light from that object as seen by that eye. Figures 5A - 5C The relationship between distance and light divergence is shown. The distances between the object and the eye 210 are represented in decreasing order of distance as R1, R2, and R3. As Figures 5A - 5C shown, as the distance to the object decreases, the light becomes more divergent. As the distance increases, the light becomes more collimated. In other words, it can be said that the light field generated by a point (object or part of an object) has a spherical wavefront curvature that is a function of how far that point is from the user's eye. As the distance between the object and the eye 210 decreases, the curvature increases. Thus, at different depth planes, the degree of divergence of the light is also different, and the degree of divergence increases as the distance between the depth plane and the viewer's eye 210 decreases. Although only a single eye 210 is shown in Figures 5A - 5C this and other figures herein for clarity of illustration, it should be understood that the discussion regarding the eye 210 can be applied to both eyes 210 and 220 of the viewer.

[0057] Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide depth perception. Thus, a highly believable simulation of perceived depth can be achieved by providing the eye with different renditions of an image corresponding to each of these limited number of depth planes. The different renditions can be individually focused by the viewer's eye, thereby contributing to providing depth cues to the user based on the eye's accommodation and / or based on observing different image features located on different depth planes that are out of focus, where the eye's accommodation is required to focus on the different image features of a scene located on different depth planes.

[0058] Figure 6 An example of a waveguide stack for outputting image information to a user is shown. The display system 250 includes a waveguide stack or stacked waveguide assembly 260 that can be used to provide three-dimensional perception to the eye / brain using a plurality of waveguides 270, 280, 290, 300, 310. In some embodiments, the display system 250 is Figure 2 system 60, where Figure 6 some portions of the system 60 are shown in more detail schematically. For example, the waveguide assembly 260 can be Figure 2 part of a display 70. It should be understood that in some embodiments, the display system 250 can be considered a light field display. Additionally, the waveguide assembly 260 can also be referred to as an eyepiece.

[0059] Continuing to refer to Figure 6, the waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 located between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 may be configured to send image information to the eye with various levels of wavefront curvature or light ray divergence. Each waveguide level may be associated with a specific depth plane and may be configured to output image information corresponding to that depth plane. The image injection devices 360, 370, 380, 390, 400 may serve as light sources for the waveguides and may be used to inject image information into the waveguides 270, 280, 290, 300, 310, as described herein, where each waveguide may be configured to distribute incident light through each respective waveguide for output to the eye 210. Light leaves the output surfaces 410, 420, 430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 and is injected into the corresponding input surfaces 460, 470, 480, 490, 500 of the waveguides 270, 280, 290, 300, 310. In some embodiments, each of the input surfaces 460, 470, 480, 490, 500 may be an edge of the corresponding waveguide or may be a portion of the major surface of the corresponding waveguide (i.e., one of the waveguide surfaces directly facing the world 510 or the viewer's eye 210). In some embodiments, a single light beam (e.g., a collimated light beam) may be injected into each waveguide so as to output the entire field of view of a cloned collimated light beam directed toward the eye 210 at a specific angle (and divergence amount) corresponding to the depth plane associated with the specific waveguide. In some embodiments, a single image injection device among the image injection devices 360, 370, 380, 390, 400 may be associated with a plurality (e.g., three) of the waveguides 182, 184, 186, 188, 190 and inject light into a plurality (e.g., three) of the waveguides 270, 280, 290, 300, 310.

[0060] In some embodiments, the image injection devices 360, 370, 380, 390, 400 are discrete displays, each generating image information for injection into a corresponding waveguide 270, 280, 290, 300, 310, respectively. In some other embodiments, the image injection devices 360, 370, 380, 390, 400 are output ports of a single multiplexed display, which can convey image information to each of the image injection devices 360, 370, 380, 390, 400 via, for example, one or more light pipes (such as fiber optic cables). It will be appreciated that the image information provided by the image injection devices 360, 370, 380, 390, 400 can include light of different wavelengths or colors (e.g., different component colors as discussed herein).

[0061] In some embodiments, the light injected into waveguides 270, 280, 290, 300, 310 is provided by a light projector system 520 including a light module 540, which may include a light emitter such as a light emitting diode (LED). The light from the light module 540 can be directed to a light modulator 530 and modified by the light modulator 530 (e.g., a spatial light modulator) via a light beam splitter 550. The light modulator 540 can be configured to change the perceived intensity of the light injected into waveguides 270, 280, 290, 300, 310. Examples of spatial light modulators include liquid crystal displays (LCDs), which include liquid crystal on silicon (LCOS) displays. It should be understood that the image injection devices 360, 370, 380, 390, 400 are schematically shown, and in some embodiments, these image injection devices can represent different optical paths and positions in a common projection system configured to output light into an associated one of waveguides 270, 280, 290, 300, 310.

[0062] In some embodiments, the display system 250 can be a scanned fiber optic display that includes one or more scanned optical fibers configured to project light into one or more waveguides 270, 280, 290, 300, 310 and ultimately into the viewer's eye 210 in various patterns (e.g., raster scan, helical scan, Lissajous pattern, etc.). In some embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 can schematically represent a single scanned optical fiber or a bundle of scanned optical fibers configured to inject light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 can schematically represent multiple scanned optical fibers or multiple bundles of scanned optical fibers, each of the multiple scanned optical fibers or each of the multiple bundles of scanned optical fibers being configured to inject light into a respective one of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers can be configured to transmit light from the optical module 540 to one or more waveguides 270, 280, 290, 300, 310. It should be understood that one or more intermediate optical structures can be provided between the one or more scanned optical fibers and the one or more waveguides 270, 280, 290, 300, 310 to redirect, for example, light exiting the scanned optical fibers into the one or more waveguides 270, 280, 290, 300, 310.

[0063] The controller 560 controls the operation of one or more of the stacked waveguide assemblies 260, including the operation of the image injection devices 360, 370, 380, 390, 400, the light source 540, and the light modulator 530. In some embodiments, the controller 560 is part of the local data processing module 140. The controller 560 includes programming (e.g., instructions in a non-transitory medium) that, for example, conditions the timing and provision of image information to the waveguides 270, 280, 290, 300, 310 according to any of the various schemes disclosed herein. In some embodiments, the controller can be a single monolithic device or a distributed system connected by wired or wireless communication channels. In some embodiments, the controller 560 can be part of the processing module 140 or 150 ( Figure 2 ).

[0064] Continuing reference Figure 6, waveguides 270, 280, 290, 300, 310 can be configured to propagate light within each respective waveguide by total internal reflection (TIR). Waveguides 270, 280, 290, 300, 310 can each be planar or have other shapes (e.g., curved), having a top major surface and a bottom major surface and an edge extending between these top and bottom major surfaces. In the illustrated configuration, waveguides 270, 280, 290, 300, 310 can each include outcoupling optical elements 570, 580, 590, 600, 610, which are configured to extract light out of the waveguide by redirecting the light propagating within each respective waveguide out of the waveguide to output image information to the eye 210. The extracted light can also be referred to as outcoupled light, and the outcoupling optical elements can also be referred to as light extraction optical elements. The extracted light beam can be output by the waveguide at the location where the light propagating in the waveguide impinges on the light extraction optical element. The outcoupling optical elements 570, 580, 590, 600, 610 can include, for example, gratings of diffractive optical features, as further discussed herein. Although illustrated as being disposed at the bottom major surface of waveguides 270, 280, 290, 300, 310 for ease of description and clarity of drawing, in some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 can be disposed at the top and / or bottom major surfaces, and / or can be disposed directly within the volume of waveguides 270, 280, 290, 300, 310, as further discussed herein. In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 can be formed in a material layer attached to a transparent substrate to form waveguides 270, 280, 290, 300, 310. In some other embodiments, waveguides 270, 280, 290, 300, 310 can be a single-piece material, and the outcoupling optical elements 570, 580, 590, 600, 610 can be formed on the surface and / or within the interior of the piece of material.

[0065] Continue to refer to Figure 6, as discussed herein, each of the waveguides 270, 280, 290, 300, 310 is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye can be configured to transmit collimated light (which is injected into such waveguide 270) to the eye 210. The collimated light can represent an optically infinite focal plane. The next upward waveguide 280 can be configured to send the collimated light passing through the first lens 350 (e.g., a negative lens) out before it can reach the eye 210; such a first lens 350 can be configured to produce a slightly convex wavefront curvature such that the eye / brain interprets the light from the next upward waveguide 280 as coming from a first focal plane that is closer inwardly toward the eye 210 from optically infinite distance. Similarly, the third upward waveguide 290 causes its output light to pass through the first lens 350 and the second lens 340 before reaching the eye 210; the combined optical power of the first lens 350 and the second lens 340 can be configured to produce another increment of wavefront curvature such that the eye / brain interprets the light from the third waveguide 290 as coming from a second focal plane that is closer inwardly toward the person from optically infinite distance than the light from the next upward waveguide 280.

[0066] The other waveguide layers 300, 310 and lenses 330, 320 are configured similarly, where the highest waveguide 310 in the stack sends its output through all the lenses between it and the eye for an aggregate focal power representative of the focal plane closest to the person. When viewing / interpreting light from the world 510 on the other side of the stacked waveguide assembly 260, to compensate for the stack of lenses 320, 330, 340, 350, a compensating lens layer 620 can be provided on top of the stack to compensate for the aggregate focal power of the underlying lens stack 320, 330, 340, 350. This configuration provides as many perceived focal planes as there are available waveguide / lens pairings. The outcoupling optical elements of the waveguides and the focusing aspects of the lenses can be static (i.e., not dynamic or electroactive). In some alternative embodiments, one or both of them can be dynamic using electroactive features.

[0067] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 can have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 can be configured to output images set to the same depth plane, or multiple subsets of the waveguides 270, 280, 290, 300, 310 can be configured to output images set to the same multiple depth planes, with each depth plane having a set. This can provide an advantage for forming tiled images to provide an extended field of view at those depth planes.

[0068] Continuing reference Figure 6 to, the out-coupling optical elements 570, 580, 590, 600, 610 can be configured to redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of out-coupling optical elements 570, 580, 590, 600, 610 that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 can be volume or surface features that can be configured to output light at a particular angle. For example, the light extraction optical elements 570, 580, 590, 600, 610 can be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, the features 320, 330, 340, 350 can not be lenses; rather, they can simply be spacers (e.g., cladding layers and / or structures for forming an air gap).

[0069] In some embodiments, the out-coupling optical elements 570, 580, 590, 600, 610 are diffraction features that form a diffraction pattern, or "diffractive optical elements" (also referred to herein as "DOEs"). Preferably, the DOE has a low enough diffraction efficiency such that only a portion of the light of the beam is deflected towards the eye 210 through each intersection of the DOE, while the remainder continues to travel through the waveguide via TIR. The light carrying the image information is thus split into multiple related outgoing beams that exit the waveguide at multiple locations, and as a result, is a fairly uniform pattern of outgoing emission towards the eye 210 for that particular collimated beam bouncing within the waveguide.

[0070] In some embodiments, one or more DOEs can be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer where microdroplets contain a diffraction pattern within a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0071] In some embodiments, a camera component 630 (e.g., a digital camera, including visible light and infrared cameras) may be provided to capture images of the eye 210 and / or the tissue surrounding the eye 210, thereby detecting user input and / or monitoring the user's physiological state, for example. As used herein, a camera may be any image capture device. In some embodiments, the camera component 630 may include an image capture device and a light source to project light (e.g., infrared light) onto the eye, and then the light may be reflected by the eye and detected by the image capture device. In some embodiments, the camera component 630 may be attached to the frame 80( Figure 2 ) and may be in electrical communication with the processing module 140 and / or 150, which may process the image information from the camera component 630. In some embodiments, one camera component 630 may be used for each eye to monitor each eye separately.

[0072] Now referring to Figure 7 , an example of an outgoing beam output from a waveguide is shown. One waveguide is shown, but it should be understood that other waveguides in the waveguide assembly 260( Figure 6 ) may function similarly, where the waveguide assembly 260 includes a plurality of waveguides. Light 640 is injected into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 impinges on the DOE 570, a portion of the light exits the waveguide as the outgoing beam 650. The outgoing beam 650 is shown as being substantially parallel, but as discussed herein, depending on the depth plane associated with the waveguide 270, the outgoing beam 650 may also be redirected at an angle (e.g., to form a diverging outgoing beam) to propagate to the eye 210. It should be understood that a substantially parallel outgoing beam may indicate a waveguide having an outcoupling optical element that outcouples light to form an image that appears to be disposed on a depth plane at a relatively large distance (e.g., optical infinity) from the eye 210. Other waveguides or other groups of outcoupling optical elements may output a more diverging outgoing beam pattern, which will require the eye 210 to accommodate to a closer distance to focus it on the retina and will be interpreted by the brain as light from a distance closer to the eye 210 than optical infinity.

[0073] In some embodiments, a full-color image may be formed at each depth plane by superimposing images of each component color (e.g., three or more component colors). Figure 8An example of a stacked waveguide assembly is shown, where each depth plane includes an image formed using multiple different component colors. The illustrated embodiment shows depth planes 240a - 240f, but more or fewer depths can also be considered. Each depth plane can have three or more component color images associated therewith, including: a first image G of a first color; a second image R of a second color; and a third image B of a third color. For the dioptric power (dpt) after the letters G, R, and B, different numbers are used in the figure to represent different depth planes. As an example, the number following each of these letters represents the dioptric power (1 / m), or the inverse distance of the depth plane from the viewer, and each box in the figure represents a separate component color image. In some embodiments, to address the differences in the focusing of the eye for different wavelengths of light, the exact placement of the depth planes of different component colors can be varied. For example, the different component color images of a given depth plane can be placed on depth planes corresponding to different distances from the user. Such an arrangement can increase visual acuity and user comfort and / or can reduce chromatic aberration.

[0074] In some embodiments, the light of each component color can be output by a single dedicated waveguide, and thus, each depth plane can have multiple waveguides associated therewith. In such embodiments, each box in the figure that includes the letter G, R, or B can be understood to represent a separate waveguide, and each depth plane can provide three waveguides, where each depth plane provides three component color images. Although, for ease of description, the waveguides associated with each depth plane are shown adjacent to each other in this figure, it should be understood that in a physical device, the waveguides can all be arranged in a stack with one waveguide per layer. In some other embodiments, multiple component colors can be output by the same waveguide, such that, for example, each depth plane can provide only a single waveguide.

[0075] Continuing to refer to Figure 8 , in some embodiments, G is green, R is red, and B is blue. In some other embodiments, in addition to red, green, or blue, other colors associated with light of other wavelengths (including magenta and cyan) can be used, or one or more of red, green, or blue can be replaced.

[0076] It should be understood that references to light of a given color throughout this disclosure will be understood to include light of one or more wavelengths within the wavelength range of light that a viewer perceives as having that given color. For example, red light can include light of one or more wavelengths in the range of approximately 620 - 780 nm, green light can include light of one or more wavelengths in the range of approximately 492 - 577 nm, and blue light can include light of one or more wavelengths in the range of approximately 435 - 493 nm.

[0077] In some embodiments, the light source 540 ( Figure 6 ) may be configured to emit light at one or more wavelengths outside the viewer's visual perception range (e.g., infrared and / or ultraviolet wavelengths). Additionally, the light coupling-in, coupling-out, and other light redirecting structures of the waveguide of the display 250 may be configured to direct the light out of the display and emit it towards the user's eye 210, e.g., for imaging and / or user stimulation applications.

[0078] Now referring to Figure 9A , in some embodiments, it may be necessary to redirect the light incident on the waveguide to couple the light into the waveguide. A coupling-in optical element may be used to redirect the light and couple the light into its corresponding waveguide. Figure 9A A cross-sectional side view of an example of a plurality of stacked waveguides or groups of stacked waveguides 660 is shown, each stacked waveguide including a coupling-in optical element. Each waveguide may be configured to output light at one or more different wavelengths, or one or more different wavelength ranges. It should be understood that the stack 660 may correspond to the stack 260 ( Figure 6 ), and the illustrated stacked waveguides 660 may correspond to a portion of the plurality of waveguides 270, 280, 290, 300, 310, except that light from one or more of the image injection devices 360, 370, 380, 390, 400 is injected into the waveguide at a location where the light needs to be redirected for coupling-in.

[0079] The stacked waveguide group 660 shown includes waveguides 670, 680, and 690. Each waveguide includes an associated coupling optical element (which may also be referred to as a light input region on the waveguide), for example, a coupling optical element 700 disposed on a major surface (e.g., upper major surface) of the waveguide 670, a coupling optical element 710 disposed on a major surface (e.g., upper major surface) of the waveguide 680, and a coupling optical element 720 disposed on a major surface (e.g., upper major surface) of the waveguide 690. In some embodiments, one or more of the coupling optical elements 700, 710, 720 may be disposed on the bottom major surface of the corresponding waveguide 670, 680, 690 (particularly in the case where one or more coupling optical elements are reflective, deflecting optical elements). As shown, the coupling optical elements 700, 710, 720 may be disposed on the upper major surface (or the top of the next lower waveguide) of their corresponding waveguides 670, 680, 690, particularly in the case where those coupling optical elements are transmissive, deflecting optical elements. In some embodiments, the coupling-in optical elements 700, 710, 720 can be disposed in the body of the respective waveguides 670, 680, 690. In some embodiments, as discussed herein, the coupling-in optical elements 700, 710, 720 are wavelength selective such that they selectively redirect light of one or more wavelengths while transmitting light of other wavelengths. Although shown on one side or corner of their respective waveguides 670, 680, 690, it should be understood that in some embodiments, the coupling-in optical elements 700, 710, 720 can be disposed in other areas of their respective waveguides 670, 680, 690.

[0080] As shown, the coupling-in optical elements 700, 710, 720 can be laterally offset from each other. In some embodiments, each coupling-in optical element can be offset so that it receives light without the light passing through another coupling-in optical element. For example, each coupling-in optical element 700, 710, 720 can be configured to be offset from, for example, Figure 6 The different image injection devices 360 , 370 , 380 , 390 , and 400 shown receive light and may be separated (eg, laterally spaced apart) from the other incoupling optical elements 700 , 710 , 720 such that it receives substantially no light from the other incoupling optical elements 700 , 710 , 720 .

[0081] Each waveguide also includes an associated light distributing element. For example, a light distributing element 730 disposed on the main surface (e.g., the top main surface) of waveguide 670, a light distributing element 740 disposed on the main surface (e.g., the top main surface) of waveguide 680, and a light distributing element 750 disposed on the main surface (e.g., the top main surface) of waveguide 690. In some other embodiments, the light distributing elements 730, 740, 750 may be respectively disposed on the bottom main surfaces of the associated waveguides 670, 680, 690. In some other embodiments, the light distributing elements 730, 740, 750 may be respectively disposed on the top and bottom main surfaces of the associated waveguides 670, 680, 690; or the light distributing elements 730, 740, 750 may be respectively disposed on different main surfaces among the top and bottom main surfaces of different associated waveguides 670, 680, 690.

[0082] Waveguides 670, 680, 690 may be separated and spaced apart by, for example, gas, liquid, and / or solid material layers. For example, as shown, layer 760a may separate waveguides 670 and 680; and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed of a low refractive index material (i.e., a material having a refractive index lower than that of the material of the adjacent waveguides among waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is less than the refractive index of the material forming waveguides 670, 680, 690 by 0.05 or more, or 0.10 or less. Advantageously, the low refractive index layers 760a, 760b may be used as cladding layers to facilitate total internal reflection (TIR) of light passing through waveguides 670, 680, 690 (e.g., TIR between the top and bottom main surfaces of each waveguide). In some embodiments, layers 760a, 760b are formed of air. Although not shown, it should be understood that the top and bottom of the illustrated waveguide group 660 may include adjacent cladding layers.

[0083] Preferably, for ease of manufacturing and other considerations, the materials forming waveguides 670, 680, 690 are similar or the same, and the materials forming layers 760a, 760b are similar or the same. In some embodiments, the materials forming waveguides 670, 680, 690 may be different between one or more waveguides, and / or the materials forming layers 760a, 760b may be different, while still maintaining the various refractive index relationships described above.

[0084] Continuing to refer to Figure 9A, light rays 770, 780, 790 are incident on waveguide group 660. It should be understood that light rays 770, 780, 790 can be injected into waveguides 670, 680, 690 through one or more image injection devices 360, 370, 380, 390, 400( Figure 6 ).

[0085] In some embodiments, light rays 770, 780, 790 have different properties, e.g., different wavelengths or different wavelength ranges, which may correspond to different colors. Each of the light-coupling optical elements 700, 122, 720 deflects the incident light such that the light propagates through a corresponding one of waveguides 670, 680, 690 by TIR. In some embodiments, each of the light-coupling optical elements 700, 710, 720 selectively deflects light of one or more specific wavelengths while transmitting other wavelengths to the underlying waveguides and associated light-coupling optical elements.

[0086] For example, the light-coupling optical element 700 can be configured to deflect the light ray 770 having a first wavelength or wavelength range while transmitting the light rays 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. The transmitted light ray 780 impinges on the light-coupling optical element 710 and is deflected by the light-coupling optical element 710, which is configured to deflect light having the second wavelength or wavelength range. The light ray 790 is deflected by the light-coupling optical element 720, which is configured to selectively deflect light having the third wavelength or wavelength range.

[0087] Continuing to refer to Figure 9A , the deflected light rays 770, 780, 790 are deflected such that they propagate through the corresponding waveguides 670, 680, 690; that is, the light-coupling optical elements 700, 710, 720 of each waveguide deflect the light into the corresponding waveguides 670, 680, 690 to couple the light into the corresponding waveguides. The light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the corresponding waveguides 670, 680, 690 by TIR. The light rays 770, 780, 790 propagate through the corresponding waveguides 670, 680, 690 by TIR until they impinge on the corresponding light distribution elements 730, 740, 750 of the waveguides.

[0088] Now referring to Figure 9B , there is shown Figure 9APerspective view of an example of multiple stacked waveguides. As described above, the coupled-in light rays 770, 780, 790 are respectively deflected by the optical elements 700, 710, 720 and then propagate through TIR in the waveguides 670, 680, 690 respectively. Then, the light rays 770, 780, 790 respectively irradiate the light distribution elements 730, 740, 750. The light distribution elements 730, 740, 750 deflect the light rays 770, 780, 790 such that they respectively propagate towards the out-coupling optical elements 800, 810, 820.

[0089] In some embodiments, the light distribution elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPE deflects or distributes light to the out-coupling optical elements 800, 810, 820, and in some embodiments, the OPE also increases the beam or spot size of the light when the light propagates to the out-coupling optical elements. In some embodiments, the light distribution elements 730, 740, 750 can be omitted, and the coupled-in optical elements 700, 710, 720 can be configured to deflect light directly to the out-coupling optical elements 800, 810, 820. For example, referring to Figure 9A , the light distribution elements 730, 740, 750 can be respectively replaced by the out-coupling optical elements 800, 810, 820. In some embodiments, the out-coupling optical elements 800, 810, 820 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light in the viewer's eye 210 ( Figure 7 ). It should be understood that the OPE can be configured to increase the size of the eye box on at least one axis, and the EPE can increase the eye box on an axis orthogonal to, for example, the axis of the OPE. For example, each OPE can be configured to redirect a portion of the light incident on the OPE to the EPE of the same waveguide while allowing the remaining portion of the light to continue propagating along the waveguide. When the remaining light irradiates the OPE again, another portion of the remaining light is redirected to the EPE, and the remaining portion of the remaining light continues to propagate further along the waveguide, and so on. Similarly, when irradiating the EPE, a portion of the irradiated light is directed out of the waveguide towards the user, and the remaining portion of the light continues to propagate through the waveguide until it irradiates the EP again, at which time another portion of the irradiated light is directed out of the waveguide, and so on. Therefore, whenever a portion of the light is redirected by the OPE or EPE, the coupled-in light of a single beam can be "copied", thereby forming a field of cloned light beams, as shown in Figure 6 . In some embodiments, the OPE and / or EPE can be configured to modify the size of the light beam.

[0090] Therefore, referring to Figure 9A and 9B, in some embodiments, the waveguide group 660 includes waveguides 670, 680, 690; input coupling optical elements 700, 710, 720; light distribution elements (e.g., OPE) 730, 740, 750; and output coupling optical elements (e.g., EP) 800, 810, 820 for each component color. The waveguides 670, 680, 690 may be stacked with an air gap / cladding layer present between each. The input coupling optical elements 700, 710, 720 redirect or deflect the incident light (using different input coupling optical elements that receive light of different wavelengths) into their waveguides. The light then propagates at an angle, which results in TIR within the respective waveguides 670, 680, 690. In the example shown, the light ray 770 (e.g., blue light) is deflected by the first input coupling optical element 700 and then continues to jump into the waveguide, interacting with the light distribution element (e.g., OPE) 730 in the manner described previously, and then interacting with the output coupling optical element (e.g., EPs) 800. The light rays 780 and 790 (e.g., green light and red light, respectively) will pass through the waveguide 670, where the light ray 780 impinges on the input coupling optical element 710 and is deflected by the input coupling optical element 710. The light ray 780 then jumps into the waveguide 680 via TIR, continues to the light distribution element (e.g., OPE) 740 that it reaches, and then reaches the output coupling optical element (e.g., EP) 810. Finally, the light ray 790 (e.g., red light) passes through the waveguide 690 to impinge on the light input coupling optical element 720 of the waveguide 690. The light input coupling optical element 720 deflects the light ray 790 such that the light ray propagates to the light distribution element (e.g., OPE) 750 via TIR, and then propagates to the output coupling optical element (e.g., EP) 820 via TIR. Then, the output coupling optical element 820 finally couples out the light ray 790 to the viewer, who also receives the output light from the other waveguides 670, 680.

[0091] Figure 9C Shows Figure 9A and 9B A top plan view of an example of a plurality of stacked waveguides. As shown, the waveguides 670, 680, 690 and the light distribution elements 730, 740, 750 associated with each waveguide and the associated output coupling optical elements 800, 810, 820 may be vertically aligned. However, as discussed herein, the input coupling optical elements 700, 710, 720 are not vertically aligned; rather, the input coupling optical elements are preferably non-overlapping (e.g., laterally spaced apart as seen in the top view). As further discussed herein, this non-overlapping spatial arrangement helps to inject light from different sources one-to-one into different waveguides, thereby allowing a particular light source to be uniquely coupled to a particular waveguide. In some embodiments, an arrangement including non-overlapping spatially separated input coupling optical elements may be referred to as a shifted pupil system, and the input coupling optical elements within these arrangements may correspond to sub-pupils.

[0092] Eye imaging and environmental imaging

[0093] As described above, a head-mounted display can be used to provide image content to a user, which is integrated with, combined with, and / or superimposed on a view of the world in front of the wearer. Such a head-mounted display system can be configured to project light into the user's eyes to form augmented reality image content, and to transmit light from the environment in front of the user to the user. The head-mounted display system can include one or more cameras for imaging the environment and / or the user's eyes. An outward-facing camera can be used to directly image the environment, for example, to determine where to place the augmented reality image content relative to objects in the environment. For example, imaging the environment can provide the location of a table such that the head-mounted display can render an image of a person standing near the table rather than on or in the table. An inward-facing camera can be used to directly image the eye, for example, for eye tracking. Disclosed herein are examples of head-mounted display systems and / or imaging systems that can be configured to also image the eye and / or the environment. In some designs, the system does not require an inward-facing and / or outward-facing camera to directly image the eye and / or the environment, respectively. Such a system can employ one or more cameras that are configured to receive light from the eye / environment via an eyepiece, such as one or more waveguides in an eyepiece that is optically communicable with the one or more cameras. Through the light collected by the one or more waveguides, the one or more cameras can generate an image of the eye or an image of the environment in front of the user. Using waveguides to collect light for imaging the eye and / or the environment can potentially reduce the form factor of the head-mounted display, making the head-mounted display potentially more compact and / or aesthetically desirable.

[0094] Figure 10 An example imaging system 900 configured to image the eye is shown, which is integrated with an eyepiece 950 that can be used in a head-mounted display. The eyepiece 950 that can be disposed in front of the user's eye 210 can be used to inject image content into the eye and to image the eye. Figure 10 An eyepiece 950 is shown in front of one eye 210. Such as Figure 2 The various head-mounted display systems shown can include a pair of eyepieces 950 and associated components disposed in front of the respective left and right eyes 210. In Figure 10 a single waveguide 940 is shown, but the waveguide 940 can include one, two, three, four, six, seven, eight, or more waveguides (e.g., a stack of one or more waveguides).

[0095] The imaging system 900 may include a light source or illumination source 960 that illuminates the eye to facilitate image capture, an eyepiece 950 that includes a waveguide 940 configured to propagate light therein, and / or an imaging device 920 such as a camera for image capture. An image projector 930 for generating an image that may be injected into the eye via the eyepiece 950 is also shown. The eyepiece 950 may include one or more waveguides 940 configured to transmit light from the illumination source 960 and / or the image projector 930 to the eye, and to transmit light from the eye to the camera 920. The eyepiece 950 may further include one or more coupling optical elements 944 for coupling light out of the waveguide 940 and into the eye for illuminating the eye and for image injection, and / or for coupling light out of the eye and into the waveguide for image capture. The eyepiece 950 may additionally include one or more input optical elements 942 for coupling light from the illumination source 960 and / or the image projector 930 into the waveguide 940 and one or more output optical elements 952 for coupling light from the waveguide out to the camera 920.

[0096] The eyepiece 950 may be disposed on a frame wearable on the head. The eyepiece 950 may be disposed in front of the eye 210. The eyepiece 950 may have an inner or nasal side closer to the wearer's nose and an opposite outer or temporal side closer to the temple and away from the wearer's nose. In Figure 10 this case, the coupling optical element 944 is on the inner or nasal side relative to the input optical element 942 and the output optical element 952 (which are on the outer or temporal side of the coupling optical element 944). The illumination source 960 is also more on the inner or nasal side relative to the image projector 930 (or the image projector is located on the outer or temporal side relative to the illumination source 930). However, the relative positions may be different. For example, in some designs, the illumination source 960 may be more on the outer or temporal side than the image projector 930.

[0097] The waveguide 940 may include a sheet or layer having two major surfaces (a front surface and a rear surface) that have the largest surface areas and are disposed opposite each other. When the user wears the head-mounted display, the front surface may be away from the user's eye 210 (closer to the environment in front of the wearer), and the rear surface may be closer to the user's eye (and away from the environment in front of the wearer). The waveguide 940 may include a transparent material (e.g., glass, plastic) having a refractive index greater than 1.0 such that light can be guided therein by total internal reflection between the major surfaces. For one or more embodiments described herein, elements having the same number may have the same function.

[0098] The coupling optical element 944 for coupling light from the waveguide 940 to the eye 210 and / or from the waveguide to the eye may be disposed on or within the waveguide 940. AsFigure 10 As shown, the coupling optical element 944 can be disposed in the optical path between the user's eye 210 and the waveguide 940 such that light coupled from the waveguide 940 via the coupling optical element 944 can be incident on the user's eye 210 (e.g., to illuminate the eye and / or for image injection). The coupling optical element 944 can include a plurality of turning features configured to turn light guided within the waveguide out of the waveguide or to turn light incident on the coupling optical element 944 at an angle into the waveguide to be guided therein by total internal reflection. The coupling optical element 944 and the turning features can be physically joined to the waveguide 940. For example, the coupling optical element 944 can include a holographic or diffractive optical element (e.g., a surface relief grating) patterned (e.g., etched) within or on the waveguide 940. The coupling optical element 944 can include a layer disposed on the waveguide 940 or can be formed within the waveguide 940. For example, a volume hologram or other diffractive optical element can be formed by varying the refractive index of a material including the waveguide or a layer disposed thereon. Thus, the coupling optical element 944 can be disposed within the volume of the waveguide 940 or as a layer thereon.

[0099] Depending on the design, the coupling optical element 944 can be transmissive or reflective and can operate in a transmissive or reflective manner. For example, the coupling optical element 944 can include a transmissive or reflective diffractive optical element (e.g., a grating) or a holographic optical element that operates in a transmissive or reflective manner, respectively, to turn, for example, light transmitted therethrough or reflected therefrom. The coupling optical element 944 can include a polarization optical element, such as a polarization-selective turning element (e.g., a polarizer). The polarization-selective turning element can include one or more polarization gratings, diffractive optical elements, and / or holographic optical elements and can include a liquid crystal structure, such as a liquid crystal polarization grating. The coupling optical element 944 can be configured to direct light from the image projector 930 and / or the light source 960 guided within the waveguide 940 by total internal reflection (TIR) to the user's eye 210 at an angle less than (e.g., more normal to) the critical angle so as to exit the waveguide to the eye. Additionally or alternatively, the coupling optical element 944 can be configured to couple light from the eye 210 into the waveguide 940 at an angle greater than (e.g., less normal to) the critical angle so as to be guided therein by total internal reflection to the camera 920.

[0100] As Figure 10As shown, the optical coupling element 942 for coupling light from the illumination source 960 and / or the image projector 930 into the waveguide 940 may be disposed above or within the waveguide 940. The optical coupling element 942 may be disposed on the optical path between the light source 960 and the waveguide 940 such that the light coupled from the light source 960 via the optical coupling element 942 is guided within the waveguide 940. The optical coupling element 942 may include, for example, a plurality of turning features configured to turn the light incident thereon at an angle into the waveguide for guiding in the waveguide by total internal reflection. The optical coupling element 942 may include a liquid crystal structure, such as a liquid crystal polarization grating. Additionally or alternatively, the optical coupling element 942 may include a blazed grating. The optical coupling element 942 may include a layer disposed on the waveguide 940, or may be formed above or within (e.g., patterned) the waveguide 940, or may otherwise be fabricated therein. For example, a surface holographic or diffractive optical element (e.g., a surface relief grating) may be fabricated by patterning (e.g., etching) the surface of the waveguide or a layer thereon. Volume holographic or diffractive optical elements may also be formed by changing the refractive index of a material including the waveguide or a layer thereon. Thus, the optical coupling element 942 may be disposed in the volume of the waveguide 940 or in a layer thereon. Depending on the design, the optical coupling element 942 may be transmissive or reflective and may operate in a transmissive or reflective manner. For example, the optical coupling element 942 may include transmissive or reflective diffractive optical elements (e.g., gratings) or holographic optical elements that operate in a transmissive or reflective manner, respectively, such that the light transmitted through or reflected from it is turned.

[0101] The optical coupling-in element 942 may include a reflective optical element (e.g., a mirror). For example, the optical coupling-in element 942 may include an off-axis reflector. Additionally or alternatively, the optical coupling-in element 942 and / or the optical coupling element 944 may include polarization optical elements, such as polarization-selective steering elements (e.g., polarizers). The polarization-selective steering element may include one or more polarization gratings, diffractive optical elements, and / or holographic optical elements, and may include a liquid crystal structure, such as a liquid crystal polarization grating. For example, one or both of the optical coupling-in element 942 and / or the optical coupling element 944 may include a liquid crystal polarization grating (LCPG). The LCPG may potentially provide efficient diffraction over a wide wavelength. Thus, the LCPG can be used for the optical coupling-in element 942 and / or the optical coupling element 944. The LCPG may be polarization-dependent. The LCPG or other types of liquid crystal gratings, diffractive optical elements, or optical elements may include a pattern or arrangement of liquid crystal molecules configured to provide one or more functions, such as turning light into or out of a waveguide. Thus, the optical coupling-in element 942 and / or the optical coupling element 944 may include a polarization grating. Additionally or alternatively, the optical coupling-in element 942 and / or the optical coupling element 944 may include liquid crystal, and thus in some implementations, one or both may be a liquid crystal grating or a liquid crystal diffractive optical element. Additionally or alternatively, one or both of the optical coupling-in element 942 and / or the optical coupling element 944 may include a blazed grating. In some designs, the optical coupling-in element 942 includes a liquid crystal reflector, such as a cholesteric liquid crystal reflecting lens (e.g., a reflective liquid crystal diffractive lens, a Bragg reflection structure, a reflective liquid crystal diffraction grating, etc.).Some non-limiting examples of liquid crystal gratings, liquid crystal polarization gratings, and other liquid crystal optical elements are discussed in the following published applications, the entire contents of which are hereby incorporated by reference for all purposes: U.S. Publication No. 2018 / 0143438, titled "MULTILAYER LIQUID CRYSTAL DIFFRACTIVEGRATINGS FOR REDIRECTING LIGHT OF WIDE INCIDENT ANGLE RANGES", filed on November 16, 2017; U.S. Publication No. 2018 / 0143485, titled "SPATIALLY VARIABLE LIQUID CRYSTAL DIRRRACTION GRATINGS", filed on November 16, 2017; U.S. Publication No. 2018 / 0143509, titled "WAVEGUIDELIGHT MULTIPLEXER USING CROSSED GRATINGS", filed on November 16, 2017; U.S. Publication No. 2018 / 0239147, titled "DISPLAY SYSTEM WITH VARIABLEPOWER REFLECTOR", filed on February 22, 2018; U.S. Publication No. 2018 / 0239177, titled "VARIABLE-FOCUS VIRTUAL IMAGE DEVICES BASED ONPOLARIZATION CONVERSION", filed on February 22, 2018; and U.S. Publication No. 2018 / 0164627, titled "DIFFRACTIVE DEVICES BASED ONCHOLESTERIC LIQUID CRYSTAL", filed on December 7, 2017. However, the design of the optical element 942 for coupling in and / or the optical element 944 for coupling is not limited to these and may include other types of optical elements, diffractive optical elements, liquid crystal optical elements, liquid crystal gratings, and liquid crystal polarization gratings. More information regarding examples of cholesteric liquid crystal structures for, e.g., reflectors, can also be found in the section below titled "Cholesteric Liquid Crystal Mirrors". As described above, other liquid crystal optical elements as well as other non-liquid crystal optical elements may be used.Accordingly, many types of coupling optical elements (e.g., the light-coupling-in optical element 942 and / or the light-coupling optical element 944), diffractive optical elements, gratings, polarization gratings, etc. can be used, typically those described herein as well as other types of gratings, diffractive optical elements, liquid crystal elements, and optical elements. In various implementations, the light-coupling-in optical element 942 can be configured to couple light from the image projector 930 and / or the light source 960 into the waveguide at an angle greater than the critical angle so as to be guided within the waveguide 940 to the user's eye 210 of the user by total internal reflection.

[0102] The waveguide 940 can include one or more waveguides. In some implementations, the one or more waveguides 940 include a stack of waveguides. For example, in some designs, different waveguides in the waveguide stack are configured to output light with different wavefront divergences, as if projected from different distances from the user's eye. For example, the first waveguide or group of waveguides can be configured to output collimated light or light with a first divergence, as if projected from a first depth, and the second waveguide or group of waveguides can be configured to output divergent (un-collimated) light or light at a second divergence (greater than the first divergence), as if projected from a second depth closer than the first depth. In some designs, different waveguides can be configured to output light with different associated colors. For example, the first waveguide can be configured to output red light, the second waveguide can be configured to output green light, the third waveguide can be configured to output blue light. The fourth waveguide can be configured to output and / or input infrared light.

[0103] For example, as Figure 10 shown, the light-coupling-out optical element 952 for coupling light from the waveguide 940 to the camera 920 can include, for example, a plurality of turning features configured to turn the light incident thereon at an angle such that the light is not guided within the waveguide and causes the light to turn out of the waveguide to the camera. The light-coupling-out optical element 952 can be disposed inside the waveguide 940 or can be patterned (e.g., etched) within or on the surface (e.g., the main surface) of the waveguide 940. For example, a surface holographic or diffractive optical element (e.g., a surface relief grating) can be fabricated by patterning (e.g., etching) the surface of the waveguide or a layer thereon. A volume holographic or diffractive optical element can also be formed by changing the refractive index of a material including the waveguide or a layer disposed thereon. Depending on the design, the light-coupling-out optical element 952 can be transmissive or reflective and can operate in a transmissive or reflective manner. For example, the light-coupling-out optical element 952 can include, for example, a transmissive or reflective diffractive optical element (e.g., a grating) or a holographic optical element that operates in a transmissive or reflective manner, respectively, so as to turn the light transmitted through or reflected from it.

[0104] The output optical element 942 may include a reflective optical element (e.g., a mirror). For example, the output optical element 952 may include an off-axis reflector. In some designs, the output optical element 952 may include a polarization optical element, such as a polarization-selective steering element (e.g., a polarizer). Thus, the polarization-selective steering element may include one or more polarization gratings, diffractive optical elements, and / or holographic optical elements, and may include a liquid crystal structure such as a liquid crystal polarization grating. In some implementations, for example, the output optical element 952 may include a liquid crystal polarization grating (LCPG). The LCPG may potentially provide efficient diffraction at a wide wavelength. Similarly, the LCPG may be used for the output optical element 952. The LCPG may be polarization-dependent. The LCPG or other types of liquid crystal gratings may include a pattern or arrangement of liquid crystal molecules configured to provide one or more functions, such as turning light into or out of a waveguide. Thus, the output optical element 952 may include a polarization grating. Additionally or alternatively, the output optical element 952 may include a liquid crystal and, thus, in some implementations, may be a liquid crystal grating or other liquid crystal optical element such as a liquid crystal diffractive optical element. Additionally or alternatively, the output optical element 952 may include a blazed grating. In some designs, the output optical element 952 includes a liquid crystal reflector, such as a cholesteric liquid crystal reflecting lens (e.g., a reflective liquid crystal diffractive lens, a Bragg reflection structure, a reflective liquid crystal diffraction grating, etc.).Some non-limiting examples of liquid crystal gratings, liquid crystal polarization gratings, and other liquid crystal optical elements are discussed in the following published applications, the entire contents of which are hereby incorporated by reference for all purposes: U.S. Publication No. 2018 / 0143438, filed November 16, 2017, titled "MULTILAYER LIQUID CRYSTAL DIFFRACTIVE GRATINGS FOR REDIRECTING LIGHT OF WIDE INCIDENT ANGLE RANGES"; U.S. Publication No. 2018 / 0143485, filed November 16, 2017, titled "SPATIALLY VARIABLE LIQUID CRYSTAL DIRRRACTION GRATINGS"; U.S. Publication No. 2018 / 0143509, filed November 16, 2017, titled "WAVEGUIDE LIGHT MULTIPLEXER USING CROSSED GRATINGS"; U.S. Publication No. 2018 / 0239147, filed February 22, 2018, titled "DISPLAY SYSTEM WITH VARIABLE POWER REFLECTOR"; U.S. Publication No. 2018 / 0239177, filed February 22, 2018, titled "VARIABLE-FOCUS VIRTUAL IMAGE DEVICES BASED ON POLARIZATION CONVERSION"; and U.S. Publication No. 2018 / 0164627, filed December 7, 2017, titled "DIFFRACTIVE DEVICES BASED ON CHOLESTERIC LIQUID CRYSTAL". However, the design of the outcoupling optical element 952 is not limited to these and may include other types of optical elements, diffractive optical elements, liquid crystal optical elements, liquid crystal gratings, and liquid crystal polarization gratings. More information regarding examples of cholesteric liquid crystal structures for reflectors, for example, can also be found in the section below titled "Cholesteric Liquid Crystal Mirrors". As described above, other liquid crystal optical elements as well as other non-liquid crystal optical elements may be used. Thus, many types of coupling optical elements (e.g., the outcoupling optical element 952), diffractive optical elements, gratings, polarization gratings, etc. may be used, generally those described herein as well as other types of gratings, diffractive optical elements, liquid crystal elements, or optical elements.As described above, the out-coupling optical element 952 can be configured to redirect light guided within the waveguide 940 at an angle less than the critical angle so as to be emitted into the camera 920 rather than being guided within the waveguide by total internal reflection.

[0105] In various designs, the coupling optical element 944 can be transparent in the visible spectrum such that a user can see the environment in front of the user through the coupling optical element 944 and the eyepiece 950. For example, if the in-coupling optical element 942 is used to receive light from the image projector 930 and / or if the illumination source 960 is configured to output visible light to illuminate the eye 210 with visible light, the in-coupling optical element 942 can also redirect light in the visible spectrum. In some embodiments, for example, if the illumination source 960 is configured to output infrared light to illuminate the eye 210 with infrared light, the in-coupling optical element 942 is configured to redirect the infrared light. In some designs such as Figure 10 shown, the in-coupling optical element 942 can be more medial or nasal than the out-coupling optical element 952. However, in other designs, the in-coupling optical element 942 can be more lateral or temporal than the out-coupling optical element 952. In some implementations such as Figure 10 shown, the out-coupling optical element 952 can be adjacent to the in-coupling optical element 942, although non-adjacent positioning is possible.

[0106] As Figure 10 shown, the illumination source 960 can be disposed on the same side (e.g., rearward or proximal) of the eyepiece 950 as the eye 210. (Proximal can refer to the side closest to the eye 210.) Optionally, the illumination source 960 can be disposed on the side opposite the eye 210 (e.g., forward or distal side). The illumination source 960 can be configured to direct light into at least one of the major surfaces of the waveguide 940 via the in-coupling optical element 942. The light source 960 can be configured to emit invisible light (e.g., infrared). The light source 960 can include one or more LEDs. The LEDs can include infrared LEDs. The light source 960 can be configured to emit coherent light. In some designs, the light source 960 includes a laser (e.g., an infrared laser). In some designs, the light source 960 emits pulsed light. For example, the camera 920 can be configured to periodically capture images. Thus, the illumination source 960 can be pulsed to coincide with the period during which the camera acquires images. When the camera is not acquiring an image, the intensity output from the illumination source 960 can be reduced. By concentrating the total energy of illumination in a short period of time, an increased signal-to-noise ratio can be obtained without exposing the eye 210 to unsafe intensity levels. In some cases, for example, the camera 920 captures an image every 30 milliseconds and the exposure time of the camera is a few milliseconds. The illumination source 960 can be configured to output pulses having a similar period and duration to match the pulses of the camera 920.

[0107] In some implementations, different light sources having different wavelengths are pulsed alternately to provide illumination of different wavelengths at different times, as described below.

[0108] The light-coupling-in optical element 942 can be in direct optical communication with, for example, the illumination source 960 and / or the image projector 930 to direct light from the image projector 930 and / or the light source 960 therethrough. For example, the light emitted by the light source 960 can be incident on the light-coupling-in optical element 942 before optically interacting with the coupling optical element 944 and / or the coupling optical element 952.

[0109] As Figures 11A - 11E shown, the light 902 projected from the image projector 930 can form an image on the retina. The image projector 930 can include a light source, a modulator, and / or projection optics. The light source for the image projector 930 can include one or more LEDs, lasers, or other light sources, and can include one or more visible light sources. The modulator can include a spatial light modulator, such as a liquid crystal spatial light modulator. Such a spatial light modulator can be configured to modulate the intensity of light at different spatial positions, for example. The projection optics can include one or more lenses. Other types of image projectors 930 capable of projecting and / or forming an image can be employed. For example, the image projector 930 can include a scanning optical fiber.

[0110] The image projector 930 and the light-coupling-in optical element 942 can be in direct optical communication with each other. The image projector 930 can be aligned, for example, with the light-coupling-in optical element 942 into which the light from the image projector 930 is directed. In some cases, the image projector 930 is disposed adjacent to the corresponding light-coupling-in optical element 942 and / or the waveguide 940. The image projector 930 can also be disposed in the optical path including the light-coupling-in optical element 942, the coupling optical element 944, and the eye 210.

[0111] The image projector 930 can be a separate element compared to the illumination source 960, as Figure 10 and Figures 11A - 11E shown. However, in some cases, the image projector 930 can be used as an illumination source. For example, in addition to injecting an image into the eye 210, the image projector 930 can be used to direct visible light and / or infrared light into the eye to illuminate the eye for image capture. However, optionally, one or more separate light sources 960 can be used to illuminate the eye 210 for image capture.

[0112] The light emitted by the illumination source 960 may include light in a specific wavelength range, such as invisible light. The illumination source 960 may be configured to project invisible (e.g., infrared) light onto / into the eye 210 to image one or more parts of the eye 210 (e.g., the cornea, the retina). In some example implementations, the light source 960 may be configured to emit light in the range of about 850 nm to 940 nm. The light source 960 may be configured to emit light extending over a wavelength range of at least about 20 nm. Other ranges are possible. The emitted wavelength range may be 5 nm, 10 nm, 15 nm, 50 nm, 75 nm, 100 nm, 150 nm, 200 nm, or any range between any of these values. The light source 960 may be configured to emit light having a broadband wavelength across any range within, for example, the infrared spectrum.

[0113] The imaging device 920, which may include a camera, may include a detector array and possibly imaging optics. The detector array may include, for example, a CCD or CMOS detector array, and the imaging optics may include one or more lenses. The one or more lenses may have a positive optical power and an associated focal length. In some designs, the camera 920 is focused at infinity. For example, the optics may have a focal length f, and the detector array may be disposed at a distance from the optics corresponding to the focal length such that an object at a large distance is imaged onto the detector array. Similarly, collimated light from an object in the eye or the environment will be focused onto the detector array to form an image of the eye or the object thereon.

[0114] The imaging device 920 may be disposed on the side of the waveguide 940 opposite the illumination source 960 and / or the eye 210. In some designs, the imaging device 920 may be disposed on the same side of the waveguide 940 as the illumination source 960 and / or the eye 210. As Figure 10 shown, the imaging device 920 may be disposed near the outer or temporal edge of the eyepiece 950, although other locations are possible.

[0115] Figures 11A - 11E is shown Figure 10 the operation of an example imaging system 900. Figure 11A shown is the illumination source 960 emitting light 902 towards the light-coupling optical element 942 on the waveguide 940. As shown, the light 902 may generally be directed to the eyepiece 950 at normal incidence, although other angles are possible. In some designs, the light source 960 is configured to emit collimated light into the eyepiece 950. As Figure 11BAs shown, the illumination light 902 can be coupled into the waveguide 940 via the input coupling optical element 942. In some designs, the input coupling optical element 942 includes a diffractive optical element (e.g., a grating, a holographic element), and the light incident thereon is diffracted at an angle greater than the critical angle of the waveguide so that the coupled-in light 904 is guided within the eyepiece 950 by total internal reflection (TIR). In some designs, the input coupling optical element 942 can be configured to direct light toward the coupling optical element 944. The input coupling optical element 942 can be polarization selective. For example, the input coupling optical element 942 can include a polarization selective steering element, such as a polarization grating, e.g., a liquid crystal polarization grating. Figure 11C Shows how the coupled-in light 904 propagates through the waveguide 940 by TIR.

[0116] Figure 11D An exemplary imaging system 900 that couples out light from the eyepiece 950 is shown. As the coupled-in light 904 propagates through the waveguide 940, some of the light can be incident on the coupling optical element 944. The coupling optical element 944 can be configured to couple the coupled-in light 904 out of the eyepiece 950 and toward the user's eye 210. The coupling optical element 944 can be configured to couple light, which is collimated light, toward the eye 210. The coupling optical element 944 can be tuned for light within a specific wavelength range. For example, the coupling optical element 944 can be configured to couple out infrared light (e.g., between approximately 700 nm and 15000 nm) from the waveguide 940. In some designs, the coupling optical element 944 can be configured to couple out light of multiple wavelengths from the eyepiece 950. For example, the coupling optical element 944 can be tuned with respect to both infrared and visible light. The coupling optical element 944 can also be configured to couple light into the waveguide 940, as described more fully below.

[0117] The coupling optical element 944 can be configured to increase one or more dimensions of the eye box for the user. For example, one or more dimensions can be measured along a first axis (e.g., the x-axis). The eyepiece 950 can further include an orthogonal pupil expander (OPE). The OPE can have at least one light redirecting element disposed above or within the waveguide (e.g., on one of the major surfaces), or the OPE can be disposed within the waveguide 940. The OPE can include features similar or identical to those described above with respect to the light distribution elements 730, 740, 750. In some implementations, the light redirecting element can include a diffractive optical element. The OPE can be configured to increase the size of the eye box along a second axis (e.g., the y-axis) orthogonal to the first axis.

[0118] Figure 11DShows some of the light leaving the eyepiece 950 towards the user's eye 210. In some designs, the coupling optical element 944 is configured such that the coupled-in light 904 incident on the coupling optical element 944 at various portions of the coupling optical element along a first axis (e.g., parallel to the x-axis) exits the eyepiece 950 at each portion of the coupling optical element 944 along the first axis. This can provide the user with light for projecting an image or illuminating the eye for different eye positions or locations.

[0119] As Figures 11D - 11E shown, the coupling optical element 944 can be configured to couple out the coupled-in light 904 from the eyepiece 950 as collimated light. This light can also be directed substantially normal to the main surface of the eyepiece 950 and / or the waveguide 940. The collimated light can be directed into the eye and focused by the eye (e.g., the cornea and natural lens of the eye) onto the retina. The light 908 incident on the retina can provide illumination for imaging the retina and / or providing image content to the eye. Some of this light 908 can, for example, be reflected or scattered off the retina, exit the eye, and provide a retinal image to be captured. The light source 960 can be an extended light source such that the light will illuminate an area of the retina.

[0120] Figures 12A - 12E Shows Figures 11A - 11E how the imaging system 900 can additionally or alternatively be used for image collection of the eye 210. Figure 12A Shows the light 910 leaving the eye 210 and reflected from the retina. As shown, the light 910 scattered or reflected from the retina and passing through the natural lens of the eye, the pupil in the eye, and the cornea can be collimated. This light can also be incident on the eyepiece 950 at normal incidence (e.g., at a right angle to the main surface of the waveguide 940 and / or the coupling optical element 944). The coupling optical element 944 can be configured to couple the light 910 reflected from the retina into the waveguide 940.

[0121] Figure 12BAn exemplary imaging system 900 that couples light into an eyepiece 950 is shown. The coupling optical element 944 can include steering features such as diffractive optical elements or other structures that redirect light at an angle greater than the critical angle to guide the light within the waveguide 940. The coupling optical element 944 can be configured to direct the coupled-in light 914 generally toward the light source 960 and / or the imaging device 920. The coupling optical element 944 can be configured to couple out less than a small fraction of the light that propagates toward the camera 920 from the waveguide 940. For example, a partially reflective element (e.g., a semi-transparent mirror) can be disposed above or within the waveguide 940 such that a portion of the coupled-in light 914 continues to propagate in the waveguide 940 by total internal reflection while reducing leakage of the coupled-in light 914 from the waveguide 940 along the portion of the waveguide 940 where the coupling optical element 944 is disposed. The fraction of the light that does not leak out can be any fraction between 0 and 1. For example, the fraction can be 0.90, where 90% of the light rays that propagate through the waveguide 940 along the coupling optical element 944 are kept within the waveguide at each reflection of the light rays. Other fractions are possible (e.g., 0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80, or a range between any of these values). Such a partially reflective element can be similarly used in the implementations described below.

[0122] As Figure 12C shown, the collimated coupled-in light 914 can continue to propagate through the waveguide 940 toward the imaging device 920. Figure 12D It is shown how some of the coupled-in light 914 continues to propagate until it impinges on one or more out-coupling optical elements 952. To reduce the amount of leakage of the coupled-in light 914 from the in-coupling optical element 942, the in-coupling optical element 942 can be configured to couple out very little of the light that propagates toward the camera 920 from the waveguide. For example, a partially reflective element (e.g., a semi-transparent mirror) can be disposed on or within the waveguide 940 such that a portion of the coupled-in light 914 continues to propagate within the waveguide 940 by total internal reflection while reducing leakage of the coupled-in light 914 from the waveguide 940 along the portion of the waveguide 940 where the in-coupling optical element 942 is disposed. The fraction of the light that does not leak out can be any fraction between 0 and 1. For example, the fraction can be 0.90, where 90% of the light rays that propagate through the waveguide 940 along the coupling optical element 944 are kept within the waveguide at each reflection of the light rays. Other fractions are possible (e.g., 0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80, or a range between any of these values). Such a partially reflective element can be similarly used in the implementations described below.

[0123] As Figure 12EAs shown, the output optical element 952 can be configured to couple out the light guided in the waveguide 940 from the waveguide 940 and couple it to the imaging device 920. As a result, the light propagating in the waveguide 940 incident on the output element 952 is redirected so that it exits the waveguide 940, for example, from the main surface of the waveguide 940 (e.g., the front or rear side of the waveguide 940) and is guided to the imaging device 920. The output optical element 952 can be configured to guide the light 926 to exit the waveguide 940 perpendicular to (e.g., normal to) the main surface of the waveguide 940. In some designs, the output optical element 952 is configured to guide the collimated light 924 to the imaging device 920 at a normal incidence to the photosensitive portion of the imaging device 920. As described above, the camera 920 can be focused at infinity. For example, the imaging optics can be configured to focus the collimated light onto the detector array.

[0124] Accordingly, the waveguide 940 can be configured to guide the light coupled from the user's eye 210 into the waveguide 940 to be received by the imaging device 920 (e.g., a camera) so as to capture an image of at least a portion of the user's eye 210. The same waveguide 940 can be configured to guide the light coupled from the image projector 930 such that the light from the image projector 930 can be guided to the user's eye 210 so that the image from the image projector 930 is located in the user's field of view. In some implementations, the same waveguide is configured to guide the light coupled from the illumination source 960 such that the light from the illumination source can be guided to the user's eye 210 to illuminate the eye so that an image of the eye can be captured by the camera 920.

[0125] In some implementations, the same coupling optical element 944 can be configured to (i) couple the light from the user's eye 210 into the waveguide 940 to be received by the imaging device 920, and (ii) couple out the light from the image projector 930 from the waveguide 940 to the user's eye 210 to project the image content into the user's field of view. In some implementations, the same coupling optical element 944 can be configured to couple out the light from the illumination source 960 from the waveguide to the user's eye 210 such that the light from the illumination source can illuminate the eye.

[0126] In other designs, different waveguides may be used and / or different coupling optical elements 944 may be used. For example, in some designs, the first waveguide 940 may be configured to direct light coupled from the user's eye 210 to be received by the camera 920, thereby capturing an image of at least a portion of the user's eye 210. The second waveguide may be configured to direct light coupled from the image projector 930 such that the light from the image projector 930 can be directed to the user's eye 210. The first waveguide and the second waveguide may be stacked on top of each other. Additionally or alternatively, another waveguide may be configured to direct light coupled from the illumination source 960 such that the light from the illumination source can be directed to the user's eye 210 to illuminate the eye.

[0127] Additionally, in some implementations, the first coupling optical element 944 may be configured to (i) couple light from the user's eye 210 into the waveguide 940 to be received by the imaging device 920, and (ii) couple out light from the image projector 930 from the waveguide 940 to the user's eye 210 to project image content into the user's field of view. Additionally or alternatively, another coupling optical element may be configured to couple out light from the illumination source 960 from the waveguide to the user's eye 210 such that the light from the illumination source can illuminate the eye.

[0128] In some designs, the coupling optical element 944 may include a plurality of diffractive optical elements (DOEs). For example, the first DOE may be configured to couple light from the user's eye 210 into the waveguide 940 to be received by the imaging device 920. The second DOE may be configured to couple out light from the image projector 930 from the waveguide 940 to the user's eye 210 to project image content into the user's field of view. Optionally, the third DOE may be configured to couple out light from the light source 960 from the waveguide 940 to the user's eye 210 to illuminate the eye. The first and second (and possibly third) DOEs may be stacked such that, for example, in some implementations, light from the user's front environment passes through the first DOE, then impinges on the second DOE, then impinges on the third DOE and onto the user's eye. However, the order may be different.

[0129] In some designs, the first DOE and the second DOE are integrated in a single element or volume of the waveguide 940. In some implementations, for example, both the first DOE and the second DOE are stacked on top of each other within the waveguide 2102 (e.g., occupy the same or approximately the same volume). For example, the first and second DOEs may be recorded in the same medium.

[0130] As described above, image capture of an eye such as the retina can facilitate eye tracking. For example, Figure 13AIllustrated is an imaging system 900 configured to image various parts (e.g., the retina) of an eye 210 at different times, such as when the eye is in different positions. Phases A and B may refer to images of the eye 210 during different orientations of the eye. Figure 13A Illustrated is the imaging of the eye 210 and its results during both phase A and phase B imaging.

[0131] In some implementations, light emission 928 (e.g., from the illumination source 960 as described above or from one or more illumination sources configured and / or positioned differently) may be used to obtain one or more images of the retina 962, as Figure 13A shown. The images of the retina 962 may include one or more regions 964, 966 imaged during different orientations of the eye 210. Figure 13A Illustrated are two regions 964, 966 of an image of the retina 962. For example, while the eye 210 is directed at an angle normal to the waveguide 940, the region 964 of the retina imaged in phase A may be imaged. While the eye 210 is oriented at an acute angle to the waveguide 940, image data of the region 966 of the retina imaged in phase B may be obtained. By using one or more orientations of the eye 210 during one or more stages of imaging, a composite image or map of the retina 962 may be obtained. Processing electronics or a processor such as the data module 140 (see Figure 2 ) may be used to find overlapping image data between two adjacent regions. Using the overlapping regional image data, a composite image of the retina 962 may be determined. A composite image or map of a larger size (e.g., full size) of the user's retina may be stored.

[0132] As described herein, a head-mounted display may be used to map a user's eye retina based on the direction in which the user's eyes are directed. To provide realistic and intuitive interaction with objects in the user's environment using eye gaze and / or to identify the wearer of a head-mounted display device, the head-mounted display system may use retina mapping in combination with the uniqueness of the user's eye characteristics and other conditions that may have some effect on eye measurements. For example, an image may be identified based on the position of blood vessels in the corresponding retina image.

[0133] Retina mapping may involve a process that enables a computing device to learn how to associate a user's eye gaze (e.g., as identified in a retina image) with a gaze point in 2D or 3D space. Eye gaze may be associated with a single point in 2D or 3D space. Eye gaze may also be associated with multiple points in space, which may describe the movement of a virtual object (e.g., a series of points, the positions of a moving image).

[0134] A head-mounted display system can determine a user's eye gaze based on a retinal image. The head-mounted display system can obtain a retinal image using a sensor (e.g., an eye camera such as imaging device 920). The head-mounted display system can image one or both of the user's eyes when the user changes his or her eye gaze (e.g., when the user looks around to follow a moving or shifting calibration target or fixation target). To map the user's retina, the head-mounted display system can present a virtual target, such as a fixation target, for the user to view. The virtual target can be associated with one or more known gaze points in 2D or 3D space. When the user fixates on the target, the head-mounted display system can acquire the retinal image and associate the image with the gaze point. The head-mounted display system can calculate and / or generate a mapping matrix based on the association of the respective retinal images and the gaze points associated with the target.

[0135] The retinal mapping result can reflect the uniqueness in each person's eyes. For example, the head-mounted display system can generate a mapping matrix customized for one or both eyes of a specific individual. For example, in response to a specific target, a user may have a different number of eye movements or eye gazes. Additionally or alternatively, the user can have different positions, sizes, shapes, and / or orientations of blood vessels in the retina. As a result, by generating calibration results specific to an individual user, the head-mounted display system can allow for more accurate user interaction with eye gaze and / or can allow for the identification of a specific user.

[0136] Thus, when a user wears the head-mounted display device, the system can detect whether the user is a previous user or a new user. A confusion matrix can be calculated, where the score of a specific eye gaze image stored in the system memory is compared with the corresponding image of the current user. The confusion matrix can include comparison scores for multiple eye gazes and associated retinal images. Based on the comparison scores, the system can be able to make a determination about the identity of the user (e.g., whether the user is associated with the stored retinal image or composite image) and / or the confidence level of the determination. The confidence level can include, for example, an identity coefficient. The stored image (e.g., composite image or map) can be compared with a later obtained image (referred to as an immediate or real-time image obtained for the current user). If the system detects that the user is a new user, the system can provide an alert or can take other measures.

[0137] The system can apply filtering such as digital filtering or mapping image processing to an image of the retina captured by a camera. Such filtering or imaging processing can enhance, for example, features that can be used for identification, stitching, combining composite images, eye tracking, etc. Such filtering or mapping image processing can include edge enhancement. Although other types of filters can be used, such a filter can include, for example, a Frangi filter. Such a filter or processing (e.g., edge enhancement or Frangi filter) can be used to enhance and / or detect image features such as blood vessels or tubular structures or fibers in a retinal image.

[0138] Figure 13B A pattern of fixation targets shown in sequential display that can be used during the retinal mapping process is presented. These virtual targets to which the user's eyes are to direct their gaze can cause the eye gaze to be redirected to various different directions, during which the retina can be imaged. The resulting images associated with different gaze directions correspond to different parts of the retina. As described above, when the eyes gaze in different directions to view fixation targets at different positions on the display, the images captured by the camera include different parts of the retina. These images can be combined to form a larger retinal map or composite image.

[0139] Figure 13B Virtual targets at sixteen different positions in the user's field of view (FOV) 1200 are shown. In various implementations, the virtual targets will be presented at a given location at a given time. During the time when the virtual target is presented to the user at that particular location, one or more retinal images will be obtained. The image or images can be associated with that target location and / or the corresponding gaze direction. More or fewer target locations can be used. In Figure 13BIn the example shown, sixteen target locations 1202a–1202p are shown. More or fewer target locations may be used. The target locations may also be different. The order in which the targets are shown at different locations may be different. For example, the targets may move from the left side of the user's field of view to the user's field of view in a raster pattern, return from right to left, and then move from left to right, thereby lowering the position of the targets in the field of view, where each horizontal channel spans the field of view. However, other patterns and methods are possible. Similarly, the targets may be rendered the same or differently at different locations. For example, the rendered targets may be of different sizes, shapes, colors, etc. The targets may be sequentially rendered to the user during the eye tracking calibration process. For example, as described above, the head-mounted display system may render the targets in a serpentine pattern. For example, after target 1202a may be 1202b, then 1202c, then 1202d, then 1202h, then 1202g, and so on. Other patterns are possible. For example, the targets may be shown in a more random or non-sequential pattern. In some embodiments, a single target is shown to the user and the target moves around the user's field of view (e.g., passing through or pausing temporarily at locations 1202a-1202p during target movement). The head-mounted display system may acquire an image of the user's retina when the user is looking at these targets. For example, when the user is looking at the target at the first location 1202a, the head-mounted display system may acquire a first image, and when the user is looking at the target at the second location 1202b, a second image may be acquired, and when the user is looking at the target at the third location 1202c, a third image may be acquired, and so on. The wearable system may associate the first image with the first location 1202a, the second image with the second location 1202b, the third image with the third location 1202c, and so on. Adjacent images may be stitched together in a database to create a complete or partial retinal map. For example, features or parts of features (e.g., blood vessels or parts thereof) common to multiple images may be used to stitch two images together with appropriate registration. In various implementations, adjacent target locations will produce overlapping images that can be aligned and stitched together. For example, target location 1202a and target location 1202b and target location 1202b and target location 1202c may produce overlapping and adjacent retinal images that can be stitched to each other. Thus, many different retinal images may be obtained in different eye gaze situations, thereby combining a larger image of the retina (e.g., a composite image or map).

[0140] As described above, synthetic retinal images or maps can be used to perform eye tracking. For example, after a target is no longer displayed, when the user looks at different real objects located in front of the user and the head-mounted display or the augmented reality (virtual) image content displayed by the head-mounted display, the user can move their eye gaze. One or more retinal images can be obtained at these times. The terms "instantaneous" or "real-time" images can be used herein to describe these images obtained after calibration, which can be used for eye tracking (or other purposes, such as obtaining biometric data). These "instantaneous" or "real-time" images may correspond to a part of the synthetic retinal image or map. The system can be configured to sufficiently match such "instantaneous" or "real-time" retinal images with a part of the composite retinal image or retinal map. Such a match can be based on features or parts of features (blood vessels or parts thereof) common to the "instantaneous" or "real-time" retinal image and a part of the composite retinal image or map. Based on the position where the "instantaneous" or "real-time" retinal image coincides with a part of the synthetic retinal image or map, the gaze direction can be inferred. Different gaze directions will result in retinal images corresponding to different parts of the retinal map. Therefore, identifying the position of the "instantaneous" or "real-time" retinal image on the synthetic retinal image or map will provide information about the user's gaze direction. Such or similar methods can be used to perform eye tracking, for example, tracking the movement of the eyes and changes in eye gaze. As described above, edge enhancement, edge detection, or other digital filtering and / or processing can be used to enhance the features of different images and / or correlate the features of different images with the synthetic retinal image or retinal map.

[0141] In various implementations after completing an initial calibration process in which a virtual target or fixation target (e.g., at multiple locations) is displayed to combine the synthetic retinal image or map, the synthetic retinal image or map can still be refined. For example, when additional retinal images are obtained, the additional images can be used to further refine or improve the synthetic retinal image or map. Thus, when additional "instantaneous" or "real-time" retinal images are obtained, for example, for the purpose of providing eye tracking, the "instantaneous" or "real-time" images can be used to further refine or improve the synthetic retinal image or map. When the user continues to look at various positions in the display (with or without a calibration target), additional images obtained after the initial calibration in which a virtual target or fixation target is displayed can be used to further refine the synthetic retinal image or map. Thus, the quality of the synthetic retinal image or map can be increased.

[0142] Examples of how eye tracking can be accomplished and / or how synthetic retinal images or maps can be generated, as well as additional non-limiting examples of retinal images used, are described in U.S. Publication No. 2017 / 0205875, titled "EYE IMAGE COLLECTION," filed on January 17, 2017, the disclosure of which is incorporated herein by reference in its entirety.

[0143] Thus, as described above, a relatively large portion of the retina can be recorded and mapped by obtaining retinal images and / or other images of the eye using an imaging system such as those described herein, and such images can facilitate eye tracking. For example, when the eye is in any position, an Figure 13A image of the eye 210 as shown can be captured. Then, processing electronics or a processor (e.g., the same or different processing electronics or processor as described above for forming synthetic images) can compare the captured image of the user's retina in real time with a stored synthetic or larger-sized (e.g., full-sized) image to track the movement of the eye. A given image of the user's retina captured in real time may show a particular portion of the user's retina. As described above, by comparing such a captured image with a stored image that maps a relatively large portion of the user's retina, the system can determine which portion of the user's retina is shown in the captured image, and thus can determine the position / orientation of the eye that produced such an image. See FIG. Example 13A, which shows images of two different retinal portions produced when the eye is in two different positions and / or orientations. Thus, the position and / or orientation of the eye can be determined by capturing different images of the retina and determining which portion of the retina is visible. This determination can be performed even if no synthetic image is formed but rather multiple images of the retina at different eye positions / orientations are recorded and stored in a database. When a future image of the retina is obtained, the image can be compared with the images in the stored image database to determine which image in the database is similar to the most recently obtained image of the eye. Matching the most recent image with one or more of the images in the database that have an associated position and / or orientation can enable determination of the orientation and / or position of the newer image. Other eye tracking methods can be used based on images captured using the designs described herein.

[0144] As described herein, retinal images can also be used for other purposes. For example, retinal images can be used to verify that the user for whom the synthetic retinal image or mapping was obtained is the same user. A retinal image obtained while the user is wearing a head-mounted display system (e.g., during a calibration process and / or during subsequent use) can be compared to a previously obtained synthetic retinal image or mapping that is stored (e.g., created the previous day or when the head-mounted display was previously activated). If the recently obtained retinal image does not match a portion of the synthetic retinal image or mapping, it can be concluded that the current user is different from the previous user (e.g., for whom the synthetic virtual image or mapping was created). Such a method can be used for security purposes, e.g., to verify that the current user of the head-mounted display device is the owner or typical user of the device. Thus, biometric data obtained via retinal imaging can be used for security purposes.

[0145] Retinal imaging can also be used to collect biometric data for monitoring a user's health. Medically relevant data can be obtained from a retinal image. Such medical data may be useful for monitoring a user's health.

[0146] Although various applications of eye imaging, such as eye tracking, collection of biometric data for health monitoring and security, are discussed herein in the context of retinal imaging, other parts of the user (e.g., the user's eyes) can be imaged for these and other purposes.

[0147] Although the above describes that the eyepiece 950 can be used to facilitate imaging of the eye, the eyepiece can also be used to image the world in front of the user. For example, Figures 14A - 14B An example imaging system 900 is shown that can be used to image a portion of the environment located in front of the user and / or an object in a portion of the environment. The imaging system 900 used can be a system similar to that described with respect to Figures 11A - 11E and / or with respect to Figures 12A - 12E except that light from the environment toward the eyepiece and the user is collected by the eyepiece 950. For example, Figure 14A Light 970 from the environment is shown that is reflected and / or emitted by one or more physical objects 972 in the environment and toward the user and the eyepiece 950. As shown, the light 970 from the environment can be approximately collimated (e.g., at infinity), e.g., because the physical objects 972 in the environment may be located at a sufficiently large distance from the imaging system 900 such that the light rays reaching the imaging system 900 are collimated or approximately collimated. In some implementations, the imaging system 900 can be configured to image the environment and / or an object in the environment without using any optical elements (e.g., lenses, mirrors) having a focal power in the imaging system 900.

[0148] Figure 14Aand 14B The imaging system 900 shown is similar to the above-described imaging system. The imaging system includes an eyepiece 950 that includes one or more waveguides 940, and the one or more waveguides 940 include coupling optical elements 944 configured to direct light from an image projector 930 (not shown) into the eye 210 to form an image therein. The one or more waveguides may include a plurality of waveguides (e.g., a stack of waveguides) configured to couple in / out a plurality of corresponding colors / wavelengths. Each waveguide in the waveguide stack may be configured to direct light of a specific color (e.g., red, green, blue). For example, the outermost waveguide (e.g., the stack of waveguides) may be configured for visible light (e.g., red, blue, green) such that the waveguide is configured to couple in and out visible light of the same wavelength. Additionally or alternatively, waveguides configured to couple in and out invisible (e.g., infrared) light may be provided near the eye 210. Such a plurality of waveguides corresponding to the waveguide 940 may be used in any other implementation described herein. The imaging system 900 may also include an imaging device (e.g., a camera) 920 and an outcoupling optical element 952 configured to direct light reflected from the eye 210 and propagating within the waveguide 940 to the camera. In Figure 14A and 14B , an illumination source 960 is excluded because an illumination source may not be needed to image the environment in front of the user. However, an illumination source (e.g., the above-described light source 960) may be used in some designs.

[0149] The eyepiece 950, waveguide 940, coupling optical element 944, output optical element 952, and camera 920 may be the same as or similar to those described above. For example, the coupling optical element 944 may be physically joined to the waveguide 940. For example, the coupling optical element 944 and / or the output optical element 952 may be disposed in the optical path between the environment in front of the eyepiece 950 and the camera 920 such that light from the environment is coupled into the waveguide 940 via the coupling optical element 944 and is output from the waveguide via the output optical element to be incident on the camera 210 (e.g., to form an image of at least a portion of the environment). The coupling optical element 944 may include a plurality of turning features configured to turn light guided within the waveguide out of the waveguide or to turn light incident on the coupling optical element 944 at an angle into the waveguide to be guided therein by total internal reflection. The output optical element 952 may include a plurality of turning features configured to turn the light (from the environment) guided within the waveguide at an angle such that the light is not guided in the waveguide by total internal reflection but is guided out towards the camera. The coupling optical element 944, the output optical element 952, and the turning features associated therewith may be physically joined to the waveguide 940. For example, the coupling optical element 944 and / or the output optical element 952 may include one or more holographic or diffractive optical elements (e.g., surface relief gratings) patterned (e.g., etched) within or on the waveguide 940. The coupling optical element 944 and / or the output optical element 952 may include a layer disposed on the waveguide 940 or may be formed within the waveguide 940. For example, a volume holographic or diffractive optical element may be formed by changing the refractive index of a material including the waveguide or a layer disposed thereon. Thus, the coupling optical element 944 and / or the output optical element 952 may be disposed within the volume of the waveguide 940 or a layer disposed thereon. Depending on the design, the coupling optical element 944 and / or the output optical element 952 may be transmissive or reflective and may operate in a transmissive or reflective manner. For example, the coupling optical element 944 and / or the output optical element 952 may include transmissive or reflective diffractive optical elements (e.g., gratings) or holographic optical elements that operate in a transmissive or reflective manner, respectively, such that the light transmitted therethrough or reflected therefrom is turned. The coupling optical element 944 and / or the output optical element 952 may include polarization optical elements, such as polarization-selective turning elements (e.g., polarizers). The polarization-selective turning element may include one or more polarization gratings, diffractive optical elements, and / or holographic optical elements and may include a liquid crystal structure, such as a liquid crystal polarization grating. In some implementations, the reflective optical element may include a reflector (e.g., a mirror). Other elements of the waveguide 940, for example, may also be similar to those described above.

[0150] Figure 14B is shown Figure 14AOperation of the imaging system 900 shown. Light 970 from the environment is coupled into waveguide 940 by coupling optical element 944. The coupling optical element 944 can be configured to turn collimated light at an angle greater than the critical angle of the waveguide 940 such that at least a portion of the collimated light is guided within the waveguide toward the camera 920 by total internal reflection. The decoupling optical element 952 can be configured to receive at least a portion of the light from the environment in front of the user, at least a portion of which is coupled into the waveguide 940 via the coupling optical element 944 and guided therein. The decoupling optical element 952 can be configured to decouple the coupled-in light from the waveguide 940 to the camera 920 such that an image of the environment can be captured by the camera 920. The environmental image can be passed to processing electronics (e.g., one or more processors), such as data module 140 (see Figure 2 ). The data module 140 can be configured to reproduce a modified image of the environment in an augmented reality context. The processing electronics can communicate with the camera 920 via wired or wireless electrical signals. Additionally or alternatively, the processing electronics can communicate with the camera 920 using one or more remote receivers. The processing electronics can be remotely located (e.g., a cloud computing device, a remote server, etc.).

[0151] Thus, the imaging system 900 can be used to directly image the environment, which can be useful for a variety of reasons. For example, imaging the environment can be used to determine where to place augmented reality image content relative to objects in the environment. For example, imaging the environment can provide the location of a table such that a head-mounted display can render an image of a person standing near the table rather than on or in the table. The described imaging system 900 for imaging the environment can also be used to image the eye 210, such as with respect to Figure 10 , 11A -11E and / or as described in 12A-12E.

[0152] It may be desirable to use the imaging system 900 to image a wide field of view of the environment. Figure 14C The imaging system 900 is schematically shown for collecting light from the environment using a focusing optical element or lens (e.g., refractive optical element 980 (e.g., a wide field of view lens)) located in front of the eyepiece. The refractive optical element 980 can have a positive optical power. The refractive optical element 980 (e.g., a positive lens) converges the collimated light 970 from the environment toward the waveguide 940. Other than Figure 14COther types of lenses than the lens shown. The transmitted light (not shown) may pass through an optical element or lens with a focal power, such as a refractive optical element 990 (e.g., a negative lens) configured to have a negative focal power equal and opposite to that of the refractive optical element 980. The negative lens 990 may have a focal power similar or identical to that of the positive lens 980 to compensate for or cancel out the focal power of the positive lens or a portion thereof. In this way, light from the environment (e.g., the distal end of the waveguide 940) may pass through the negative lens 990, the eyepiece 950, and the positive lens 980, with substantially no net change in the focal power introduced into the eye by the two lenses. The negative lens 990 may be configured to compensate for or cancel out the focal power of the positive lens 980 such that when viewing the environment in front of the eyepiece 950, the user will not experience the focal power of the positive lens. The negative lens 990 will also cancel out the effect of the positive lens 980 to invert the image of an object in the environment in front of the wearer. Although some light rays converge, some light 970 from the environment may be coupled into the waveguide 940 through the coupling optical element 944. The coupled-in light incident on the outcoupling optical element 952 may exit the waveguide 940.

[0153] Implementations may be used outside of an augmented reality environment (e.g., Figures 14A - 14C the implementations described). For example, an imaging system 900 configured to image the environment is intended to be implemented in a wearable device (e.g., glasses (including afocal glasses) or bifocals). Such an imaging system 900 may not require the image projector 930 and / or the light source 960. Additionally or alternatively, such an imaging system 900 may not require the coupling optical element configured for the corresponding image projector 930 and / or the light source 960.

[0154] Implementing such an imaging system 900 may be advantageous for imaging the environment on a viewing screen (e.g., a TV screen, a computer screen) such as a handheld device (e.g., a cellular phone, a tablet). The imaging system 900 may improve video chat capabilities. For example, a viewer seeing a chat partner looking at the screen may appear to be directly located at the viewer. This would be possible because the light captured by the imaging system 900 would be captured in the same area where the user is looking (e.g., opposite the viewing screen but with light captured by a separate outward-facing camera located at a different position).

[0155] In which Figure 14CThe imaging system 900 is also used in an implementation for imaging the eye 210, and the light source 960 and / or the image projector 930 can be configured to inject light into the waveguide 940. Since the light reflected from the eye and coupled into the waveguide will pass through the refractive optical element 990 (e.g., a negative lens), a positive refractive optical element can be provided between the light source 960 and / or the image projector 930 and the waveguide 940. Before the coupled light from the light source and / or the light projector is incident on the eye 210, the positive lens can be configured to compensate for or cancel any optical power provided by the refractive optical element 990. Other types of lenses other than those shown in Figure 14C can be used as the optical element 990. Optionally or additionally, the processing electronics in communication with the light source and / or the image projector can be configured to modify the image to be sufficient to present an undistorted image to the user after the light has passed through the refractive optical element 990. In some designs, the corresponding input optical element, output optical element, and / or coupling optical element can be configured to operate on non-collimated light (e.g., divergent, convergent light).

[0156] In various implementations, the same waveguide 940 can be used for (i) propagating light from the eyepiece 950 and the environment in front of the user to the camera 940, and (ii) propagating light from the image projector 930 to the eye 210 to form image content therein. Using the same waveguide 940 can simplify the system and / or the eyepiece, and can make the system and / or the eyepiece more compact, thereby possibly providing a reduced form factor. For other reasons, it is also advantageous to reduce the thickness of the eyepiece 950 by reducing the number of waveguides 940. Lower cost and a more simplified manufacturing process may be some of such advantages.

[0157] Also in various designs, the same or different imaging systems can be used in the same head-mounted display to propagate light from the eye to the camera 940 via the waveguide in the eyepiece 950 as described above, for example. Such a system can also use the eyepiece to transmit light from the illumination source to the eye 210 to illuminate the eye. In some designs, the eyepiece can additionally be used to propagate light from the image projector 930 to the eye 210 to form image content therein. Using the eyepiece to assist in imaging the environment and imaging the eye (and possibly illuminating the eye) can simplify the system and / or can make the system more compact, thereby possibly providing a reduced form factor.

[0158] In addition, in some implementations, the same waveguide 940 can be used to (i) propagate light from the environment in front of the eyepiece 950 to the camera 940, and (ii) propagate light from the eye 210 to the camera to capture an image of the eye. The same waveguide can be used to propagate light from the image projector 930 to the eye 210 to form image content therein and / or to propagate light from the illumination source 960 to the eye 210 to illuminate the eye for image capture. Using the same waveguide 940 can simplify the system and / or the eyepiece and can make the system and / or the eyepiece more compact, thereby potentially providing a reduced form factor. For other reasons, it is also advantageous to reduce the thickness of the eyepiece 950 by reducing the number of waveguides 940. Lower cost and a more simplified manufacturing process may be some such advantages.

[0159] Similarly, in addition to coupling light from the environment into the waveguide 940, the same coupling optical element 944 can be configured to direct light from the image projector 930 to the eye 210 to form image content therein and / or to direct light from the eye into the waveguide 940 to be directed to the camera 920 therein. Additionally or alternatively, the same coupling optical element 944 can be configured to couple out light from the illumination source 960 that is directed within the waveguide 940 from the waveguide 940 to the user's eye 210.

[0160] As described above, one or more of the coupling optical element 944, the input coupling optical element 942, or the output coupling optical element 952 can include a polarization-selective coupling element. Thus, in various designs, the light input to the eyepiece 950 or the waveguide 940 is polarized so as to be acted upon appropriately by the polarization-selective steering element.

[0161] Accordingly, in some embodiments, the illumination source 960 includes a polarized light source having an appropriate polarization to be acted upon appropriately by the polarization-selective coupling / steering element.

[0162] One or more polarization-specific filters and polarization modification elements can be included in various imaging systems 900, such as those in which the image projector 930 and / or the light source 960 are placed directly opposite each other through the waveguide 940. For example, in a configuration where polarization-sensitive elements are aligned on opposite sides of the waveguide 940 at the same lateral position, the polarization-sensitive elements can help reduce the directional light emission into the imaging device 920 and / or reduce the saturation of the imaging device 920. Figures 15A - 15B Such a configuration is shown. As Figure 15AThe light source 960 shown can be configured to direct light through a polarization-specific filter 982 such as a polarizer (e.g., a linear polarizer) and / or through a polarization modification element 986 (e.g., a polarization rotator) configured to change the polarization state of the incident light. A retarder such as a half-wave retarder can, for example, rotate linear polarization. Thus, a properly oriented half-wave retarder or half-wave plate can rotate s-polarized light to p-polarized light and vice versa. Thus, in various implementations, the polarization-specific filter 982 and / or the polarization modification element 986 are disposed in the optical path between the light source 960 and the coupling-in optical element 942 to provide properly oriented polarization to the coupling-in optical element. In some implementations, the imaging system 900 does not include a polarization modification element but includes a properly oriented polarization filter such as a polarizer.

[0163] The light emitted by the light source 960 can pass through an arrangement of optical elements in a specific order. For example, as Figure 15A shown, the light from the light source 960 can first pass through the polarization-specific filter 982 (e.g., a polarizer) and then through the polarization modification element 986 (e.g., a rotator). After the light has passed through the polarization modification element 986, the light can be incident on the coupling-in optical element 942, which can direct the light into the waveguide 940 to be guided therein.

[0164] For example, the light source 960 can be configured to emit light of mixed polarization (e.g., s-polarization and p-polarization). The polarization-specific filter 982 can be configured to transmit only light of a first polarization state (e.g., p-polarization). As the light continues, the polarization modification element 986 can be configured to change the polarization state of the light (e.g., from p-polarization to s-polarization). The coupling optical element can be configured to turn the s-polarized light to an angle greater than the critical angle of the waveguide such that the s-polarized light is guided within the waveguide. As the coupled-in light 904 propagates through the waveguide 940, the coupled-in light 904 can be polarized substantially in a second polarization (s-polarization). The coupling optical element 944 can be configured to turn only the light of the second polarization state (s-polarization). The coupling optical element 944 can be configured to couple the coupled-in light 904 out of the waveguide 940 and couple it to the eye 210 to provide illumination for image capture.

[0165] To prevent direct illumination (e.g., saturation) of the imaging device 920, a polarization modifying element 958 and / or a polarization specific filter 984 can be disposed within or on the waveguide 940 such that only light of a specific polarization state (e.g., p-polarization) can pass through the polarization specific filter 984 and reach the imaging device 920. The polarization modifying element 958 (e.g., a half-wave plate) can be configured to change the state of polarization (e.g., from s-polarization to p-polarization). The polarization specific filter 984 can be configured to transmit only light of a specific polarization (e.g., p-polarized light) through it. In this way, the light passing through the polarization specific filter 982 will not be configured to directly transmit through the polarization specific filter 984. In the above implementations such as in Figure 10 , 11A -11E and 12A-12E (e.g., where the image projector 930 and / or the light source 960 are on the same optical axis, as Figure 15A shown), the polarization specific filter 982, the polarization modifying element 986, the coupling-in optical element 942, the polarization modifying element 958 and / or the polarization specific filter 984 can be implemented according to the Figure 15A design. The polarization specific filter 984 can be a transmissive-reflective polarizer (e.g., a polarizing beam splitter) that is configured to transmit light of a first polarization and redirect or reflect light of a second polarization different from the first polarization.

[0166] A partial reflection element (e.g., a semi-transparent mirror) can be included to direct the coupled-in light 904 towards the imaging device 920. The partial reflection element can be disposed between the coupling-in optical element 942 and the polarization changing element 986 such that a portion of the coupled-in light 914 is reflected towards the imaging device 920 while reducing leakage of the coupled-in light 914 from the waveguide 940. The portion of the non-leaking light can be any fraction between 0 and 1. For example, the portion can be 0.90, where 90% of the light rays propagating along the coupling optical element 944 through the waveguide 940 are kept within the waveguide 940 at each reflection of the light rays. Other portions are possible (e.g., 0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80 or a range between any of these values).

[0167] Figure 15B Shows the propagation of light reflected or scattered from the retina. Some of the light 910 incident on the coupling optical element 944 having a second polarization (s-polarization) and reflected by the retina is redirected by the coupling optical element 944 at an angle greater than the critical angle of the waveguide 940 and can thus be guided therein. Some of the light may not be coupled into the waveguide 940 and will pass through as non-coupled-in light 912. The coupled-in light 904 can propagate through the waveguide 940 and towards the camera.

[0168] Other implementations can benefit from the use of polarization-selective elements near the light source and the camera. For example, various systems can be configured to provide illumination with a first polarization and capture an image through the camera using light with a different polarization. For example, when imaging the retina, such a configuration can be used to reduce unwanted reflections, such as reflections from the cornea. The reflection from the cornea will be specular. Thus, if light with the first polarization is incident on the cornea, the light reflected from the cornea will maintain that first polarization. In contrast, the retina is diffuse. If light with the first polarization is incident on the retina, the light reflected from the retina will not only maintain the first polarization. Diffuse reflection is more likely to result in the generation of unpolarized light. Thus, there will be a second polarization in the reflected light that is different from the first polarization. Similarly, by illuminating with the first polarization and imaging with a second different polarization, the retina can be imaged with reduced glare from the cornea.

[0169] Thus, in various implementations, polarization-specific filters 982, 984 can be used together to reduce unwanted reflected light from the eye 210 (e.g., from the cornea). For example, unwanted light, glare, or flash may be reflected from the cornea, which may saturate the image captured by the imaging device 920. The light reflected from the cornea may be specular and maintain its polarization. In contrast, the light reflected from the retina may be more diffusely reflected and may be weakly and uniformly polarized. Similarly, a combination of polarizers can be used to remove some or most of the unwanted reflected light. Initially, polarized light can be used to illuminate the eye 210. In some designs, a polarized illumination source (e.g., light source 960) can be used. Additionally or alternatively, a first polarizer (e.g., polarization-specific filter 982) can be located at the beginning of the optical path of the illumination source to provide an initial polarization of the light. A second polarizer (e.g., polarization-specific filter 984) can be positioned in the optical path before the light enters the imaging device 920. The second polarizer can be rotated 90° from the first polarizer (e.g., polarizers 982, 984 can be "crossed"). As a result, the eye will be illuminated with some light of the first polarization reflected from the cornea. This light will not pass through the polarizer 984 near the camera. However, the light reflected from the retina will include a second polarization. Similarly, the light diffusely reflected from the retina will pass through the polarization 984 near the camera and will enable the camera to capture an image of the retina. Thus, in such a configuration, unwanted light received from the eye (e.g., from the cornea) that may enter the imaging device 920 can be reduced or eliminated. Other configurations are possible. For example, polarization-selective coupling-in optical element 942 for coupling light from light source 960 into waveguide 940 and polarization-selective coupling-out optical element for coupling light out of the waveguide to camera 920 can be employed to have different polarization-sensitive characteristics. For example, the polarization-selective coupling-in optical element can selectively direct light from the illumination source having a first polarization into the waveguide, while the coupling-out optical element can selectively direct light having a second different polarization from the waveguide to the camera. The effect can again be to reduce or remove unwanted light received from the eye (e.g., from the cornea) before entering the imaging device 920.

[0170] Various imaging systems 900 are discussed herein that are capable of using the eyepiece 950 to collect light for imaging the retina. However, the imaging system 900 can be configured to image other parts of the eye, such as the front of the eye. Figure 16Illustrates how imaging system 900 can be used to image the anterior portion of eye 210 (e.g., the cornea). Imaging system 900 can include one or more elements of the exemplary imaging system 900 described above. Additionally, the exemplary imaging system 900 can include one or more optical elements or lenses with a focal power, such as refractive optical elements 980, 990 with a focal power. For example, a positive focal power lens or positive lens 980 can be disposed proximal (e.g., closer to eye 210) to eyepiece 950 between eye 210 and the eyepiece. A negative focal power lens or negative lens 990 can be disposed distal to eyepiece 950 between the eyepiece and the environment in front of the user. One or both of lenses 980, 990 can be a variable focus element (e.g., a zoom lens) and / or can include liquid crystal elements. In some designs, one or both of lenses 980, 990 include Fresnel lenses. Lenses 980, 990 can incorporate liquid crystal to produce a Fresnel lens function. Such a function can allow variable focusing of one or both of lenses 980, 990. In some designs, one or more of lenses 980, 990 can be integrated with and / or fabricated (e.g., formed) on or within eyepiece 950.

[0171] In various embodiments, coupling optical element 944 is configured to direct collimated light reflected from eye 210 into the optical waveguide for guiding therein. Thus, positive lens 980 can be configured to collimate light reflected from eye 210, such as the anterior portion of the eye (e.g., the cornea). Thus, positive lens 980 can have a focal length equal to or substantially equal to the distance from the lens to the portion of eye 210 to be imaged (e.g., the cornea).

[0172] Negative lens 990 can have a focal power similar to or the same as that of positive lens 980 to compensate for or cancel out the focal power of the positive lens. In this way, light from the environment (e.g., the distal end of waveguide 940) can pass through negative lens 990, eyepiece 950, and positive lens 980, with substantially no net change in the focal power introduced by the two lenses. Thus, negative lens 990 can be configured to cancel or nullify the focal power of positive lens 980 such that the user will not experience the focal power of the positive lens when viewing the environment in front of eyepiece 950. Negative lens 990 will also counteract the effect of positive lens 980 to invert the image of an object in the environment in front of the wearer.

[0173] Figure 16Shows the light 928 scattered from it incident on the cornea. The imaging system 900 can be configured to capture this light 988 reflected from the cornea. For example, the positive lens 980 can collect a portion of the light 988 scattered from the cornea and collimate the light 988. The light 988 collimated by the positive lens 980 is incident on the coupling optical element 944, which is configured to turn the collimated light into the waveguide 940 at an angle greater than the critical angle of the waveguide, such that the light is guided therein by TIR. The coupling optical element 944, the decoupling optical element 952, and / or the waveguide 940 can be as described above. The resulting decoupled light 906 can be led out of the waveguide 940 by the decoupling optical element 952 to a camera (not shown).

[0174] Figure 16 Shows light 928 such as collimated light, which can come from the eyepiece 950 as described above. The illumination source 960 can couple light into the waveguide 940, and the coupling element 944 can decouple the light from the illumination source 960 from the waveguide. The coupling element 944 can be configured to decouple light as collimated light from the waveguide 940. This light illuminates the front part of the eye (e.g., the cornea) and is scattered from it. As described above, this scattered light 988 can be collected by the positive lens 980 and the imaging system 900 to form an image of the front part of the eye 210. Also as described above, the illumination 928 directed onto the eye 210 may be invisible (e.g., infrared) light.

[0175] Figure 16 Also shown is an alternative arrangement for illuminating the eye 210. In some designs, one or more light sources 934, such as LEDs or emitters, can be arranged relative to the eye 210 to direct light thereon without being guided through the waveguide 940 by TIR and directed to the eye 210. In some implementations, the eyepiece 950 or the waveguide 940 is not in the optical path between the one or more light sources 934 and the eye 210. In some designs, multiple such light sources 934 can be arranged in a pattern near and / or around the eye (e.g., a circular or annular pattern). In some designs, the pattern of the light sources 934 can define an illumination axis parallel (e.g., coaxial) to the optical axis of one or more lenses 980, 990. The one or more light sources 934 can be similar to the one or more light sources 960 described above and can be pulsed, for example. Similarly, the one or more light sources 934 can include infrared light sources such as infrared LEDs or another type of invisible light. Optionally, the one or more light sources can include visible light sources that emit visible light. Alternatively, the one or more light sources can emit both visible light and invisible light (e.g., infrared light).

[0176] Figure 17 Shows another example imaging system 900, which is configured to image a portion of the eye 210 (e.g., the front part of the eye (e.g., the cornea)). As withFigure 16 compared to the transmissive optical element (lens) 980 shown, Figure 17 the imaging system 900 shown employs a reflective optical element 996 configured to collimate light from the eye. The reflective optical element will have less aberration than the transmissive optical element, and chromatic aberration generally does not apply to reflective optical elements, such as Figure 17 the reflector 996 shown in. Thus, by using a reflective surface to collect light from the eye 210, less (e.g., chromatic aberration) aberration can be introduced into the captured image of the eye.

[0177] Figure 17 Shown, for example, is an imaging system 900 that includes a curved transmissive optical element 996 having a wavelength-dependent reflective coating 998. The curved transmissive optical element 996 can be disposed at the distal end of the waveguide 940 (on the ambient side of the eyepiece 950). Thus, the curved transmissive optical element 996 can be disposed between the ambient environment in front of the wearer and the waveguide 940 and / or the coupling optical element 944. Similarly, the waveguide 940 and / or the coupling optical element 944 can be disposed between the curved transmissive element 996 and the eye 210.

[0178] The wavelength-dependent reflective coating 998 can be configured to reflect light of a specific wavelength or wavelength range. In some implementations, for example, the wavelength-dependent reflective coating 998 can be configured to reflect non-visible light (e.g., infrared light) within a certain wavelength range, while the wavelength-dependent reflective coating 998 can be configured to transmit visible light. In some cases, the wavelength-dependent reflective coating 998 can be disposed on the surface of the curved transmissive optical element 996.

[0179] As described above, in various designs, the coupling optical element 944 is configured to direct the collimated light reflected from the eye 210 into the waveguide 940 for guiding therein. Thus, the reflective optical element 996 can be configured to collimate light reflected from the eye 210 (such as the front portion of the eye (e.g., the cornea)). Thus, the curved reflective optical element 996 can have a positive optical power for light incident on the proximal side of the curved reflective optical element 996 that is reflected from the wavelength-dependent reflective coating 998. In particular, in various designs, the reflective optical element 994 can have a focal length that is equal to or substantially equal to the distance from the reflective optical element 996 to the portion of the eye 210 to be imaged (e.g., the cornea, iris, etc.). Example values of the focal length can be, for example, from 2 cm to 8 cm. In some implementations, the focal length is between 4 cm and 6 cm. In some designs, the focal length is approximately 5 cm. The focal length can be within any range formed by any of these values or can be outside such ranges in different designs.

[0180] In various implementations, the reflective optical element 996 is disposed distal to the eyepiece 950 in front of the eyepiece. Thus, the reflective optical element 996 is disposed between the eyepiece 950 and the environment in front of the user. Similarly, the eyepiece 950 is disposed between the reflective optical element 996 and the eye 210.

[0181] The curved transmissive optical element 996 may have a curved reflective surface that has a curvature of any shape. In some implementations, the surface is rotationally symmetric. In some implementations, the surface may be spherical or aspherical (e.g., parabolic). Non-rotationally symmetric shapes are also possible. However, in various designs, the reflective surface has a positive optical power. The reflective optical element 996 may include, for example, a concave mirror for at least certain wavelengths and / or polarizations.

[0182] The curved transmissive optical element 996 may be configured to have a negligible optical power during transmission. Similarly, the curved transmissive optical element 996 may be configured to transmit light without introducing convergence or divergence. In one example, the curved transmissive optical element 996 may have a curvature of the inner radius that is substantially the same as the curvature of the outer radius. The thin optical element 996 may, for example, reduce optical aberrations for light transmitted through it, may be lighter and / or more compact.

[0183] In various designs, the reflective optical element 996 includes a material that transmits visible light such that the user can see the environment in front of the wearer. In some cases, to enhance transmission, the curved transmissive optical element 996 may be coated with an anti-reflection coating on an outer surface (e.g., the distal surface). The anti-reflection coating may be configured to reduce reflection of visible light such as, for example, red, green, and / or blue light. However, the reflective optical element 996 may be configured to reflect a portion of the light scattered from the eye 210 to form an image of the eye. Thus, the reflective optical element 996 may operate differently with respect to different light. For example, the reflective optical element 996 may operate differently with respect to different wavelengths. The reflective optical element 996 may be configured to reflect infrared light and transmit visible light.

[0184] As described above, one or more light sources 934 may be configured to irradiate the eye 210 with infrared light. As Figure 17As schematically shown, the resulting light 988 reflected from the eye 210 (e.g., the cornea) can diverge. The curved transmissive optical element 996 can be arranged to receive the light 988 reflected from the eye (e.g., the cornea, iris). The wavelength-dependent reflective coating 998 can be configured to reflect the light 988 reflected from the eye because the wavelength used for illuminating the eye is the same as the wavelength reflected by the reflective coating on the curved transmissive optical element 996 (e.g., 850 nm). For example, the eye can be illuminated with infrared light (e.g., 850 nm), and the curved transmissive optical element 996 can be configured to reflect infrared light (e.g., 850 nm) and pass visible light. The shape of the curved transmissive optical element 996 can also be configured to collimate the light 988 reflected from the eye and reflect the light to the coupling optical element 944, which turns the collimated light into the waveguide 940 to be guided by TIR therein.

[0185] In Figure 17 As in some other designs, one or more light sources 934, such as LEDs or emitters, can be arranged relative to the eye 210 to direct light thereon without guiding it through the waveguide 940 by TIR and directing it onto the eye 210. In some implementations, the eyepiece 950 or the waveguide 940 is not in the optical path between one or more light sources 934 and the eye 210. In some designs, multiple such light sources 934 can be arranged in a pattern near and / or around the eye (e.g., a circular or annular pattern). In some designs, the pattern of the light sources 934 can define an illumination axis parallel (e.g., coaxial) to the optical axis of one or more lenses 980, 990. One or more light sources 934 can be similar to one or more light sources 960 described above and can be pulsed, for example. Similarly, one or more light sources 934 can include an infrared light source such as an infrared LED or another type of invisible light. However, other types of light sources can be used.

[0186] Figure 18 Another example imaging system 900 is shown, which is configured to image a portion of the eye 210 (e.g., the front part of the eye (e.g., the cornea)). In Figure 18In the implementation shown, polarization selection is employed to help control the path of light reflected from the eye. In particular, in various designs, the coupling optical element 944 is polarization selective. For example, light having a first polarization is transmitted through the coupling optical element 944, while light having a second different polarization is redirected into the waveguide 940 through the coupling optical element 944 to be coupled therein by TIR. Thus, in various implementations, the eye 210 is illuminated with polarized light, or a polarizer (not shown) is disposed between the eye and the waveguide 940 such that the light from the eye incident on the waveguide is polarized. For example, the emitter 934 can emit polarized light, or a polarizer can be disposed in front of the emitter 934 such that the eye 210 is illuminated with polarized light. Thus, in various designs, the polarization of the polarized light incident and / or reflected from the eye 210 received by the optical coupling element 944 can be the first polarization such that the light is directed to the reflector 996.

[0187] Similarly, in various implementations, the coupling optical element 944 (and / or the outcoupling optical element 952) is configured to transmit light having a first polarization state (e.g., a first linear, circular, or elliptical polarization state (e.g., p-polarization, left-handed circular or elliptical polarization, etc.)) and redirect light having a second polarization state (e.g., a second linear, circular, or elliptical (e.g., s-polarization, right-handed circular or elliptical polarization, etc.)) into and / or out of the waveguide, respectively. In some implementations, the eye illuminator 934 can emit only or primarily the first polarization (e.g., p-polarization) or further include a polarization modification element (e.g., a polarizer) configured to transmit only light having the first polarization state (e.g., p-polarization). Additionally, the coupling optical element 944 and / or the outcoupling optical element 952 can be configured to redirect light having a second polarization (e.g., s-polarization) into and / or out of the waveguide, respectively.

[0188] Similar to Figure 17 the imaging system 900 shown, Figure 17 the imaging system 900 of the curved reflector 998 shown includes a curved transmissive optical element 996 having a wavelength-dependent reflective coating 998. The wavelength-dependent reflective coating 998 can be configured to reflect light of a specific wavelength or wavelength range. In some implementations, for example, the wavelength-dependent reflective coating 998 can be configured to reflect invisible light (e.g., infrared light) within a certain wavelength range, while the wavelength-dependent reflective coating 998 can be configured to transmit visible light. In some cases, the wavelength-dependent reflective coating 998 can be disposed on the surface of the curved transmissive optical element 996.

[0189] In various implementations, a curved transmissive optical element 996 is disposed distally of an eyepiece 950 in front of the eyepiece. Accordingly, the reflective optical element 996 is disposed between the eyepiece 950 and the environment in front of the user. Similarly, the eyepiece 950 is disposed between the reflective optical element 996 and the eye 210.

[0190] Accordingly, light having a first polarization (e.g., p-polarization) from the eye 210 is incident on the coupling optical element 944 and passes therethrough to the curved transmissive optical element 996. The imaging system 900 also includes a polarization modifying optical element 978, such as a retarder (e.g., a quarter-wave retarder). The retarder 978 is transmissive and imparts a quarter-wave delay to the light transmitted therethrough. This light is incident on the curved transmissive optical element 996 and is reflected from the curved transmissive optical element 996. The wavelength-dependent reflective coating 998 can be configured to reflect light of the wavelength reflected from the eye. Accordingly, this light is reflected from the curved surface of the curved transmissive optical element 996 and is collimated. The collimated light passes through the retarder 978 again, thereby imposing another quarter-wave delay on the light transmitted therethrough. The delays introduced on these two lights passing through the retarder (e.g., the total delay wave) cause the polarization to rotate. Accordingly, the first polarization (e.g., p-polarization) transmitted through the polarization-selective coupling optical element 944 on the first pass is converted to a second polarization (s-polarization) and is steered into the waveguide 940 to be guided to the camera 920 by TIR. As described above, in various designs, the coupling optical element 944 is configured to steer the collimated light reflected from the eye 210 into the waveguide 940 to be guided therein. Accordingly, the reflective optical element 996 can be configured to collimate the light reflected from the eye 210, such as the front portion of the eye (e.g., the cornea). Accordingly, the curved reflective optical element 996 can have a positive optical power. In particular, in various designs, the reflective optical element 994 can have a focal length equal to or substantially equal to the distance from the reflective optical element 996 to the portion of the eye 210 to be imaged (e.g., the cornea, iris, etc.). Example values of the focal length can be, for example, from 2 cm to 8 cm. In some implementations, the focal length is between 4 cm and 6 cm. In some designs, the focal length is approximately 5 cm.

[0191] In various designs, the reflective optical element 996 can include a curved surface configured to reflect light. In some cases, the curved surface can be spherical or rotationally symmetric. The reflective optical element 996 can include a concave mirror for at least some wavelengths and / or polarizations.

[0192] In various designs, the reflective optical element 996 includes a material that transmits visible light, enabling a user to see the environment in front of the wearer. Accordingly, a wavelength-dependent reflective coating 998 disposed on the surface of the curved transmissive optical element 996 can transmit visible light or at least a specific wavelength of visible light. The curved transmissive optical element 996 may also be coated with an anti-reflection coating on its outer surface (e.g., the distal surface). The anti-reflection coating may be configured to reduce the reflection of red, green, and / or blue light. However, the reflective optical element 994 may be configured to reflect a portion of the light scattered from the eye 210 to form an image of the eye. Accordingly, the reflective optical element 996 may operate differently with respect to different light. For example, the reflective optical element 996 may operate differently with respect to light of different polarization states (and / or wavelengths). The reflective optical element 996 may be configured to transmit visible light and reflect infrared light.

[0193] As Figure 17 shown, for example Figure 18 one or more light sources 934 such as LEDs or emitters in [description] may be disposed relative to the eye 210 to direct light thereon without being directed through the waveguide 940 via TIR and directed onto the eye 210. Accordingly, in some implementations, the eyepiece 950 or the waveguide 940 is not in the optical path between one or more light sources 934 and the eye 210. In some designs, multiple such light sources 934 may be arranged in a pattern near and / or around the eye (e.g., a circular or annular pattern). One or more light sources 934 may be similar to one or more of the light sources 960 described above and may be pulsed, for example. Similarly, one or more light sources 934 may include an infrared light source such as an infrared LED or another type of invisible light. In particular, in various implementations, the light source 934 may emit light that is reflected by the wavelength-dependent reflective coating 998 and / or the curved transmissive optical element 996. However, other types of light sources may be used.

[0194] Although the polarization-selective coupling optical element 944 is configured to be polarization-selective depending on the type of linear polarization incident thereon, other polarization-selective coupling optical elements can be polarization-selective for other types of polarization states, such as different types of circular or elliptical polarization. The polarization-selective coupling optical element 944 can be configured, for example, such that a first polarization, such as a first circular or elliptical polarization (e.g., left-handed polarization or LHP polarization), is transmitted through the polarization-selective coupling optical element 944, and a second polarization, such as a second circular or elliptical polarization (e.g., right-handed polarization or RHP), is redirected into the optical waveguide, and vice versa. Such a polarization-selective coupling optical element 944 can include a liquid crystal such as a cholesteric liquid crystal. Some examples of liquid crystal optical elements are discussed in the following sections: the section entitled "Cholesteric Liquid Crystal Mirror"; U.S. Publication No. 2018 / 0164627, entitled "DIFFRACTIVE DEVICES BASED ON CHOLESTERIC LIQUID CRYSTAL", filed on Dec. 7, 2017; U.S. Publication No. 2018 / 0239147, entitled "DISPLAY SYSTEM WITH VARIABLE POWER REFLECTOR", filed on Feb. 22, 2018; U.S. Publication No. 2018 / 0239177, entitled "VARIABLE-FOCUS VIRTUAL IMAGE DEVICES BASED ON POLARIZATION CONVERSION", filed on Feb. 22, 2018; the entire contents of which are incorporated herein by reference for all purposes.

[0195] A polarization modification element or retarder, such as a circular polarizer, can be disposed between the eye and the polarization-selective coupling optical element 944 to convert light reflected from the eye into a first polarization (e.g., LHP). The LHP light will pass through the polarization-selective coupling optical element 944, be reflected from the reflector 998, have its polarization changed to RHP, and be redirected into the optical waveguide by the polarization-selective coupling optical element 944 to reach the camera.

[0196] In some implementations, the reflector 996 can be polarization-selective in its reflectivity such that it reflects only light of a particular polarization state and / or transmits light of a different polarization state. Such an optical element can include liquid crystals, such as cholesteric liquid crystals. Examples of such optical elements are discussed in the following sections: the section titled "Cholesteric Liquid Crystal Mirror"; U.S. Publication No. 2018 / 0164627, titled "DIFFRACTIVE DEVICES BASED ON CHOLESTERIC LIQUID CRYSTAL", filed on Dec. 7, 2017; U.S. Publication No. 2018 / 0239147, titled "DISPLAY SYSTEM WITH VARIABLE POWER REFLECTOR", filed on Feb. 22, 2018; U.S. Publication No. 2018 / 0239177, titled "VARIABLE-FOCUS VIRTUAL IMAGE DEVICES BASED ON POLARIZATION CONVERSION", filed on Feb. 22, 2018; the entire contents of which are incorporated herein by reference for all purposes. Such an optical element can reflect light of a first polarization state, such as a first circular or elliptical polarization state (left-handed circular or elliptical polarization), and transmit light of a second polarization state, such as a second circular or elliptical polarization state (e.g., right-handed circular or elliptical polarization), and vice versa. In some embodiments, the liquid crystal is disposed on a curved surface of the reflector 996 such that, upon reflection, the reflector has a focal power, such as a positive focal power. In various other implementations, the liquid crystal optical element can be flat or planar. For example, the liquid crystal can be disposed on a flat or planar substrate or layer. Although flat, a focal power can be included in the liquid crystal optical element. Such an element can be referred to as a cholesteric liquid crystal reflecting lens. Thus, light from the eye can be collimated and reflected to the coupling optical element 998. For example, the reflector can reflect light of a first polarization state (e.g., left-handed circular or elliptical) and transmit light of a second polarization state (e.g., right-handed circular or elliptical). Thus, the eye 210 is irradiated with left-handed circularly polarized light, or light reflected from the eye passes through a polarizer (e.g., a circular or elliptical polarizer) that transmits light having a first polarization (e.g., left-handed circularly polarized light). The coupling optical element 944 can also be polarization-selective and can transmit LHP light and direct RHP light into the waveguide. The LHP light from the eye passes through the coupling optical element 944. This transmitted LHP light is also incident on the wavelength-selective liquid crystal reflector 996 and is reflected therefrom.In some designs, the wavelength-selective liquid crystal reflector 996 converts a first polarization state (e.g., LHP) to a second polarization state (e.g., RHP) upon reflection. Light in this second polarization state (e.g., RHP) is directed to the coupling optical element 944, which steers the light in the second polarization state (RHP) into the waveguide 940 and to the camera 920.

[0197] In some designs, the coupling optical element 944 does not include a liquid crystal grating, but instead includes, for example, a surface relief diffraction grating or a holographic grating. As described above, these coupling optical elements 944 that do not include a cholesteric liquid crystal may also include volume diffraction or holographic optical elements or gratings.

[0198] Accordingly, light scattered from the eye is reflected back into the waveguide 940 by the reflective optical element 996 to be coupled into the waveguide by the coupling element 944. However, conversely, a portion of the unpolarized light from the wearer's front environment corresponding to the second polarization state (e.g., RHP) will transmit through the reflective optical element 996. Thereby, the wearer can see an object through the reflective optical element 996.

[0199] However, in various designs, the reflective optical element 996 will have a negligible focal power upon transmission. For example, the reflective optical element 996 may have curved surfaces with the same curvature on both sides of the optical element such that the total focal power of the optical element for light transmitted therethrough is negligible.

[0200] As described above, in various implementations, the reflective optical element 996 includes, for example, a cholesteric liquid crystal reflective lens, a cholesteric liquid crystal reflective element as discussed in the following section: the section entitled "Cholesteric Liquid Crystal Mirror"; U.S. Publication No. 2018 / 0164627, entitled "DIFFRACTIVE DEVICES BASED ON CHOLESTERIC LIQUID CRYSTAL", filed on December 7, 2017; U.S. Publication No. 2018 / 0239147, entitled "DISPLAY SYSTEM WITH VARIABLE POWER REFLECTOR", filed on February 22, 2018; U.S. Publication No. 2018 / 0239177, entitled "VARIABLE-FOCUS VIRTUAL IMAGE DEVICES BASED ON POLARIZATION CONVERSION", filed on February 22, 2018; the entire contents of which are incorporated herein by reference for all purposes. Such an optical element can operate with respect to a specific wavelength or wavelength range. Thus, light such as infrared light reflected from the eye can be acted upon by the cholesteric liquid crystal reflective element. However, light that is not within this wavelength range, such as visible light from the environment, can pass through the cholesteric liquid crystal reflective element without being acted upon by the cholesteric liquid crystal reflective element. Thus, the cholesteric liquid crystal reflective element has a negligible power with respect to the visible light from the environment that passes through it.

[0201] As described above, in some implementations, the illumination source 960 couples light into the waveguide 940, and the light exits the waveguide to illuminate the eye 210. In such an embodiment, the coupling optical element 944 can be polarization selective. For example, the coupling optical element 944 can transmit a first polarization (p-polarization) and transmit a second polarization (s-polarization).

[0202] Thus, if the light from the illumination source 906 propagates through the waveguide 940 and is redirected by the coupling optical element 944, then the illumination will be s-polarized. A polarization modification optical element (e.g., a quarter-wave retarder) can be disposed between the waveguide 940 and the eye 210 to cause rotation of the polarized light reflected from the eye. The light from the light source 960 reflected from the eye 210 will pass through the quarter-wave retarder twice, and as a result, the s-polarized light that exits the waveguide by the coupling element 944 to illuminate the eye will be converted to p-polarized light.

[0203] The p-polarized light will transmit through the coupling optical element 944 and the waveguide and be incident on the reflective optical element 996.

[0204] The imaging system 900 may further include a second polarization modification element 978, which may include, for example, a retarder or wave plate as described above. The retarder may include, for example, a quarter-wave retarder. The second polarization modification element 978 may be disposed at the distal end of the waveguide 940 between the waveguide and the reflector 996. The second polarization modification element 978 may also be disposed between the coupling element light 944 and the reflector 996. The light (p-polarized) from the eye 210 that transmits through the coupling element 944 passes through the second polarization modification element 978 and is converted into circularly polarized light. If the reflector 996 reflects the circularly polarized light, the light will be reflected back into the waveguide 940 after passing through the polarization modification element 978 again. Passing through the polarization modification element (e.g., a quarter-wave retarder) 978 twice will convert the light into s-polarized light, which will be steered into the waveguide by the coupling element 944 to be guided therein and reach a camera (not shown).

[0205] As Figure 18 shown, the light 988 reflected from the eye 210 is divergent. This light is incident on the curved or positive-powered reflector 996 and can thus be collimated. The coupling optical element 944 configured to collimate and steer into the waveguide 940 will thus direct the collimated light from the curved reflective optical element 996 toward the imaging device 920 (not shown). Thus, the light reflected from the eye 210 collimated by the curved reflective optical element 996 is coupled into the waveguide 940 and guided therein toward the outcoupling optical element 952. The outcoupling optical element 952 may be configured to direct the light out of the eyepiece 950 and reach a camera (not shown).

[0206] The configuration of the imaging system can have various variations. Different types of reflectors 996 and coupling elements 944 may be employed. The reflector 996 and the coupling element 944 may be configured to operate with respect to linearly polarized light or circularly or elliptically polarized light, for example. As discussed, the reflector 996 has a focal power. The reflector 996 and the coupling element 944 may include a cholesteric liquid crystal grating reflector and / or a lens with no focal power. A polarization modification element 978 such as a retarder may be included between the coupling element 944 and the reflector and / or between the coupling element 944 and the eye. In some embodiments, a polarizer such as a circular polarizer or a linear polarizer may be disposed between the eye and the coupling element 944. If, for example, unpolarized light is reflected from the eye, the polarizer (e.g., a circular polarizer or a linear polarizer) may be disposed between the eye and the coupling element 944. In some such cases, the coupling element 944 is polarization-selective.

[0207] In configurations such as Figure 17 and 18 shown, light reflected from the eye passes through waveguide 940 to a curved reflective optical element 996 to be collimated and redirected back to the waveguide, thereby introducing background noise. This background noise is generated by light initially passing from the eye through coupling optical element 944. As described above, coupling optical element 944 can be configured to direct collimated light into waveguide 940 to be guided therein and reach camera 920 that forms an image. However, coupling optical element 944 will deflect some of the non-collimated light incident thereon. Thus, some of the non-collimated (divergent) light reflected from the eye will be coupled into the waveguide by coupling optical element 944 when initially passing through coupling optical element 944 and waveguide 940 to reach curved reflective optical element 996, and contribute background noise to the image of the eye formed by camera 920. This noise will be superimposed on the image formed by the collimated light retroreflected by curved reflective optical element 996, which is coupled into the waveguide by coupling optical element 944 and guided therein and reaches camera 920.

[0208] In some designs, this noise can be subtracted from the image. The process of subtracting noise from the signal may involve (a) measuring the amount of light deflected by coupling optical element 944 and reaching camera 920 (referred to as N) when initially passing through coupling optical element 944 to reach curved reflective optical element 996 and (b) measuring the total signal at camera 920 when light passes through coupling optical element 944 and waveguide 940 to reach curved reflective optical element 996, is collimated and reflected back to coupling optical element and deflected to camera 920. This total signal will also include some noise N because non-collimated light reflected from the eye will have passed through coupling optical element 944 to reach curved reflective optical element 996, and thus some of the non-collimated light will be deflected by coupling optical element 944 to camera 920. If noise N can be separately measured from the total signal T that includes the noise superimposed on the eye image, then noise N can be subtracted from total signal T, as shown in the following equation:

[0209] I = T - N

[0210] where I represents the image with noise component N removed.

[0211] The above two measurements (a) and (b) can be obtained in various ways. For example, as Figure 19 shown, a shutter 936 can be disposed between curved reflective optical element 996 and waveguide 940 and coupling optical element 944. This shutter 936 can be configured to block light when the shutter is in a first state and transmit light when the shutter is in a second state. Shutter 936 can include, for example, a liquid crystal shutter.

[0212] Thus, when the shutter 936 is in the first state where the light reflected from the eye 210 is incident on the coupling optical element 944 and passes through it towards the curved reflective optical element 996, the noise component N can be measured. However, the noise component N can be prevented from reaching the curved reflective optical element by the closed shutter. As described above, some of the light reflected from the eye 210, although mainly uncollimated, does couple into the coupling optical element 944 and is redirected into the waveguide and guided therein to the camera 920. As described above, the formation of this light image does not contribute but generates background noise. When the shutter 936 is closed, the camera 920 can record this noise N.

[0213] When the shutter 936 is in the second state where the shutter is open, the total signal T including both the noise N and the image can be measured. The light reflected from the eye 210 is incident on the coupling optical element 944 again. Some of this light reflected from the eye 210, although mainly uncollimated, couples into the coupling optical element 944 and is redirected into the waveguide and guided therein to the camera 920. However, most of this light reflected from the eye 210 passes through the coupling optical element 944, through the open shutter 936, and reaches the curved reflective optical element 996. The curved reflective optical element 996 collimates at least a portion of this light and reflects it back to the coupling optical element 944, which redirects the collimated light into the waveguide 920 and guides it to the camera 920 to form an image of the eye 210. The camera 920 can capture an image of the eye 210.

[0214] Processing electronics (e.g., processing electronics 140) in communication with the camera 920 can receive the noise component N measured when the shutter 936 is in the first closed state and the total signal T measured when the shutter 936 is in the second open state and can subtract the two (T - N). In this way, the noise N can be subtracted from the total image signal T, which is contributed by the uncollimated light reflected from the eye 210 that is coupled into the coupling optical element 944 when initially passing through it. The processing electronics can communicate with the camera 920 via a wired electrical signal. Additionally or alternatively, the processing electronics can communicate with the camera 920 using one or more remote receivers. The processing electronics can be remotely located (e.g., a cloud computing device, a remote server, etc.).

[0215] Other ways can be employed to perform the measurements of (a) and (b) to obtain N and T and subtract N from T. For example, if the curved reflective optical element 996 is wavelength selective, as Figure 18As shown, the eye can be irradiated with light of different wavelengths at different times. For example, to perform measurement (a) and quantify the noise N, the eye can be irradiated with the wavelength reflected by the unbent reflective optical element 996. However, to perform measurement (b) and quantify the total signal T, the eye can be irradiated with the wavelength reflected by the bent reflective optical element 996. Then, the noise N can be subtracted from the total T, as described above (e.g., T - N).

[0216] Figures 20A - 20E An example imaging system 900 is shown, which, as described above, is configured to use wavelength modulation to measure and subtract the noise component N. Figures 20A - 20E The imaging system 900 in [reference] includes a wavelength-selective bent transmissive optical element 996 (e.g., as referenced above with respect to Figure 17 and Figure 18 described). For example, the bent transmissive optical element 996 has a wavelength-dependent reflective coating 998 on its curved surface. The imaging system 900 may also include one or more light sources or illumination sources (not shown) configured to illuminate the eye 210. The one or more light sources may be configured to emit infrared light. However, the one or more light sources may be configured to emit different colors or wavelengths of light at different times. Such wavelength modulation can enable the measurement of N separately in order to subtract N from the total signal T.

[0217] In various implementations, for example, one or more illumination sources 960, 934 may be configured to emit one or more wavelengths λ 反射 reflected by the bent reflective optical element in a first state, and one or more wavelengths λ 未反射 not reflected in a second state. In the second state, an amount of wavelength λ 反射 reflected by the bent reflective optical element that is no more than negligible is emitted. Similarly, in the first state, an amount of the non-reflected wavelength λ 未反射 that is no more than negligible is emitted.

[0218] In some examples, the reflected wavelength λ 反射 may be between approximately 800 nm and 950 nm. The reflected wavelength λ 反射 may be between about 835 nm and 915 nm. The reflected wavelength λ 反射 may be between about 840 nm and 870 nm. In some designs, the reflected wavelength λ 反射 is approximately 850 nm. The light emission 928 from one or more light sources 960 can illuminate the eye.

[0219] As Figure 20B shown, having a wavelength λ 未反射Light 988 (and light λ reflected by the curved optical element in amounts that are negligible) 反射 ) reflects off a portion (e.g., the cornea) of the eye 210. Since this light includes the wavelength λ that is not reflected by the uncurved reflective optical element 未反射 , the light ray 916 is shown propagating through the curved reflective optical element 996 to the environment in front of the user.

[0220] Although the light 988 incident on the coupling optical element 944 is not collimated, the coupling optical element couples at least some of the light 914 into the waveguide 940 to be directed to the camera 920. Accordingly, the camera 920 can capture an image (Image #1) corresponding to the noise component N, which is generated by the uncollimated light that is redirected by the coupling optical element 944 when initially passing through to the curved reflective optical element 996. This image (Image #1) is background noise and is not an image of a recognizable eye. Processing electronics 140 is shown receiving this first image (Image #1).

[0221] In Figures 20C - 20E , an illumination source (not shown) emits one or more wavelengths λ that are reflected by the curved reflective optical element 反射 , and a wavelength λ in amounts that are negligible is not reflected. The wavelength λ 未反射 can be, for example, 850 nm. 反射

[0222] As Figure 20C shown, some of the light 988 reflected from the eye 210 that is incident on the coupling optical element 944 during a first pass through the coupling optical element 944 is coupled by the coupling optical element 944 into the waveguide 940 (as shown in Figure 20B ) and is directed toward the camera 920. Additionally, the curved transmissive optical element 996 that selectively reflects light of wavelength λ 反射 reflects and collimates the uncoupled light 918 incident on the curved transmissive optical element that is reflected from the eye 210. As shown in Figure 20E , the coupling optical element 944 redirects and couples this collimated reflected light into the waveguide 940 and toward the camera 920. Figure 20E Shown are two components arriving at the camera 920, the light 988 reflected from the eye 210 that is incident on the coupling optical element 944 during a first pass through the coupling optical element 944 (which is coupled by the coupling optical element into the waveguide 940), and the light reflected and collimated by the curved transmissive optical element 996 that is coupled into the waveguide by the coupling optical element. The camera 920 can capture an image (Image #2) corresponding to this total image component T. Processing electronics 140 is shown receiving this second image (Image #2).

[0223] As described above, the processing electronic device can subtract noise T-N from an image. In this example, Image #2 can be subtracted from Image #1. Thus, the processing electronic device 140 can be configured to modify a second image based on a first image. However, other methods are possible. For example, the processing electronic device 140 can be configured to create a new image that represents a version of the second image with reduced optical noise. Implementations for subtracting noise from an image can be used in the above implementations. For example, Figure 10 , Figures 11A - 11E and / or Figures 12A - 12E the implementations shown can include a shutter 936 and / or a curved transmissive optical element 996 having a wavelength-dependent reflective coating 998 configured to selectively reflect non-coupled light 912 and direct the light to an imaging device 920.

[0224] As described above, Image #1 is obtained when irradiating light of one or more wavelengths λ 未反射 that is not reflected by the uncurved reflective optical element, and no more than a negligible amount of the wavelength λ 反射 is reflected. Image #2 is obtained when irradiating light of one or more wavelengths λ 反射 that is reflected by the curved reflective optical element, and no more than a negligible amount of the wavelength λ 未反射 is not reflected. Thus, one or more illumination sources 960, 934 can be configured to modulate the wavelength. For example, in some designs, one or more illumination sources 960, 934 can include a first illumination source configured to output one or more wavelengths λ 未反射 that is not reflected by the uncurved reflective optical element, and no more than a negligible amount of the wavelength λ 反射 is reflected. The one or more illumination sources can further include a second illumination source configured to output one or more wavelengths λ 反射 that is reflected by the curved reflective optical element, and no more than a negligible amount of the wavelength λ 未反射 is not reflected. The intensities of the first and second illumination sources can be alternately increased and decreased, turned on and off, attenuated and not attenuated, passed and blocked to provide modulation of the wavelength of the light irradiating the eye. For example, during a first time interval, the first illumination source can be blocked while the second illumination source is not blocked. During a subsequent second time interval, the second illumination source can be blocked while the first illumination source is not blocked. This process can be repeated to provide modulation of the wavelength of the light irradiating the eye. In other designs, the wavelength of the light source can be tuned and detuned to move the wavelength back and forth between λ 反射 and λ 未反射 . Other arrangements are possible.

[0225] As described above, the imaging system 900 can be included in a head-mounted display such as an augmented reality display, which also provides the ability to image the eye by collecting light with the eyepiece 950. Such an imaging system 900 can be used for eye tracking. Multiple images of the retina of the eye or the front of the eye can be obtained. The movement and / or repositioning of the eye can be determined from these images to track the position and / or orientation of the eye. These imaging systems can also be used for biometric imaging and / or for identifying the user. For example, images of the user's eye (e.g., retina or iris) can be obtained and recorded. Subsequent images of the wearer's eye (e.g., retina or iris) can be obtained at a later time. The two images can be compared to determine whether the wearer in a subsequent instance is the same wearer as in a first instance. However, other uses of the imaging system are possible.

[0226] Although the illumination system can be described above as being waveguide-based and including one or more waveguides, other types of light steering optical elements can be employed instead of waveguides. Such light steering optical elements can include steering features to emit light from the light steering optical element, e.g., onto a spatial light modulator. Thus, in any of the examples described herein and in any of the examples below, any reference to a waveguide can be replaced with a light steering optical element in place of the waveguide. Such light steering optical elements can include, for example, a polarization beam splitter, such as a polarization beam splitting prism.

[0227] As described above, the systems described herein can allow for the collection and / or biometric identification of biological data. For example, the eye or a part thereof (e.g., retina) can be imaged to provide such biological data and / or biometric identification. When a user (presumably the same user) wears a head-mounted display system, images of the eye such as the retina can be obtained at different times. A collection of such images can be recorded in, for example, a database. These images can be analyzed to collect biological data. Such biological data may be useful for monitoring the health or medical condition of the user. Different medical parameters can be monitored by imaging a patient (e.g., the patient's eye (e.g., retina)). The medical parameters can be recorded and compared with subsequent measurements obtained when the user wears the head-mounted display system.

[0228] Additionally, if a person starts wearing a head-mounted display system and an image of the user's eyes is captured that does not match the images stored in the database, it can be concluded that the person currently wearing the head-mounted display system is different from the previous user. This can be useful in determining whether the target user is wearing the head-mounted headset or the headset is being worn by a new user. Such a feature can enable certain medical, security, and / or usability applications or functions. For example, a head-mounted display can be configured to identify the wearer based on the characteristics of the wearer's eyes. For example, the system can be configured to determine an individual based on the wearer's retina (e.g., blood vessels), cornea, or other eye characteristics. For example, in some implementations, a series of markers can be determined for a specific wearer. Based on the series of markers, the system may be able to determine that the previous user was wearing the head-mounted headset, or alternatively, determine that another user is wearing the head-mounted headset. The markers can include the shape or center of the user's cornea, the configuration of the blood vessels in the user's retina, the intensity and / or position of the reflection of light from the cornea, the shape of the aspect of the eye, and / or any other biometric markers. In certain implementations, a confusion matrix can be determined. As discussed above in the context of developing retinal mapping using virtual / gaze targets located at various positions (e.g., see Figure 13B )), the system can have the user's viewing or eye pose in a set of predetermined directions and develop a feature matrix of the eye or parts of the eye (e.g., cornea, retina, etc.) associated with each direction or eye pose. Using such a matrix, the system can determine the identity of an individual. Other methods are possible.

[0229] Similarly, as described above, various configurations of the system are possible. For example, Figure 21 An example eyepiece 900 is shown that can be used to project light into the user's eyes while imaging the user's eyes. The shown eyepiece 900 includes a coupling optical element 2104, a light distribution element 2108, a light consolidating element 2116, and a coupling-out optical element 2120 located on the opposite side of the coupling optical element 2112. Each of these optical elements can be disposed inside or on top of the waveguide 2102. The waveguide 2102 can correspond to, for example, one of the waveguides 670, 680, 690 described herein (e.g., see Figures 9A - 9C ). The coupling-in optical element 2104 can correspond to one of the coupling optical elements 700, 710, 720 described herein and / or correspond to the coupling-in optical element 942 (see, for example, Figure 10 ) and can be configured to inject image content from a projector into the waveguide and / or inject illumination from a light source 960. The light distribution element 2108 can correspond to one of the light distribution elements 730, 740, 750 described herein (e.g., seeFigures 9A - 9C and can be used to distribute light in a given direction and redirect light from the optical element 2104 coupled in to the optical element 2112 coupled out. The optical element 2112 coupled out can correspond to the optical element 944 coupled out described herein (for example, see Figure 10 ). In some designs, the optical element 2112 coupled out includes the functions described herein with respect to the optical elements 800, 810, 820 coupled out (see Figures 9A - 9C ). The light combining element 2116 can be configured to receive the lateral spatial extent of the light received from the optical element 2112 coupled out and redirect the light to the optical element 2120 coupled out. The optical element 2120 coupled out can correspond to the optical element 952 coupled out described herein (for example, see Figure 10 ).

[0230] The optical element 2104 coupled in can be disposed within or on the waveguide 2102 to receive light from, for example, a projector (e.g., the image projector 930) and / or an illuminator (e.g., the light source 960). The light can be transmitted through the waveguide 2102 to the associated light distribution optical element 2108. Any one of the optical element 2104 coupled in, the light distribution optical element 2108, or the optical element 2112 coupled out can be disposed on the main surface of the waveguide (e.g., on the top or bottom surface) or within the waveguide. Similarly, any one or combination of the light combining element 2116 and / or the optical element 2120 coupled out can be disposed on the main surface (e.g., the top surface or both main surfaces) of the waveguide 2102 or within the waveguide.

[0231] The optical element 2112 coupled out can receive light from the light distribution element 2108 (e.g., via TIR) and expand the light to enter the user's eye. Thus, the optical element 2112 coupled out can be disposed in front of the user's eye and project image content therein. Additionally or alternatively, the optical element 2112 coupled out can be configured to provide illumination light above and / or into the user's eye.

[0232] Light reflected from the eye (e.g., illumination light from an illumination source) can be reflected and captured by the optical element 2112 coupled out. Thus, in some embodiments, the optical element 2112 coupled out can be used to couple out light received from the light distribution element 2108 and couple in light received from the eye into the waveguide 2102.

[0233] In some embodiments, the coupling optical element 2112 may include one or more diffractive optical elements (DOEs) such that the coupling optical element 2112 has a dual function. The first DOE (e.g., grating, holographic region) may also be configured to couple out light, and the second DOE may be configured to couple the reflected light from the eye into the waveguide 2102. In some embodiments, the first and second DOEs are superimposed within the waveguide 2102 (e.g., occupy the same or approximately the same volume).

[0234] Optionally, in some embodiments, the coupling optical element 2112 includes at least two DOEs that are stacked on top of or in front of another DOE. For example, referring to Figure 21 , the first DOE of the coupling optical element 2112 may be disposed on top, while the second diffractive element may be disposed below the first DOE. In other implementations, the order of each DOE may be reversed.

[0235] Cholesteric liquid crystal mirror

[0236] Some liquid crystals are in a phase called the chiral phase or cholesteric phase. In the cholesteric phase, the liquid crystal can exhibit a twist of molecules along an axis perpendicular to the director, where the molecular axis is parallel to the director. As described herein, a cholesteric liquid crystal (CLC) layer includes a plurality of liquid crystal molecules in the cholesteric phase that extend in a direction, e.g., perpendicular to the director (e.g., the layer depth direction), and rotate or twist successively in a rotational direction, e.g., clockwise or counterclockwise. The director of the liquid crystal molecules in the chiral structure can be characterized as a helix having a pitch (p) that corresponds to the length in the layer depth direction, which corresponds to the net rotation angle of the liquid crystal molecules in the chiral structure passing through one complete rotation in the first rotational direction. In other words, the pitch is the distance over which the liquid crystal molecules undergo a complete 360° twist. Liquid crystals that exhibit chirality can also be described as having a twist angle or rotation angle (φ), which can refer, e.g., to the relative azimuthal rotation between successive liquid crystal molecules in the layer normal direction, and can have a net twist angle or net rotation angle, which can refer, e.g., to the relative azimuthal rotation between the uppermost and lowermost liquid crystal molecules across a specified length (e.g., the length of the chiral structure or the thickness of the liquid crystal layer). As described herein, a chiral structure refers to a plurality of liquid crystal molecules in the cholesteric phase that extend in a direction, e.g., perpendicular to the director (e.g., the layer depth direction), and rotate or twist continuously in a rotational direction, e.g., clockwise or counterclockwise. On the one hand, the director of the liquid crystal molecules in the chiral structure can be characterized as a helix having a pitch.

[0237] Figure 22A cross-sectional side view of a cholesteric liquid crystal (CLC) layer 1004 including a plurality of uniform chiral structures according to an embodiment is shown. In the CLC layer 1004, the chiral structures adjacent in the lateral direction (e.g., the x-direction) have similarly arranged liquid crystal molecules. In the illustrated embodiment, the chiral structures 1012-1, 1012-2, … 1012-i are similarly configured such that the liquid crystal molecules of different chiral structures located at approximately the same depth, e.g., the liquid crystal molecules closest to the light incident surface 1004S, have the same rotation angle, the continuous rotation angles of the continuous liquid crystal molecules located at approximately the same depth, and the net rotation angle of the liquid crystal molecules of each chiral structure.

[0238] The CLC 1004 includes a CLC layer 1008 that includes liquid crystal molecules arranged as a plurality of chiral structures 1012-1, 1012-2, … 1012-i, where each chiral structure includes a plurality of liquid crystal molecules, and where i is any suitable integer greater than 2. In operation, when incident light having a combination of a left-handed circularly polarized beam and a right-handed circularly polarized beam is incident on the surface 1004S of the CLC layer 1008 by Bragg reflection, the light having one of the circular polarization handednesses is reflected by the CLC layer 1004, while the light having the opposite polarization handedness is transmitted through the CLC layer 1008 without substantial interference. As described herein and throughout the disclosure, the handedness is defined as viewed along the propagation direction. According to an embodiment, when the polarization direction or the handedness of the polarization of the beams 1016-L, 1016-R matches such that it has the same rotation direction as the liquid crystal molecules of the chiral structures 1012-1, 1012-2, … 1012-i, the incident light is reflected. As shown, incident on the surface 1004S are a beam 1016-L having left-handed circular polarization and a beam 1016-R having right-handed circular polarization. In the illustrated embodiment, the liquid crystal molecules of the chiral structures 1012-1, 1012-2, … 1012-i continuously rotate in the clockwise direction (i.e., the positive x-direction) in the direction in which the incident beams 1016-L, 1016-R travel, which is the same rotation direction as the beam 1016-R having right-handed circular polarization. As a result, the beam 1016-R having right-handed circular polarization is substantially reflected, while the beam 1016-L having left-handed circular polarization is substantially transmitted through the CLC layer 1004.

[0239] As described above, by matching the handedness of the polarization of incident elliptical or circularly polarized light with the direction of rotation of the liquid crystal molecules of the chiral structure of the CLC layer, the CLC layer can be configured as a Bragg reflector. Additionally, one or more CLC layers having different pitches can be configured as a wavelength selective Bragg reflector with high bandwidth. Based on the concepts described herein with respect to various embodiments, the CLC layer can be configured as an off-axis or on-axis mirror that is configured to selectively reflect a first wavelength range, such as infrared wavelengths (e.g., near infrared), while transmitting another wavelength range, such as visible wavelengths.

[0240] Figure 23 An example of an eye tracking system 2300 employing a cholesteric liquid crystal reflector (CLCR), such as a wavelength selective CLCR 1150 configured to image a viewer's eye 302, is shown in accordance with various embodiments. Different from the CLC layer 1004 described above Figure 22 the chiral structures adjacent to each other in the lateral direction (e.g., the x direction) in the wavelength selective CLCR 1150 have liquid crystal molecules with different arrangements. That is, the chiral structures are configured such that the liquid crystal molecules of different chiral structures located at approximately the same depth (e.g., the liquid crystal molecules closest to the light incident surface 1004S) have different angles of rotation. As a result, light incident on the CLCR 1150 is reflected at an angle (θ R ) with respect to the layer depth direction, as further described below in the context of the eye tracking system 2300.

[0241] Eye tracking can be a useful feature in an interactive vision or control system, which includes a wearable display system for virtual / augmented / mixed reality display applications and other applications described elsewhere in this specification. To achieve effective eye tracking, it may be desirable to obtain an image of the eye 302 at a low viewing angle, for which it may then be necessary to place the eye tracking camera 702b near the center position of the viewer's eye. However, this position of the camera 702b may interfere with the user's viewing. Optionally, the eye tracking camera 702b can be set at a lower position or side. However, since the eye image is captured at a steeper angle, this position of the camera may increase the difficulty of obtaining robust and accurate eye tracking. By configuring the CLCR 1150 to selectively reflect infrared (IR) light 2308 (e.g., having a wavelength of 850 nm) from the eye 302 while transmitting visible light 2304 from the world, the camera 702b can be placed away from the user's viewing while capturing a normal or low viewing angle eye image. Such a configuration does not interfere with the user's viewing because visible light is not reflected. As shown, the same CLCR 1150 can also be configured as an IR illumination source 2320 by reflecting IR light from an IR source such as an IR LED into the eye 302. The low viewing angle of the IR illuminator can result in less occlusion, e.g., from eyelashes, and this configuration allows for more robust detection of specular reflections, which can be a useful feature in modern eye tracking systems.

[0242] Still referring to Figure 23, according to various embodiments, the CLCR 1150 includes one or more cholesteric liquid crystal (CLC) layers, each cholesteric liquid crystal (CLC) layer including a plurality of chiral structures, wherein each chiral structure includes a plurality of liquid crystal molecules extending in the layer depth direction (e.g., the z-direction) and continuously rotating in a first rotational direction, as described above. The arrangement of the liquid crystal molecules of the chiral structure varies periodically in a transverse direction perpendicular to the layer depth direction, such that one or more CLC layers are configured to substantially Bragg reflect a first incident light having a first wavelength (λ1) while substantially transmitting a second incident light having a second wavelength (λ2). As described above, when viewed in the layer depth direction, each of the one or more CLC layers is configured to substantially Bragg reflect the first and second incident light that is elliptically or circularly polarized with a polarization handedness that matches the first rotational direction, while being configured to substantially transmit the first and second incident light that is elliptically or circularly polarized with a polarization handedness opposite to the first rotational direction. According to an embodiment, the arrangement of the liquid crystal molecules that varies periodically in the transverse direction is arranged to have a period in the transverse direction such that the ratio between the first wavelength and the period is between approximately 0.5 and approximately 2.0. According to an embodiment, the first wavelength is in the near-infrared range between approximately 600 nm and approximately 1.4 μm, such as approximately 850 nm, and the second wavelength is in the visible range having one or more colors as described elsewhere in the specification. According to various embodiments, the liquid crystal molecules of the chiral structure are pre-tilted with respect to a direction normal to the layer depth direction. As configured, one or more CLC layers are configured such that the first incident light is reflected at an angle (θ R ) with respect to the layer depth direction (the z-direction), which angle exceeds approximately 50°, approximately 60°, approximately 70°, or approximately 80° with respect to the layer depth direction.

[0243] So configured, the wavelength-selective CLCR 1150 includes one or more cholesteric liquid crystal (CLC) layers, each cholesteric liquid crystal layer including a plurality of liquid crystal molecules extending in the layer depth direction and continuously rotating in a first rotational direction, wherein the arrangement of the liquid crystal molecules of the chiral structure varies periodically in a transverse direction perpendicular to the layer depth direction, such that one or more CLC layers are configured to substantially Bragg reflect a first incident light having a first wavelength, such as an IR wavelength, while substantially transmitting a second incident light having a second wavelength, such as a visible wavelength.

[0244] Similar liquid crystal layers and structures can be used for the reflectors 996 and coatings 998 described above in connection with Figures 17 - 20E The coating 998 can, for example, include a liquid crystal coating and can be wavelength and / or polarization selective in certain implementations. However, other types of coatings 998 and reflectors 996 can be employed.

[0245] In the foregoing specification, the invention has been described with reference to specific embodiments of the invention. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0246] In fact, it should be understood that the systems and methods of the present disclosure each have several innovative aspects, and no single one of these aspects is solely responsible for or necessary for the desired characteristics disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure.

[0247] Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. Moreover, although the features may be described above as being implemented in certain combinations, and even initially exemplified as such, in some cases one or more features from the exemplified combinations can be deleted, and the exemplified combinations may involve sub-combinations or variants of sub-combinations. No single feature or group of features is necessary or indispensable for each embodiment.

[0248] Conditional language used herein, such as, among other things, "can", "is able to", "may", "could", "for example", etc., generally is intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and / or steps, unless specifically stated otherwise or understood otherwise in context. Thus, such conditional language generally is not intended to imply that a feature, element, and / or step is in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without programmer input or prompting, whether such a feature, element, and / or step is included in or will be performed in any particular embodiment. The terms "comprising", "including", "having", etc. are synonyms and are used inclusively in an open-ended manner and do not exclude additional elements, features, acts, operations, etc. Further, the term "or" is used in its inclusive sense (and not in its exclusive sense), so that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended examples should be construed to mean "one or more" or "at least one" unless specifically stated otherwise. Similarly, although operations may be depicted in the figures in a particular order, it should be appreciated that such operations need not be performed in the particular order shown or in a sequential order, or that all illustrated operations are performed to achieve the desired result. Further, the figures may schematically depict one or more example processes in the form of a flowchart. However, other operations not shown can be incorporated into the example methods and processes schematically shown. For example, one or more additional operations can be performed before, after, concurrently with, or during any of the illustrated operations. Additionally, in other embodiments, the operations can be rearranged or resequenced. In some cases, multitasking and parallel processing may be advantageous. Further, the separation of various system components described in the above embodiments should not be understood to be required in all embodiments, and it should be understood that the program components and systems generally can be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments are within the scope of the following examples. In some cases, the acts recited in the examples can be performed in a different order and still achieve the desired result.

[0249] Accordingly, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the disclosure, the principles and novel features disclosed herein. Various example systems and methods are provided below.

[0250] Example

[0251] Example Part Ⅰ

[0252] 1. A head-mounted display system configured to project light into a user's eyes to display augmented reality image content in the user's field of view, the head-mounted display system comprising:

[0253] A frame configured to be supported on the user's head;

[0254] An image projector configured to project an image into the user's eyes to display image content in the user's field of view;

[0255] A camera;

[0256] At least one waveguide;

[0257] At least one coupling optical element configured to couple light into the waveguide and guide it therein; and

[0258] At least one outcoupling element configured to outcouple light guided in the waveguide from the waveguide and guide the light to the camera,

[0259] wherein the camera is disposed in the optical path with respect to the at least one outcoupling optical element to receive at least a portion of the light, the at least a portion of the light being coupled into the waveguide via the coupling element and guided therein, and outcoupled from the waveguide by the outcoupling coupling element such that an image can be captured by the camera.

[0260] 2. The system according to example 1, wherein the at least one coupling optical element is configured to couple light from the environment in front of the user wearing the head-mounted display system into the at least one waveguide and guide it therein such that an image of the environment can be captured by the camera.

[0261] 3. The system according to any one of the above examples, wherein the at least one coupling optical element is configured to couple light reflected from the eyes of the user wearing the head-mounted display system into the at least one waveguide and guide it therein such that an image of the eyes can be captured by the camera.

[0262] 4. The system according to any one of the above examples, wherein the at least one coupling optical element is configured to couple light reflected from the eyes of the user wearing the head-mounted display system into the waveguide and guide it therein such that an image of the eyes can be captured by the camera, and the system is configured to image the front part of the eyes.

[0263] 5. The system according to any one of the above examples, wherein the at least one coupling optical element is configured such that light reflected from the eyes of the user wearing the head-mounted display system is coupled into the waveguide and guided therein, so that an image of the eyes can be captured by the camera, and the system is configured to image the corneal surface of the eyes.

[0264] 6. The system according to any one of the above examples, wherein the at least one coupling optical element is configured such that light reflected from the eyes of the user wearing the head-mounted display system is coupled into the waveguide and guided therein, so that an image of the eyes can be captured by the camera, and the system is configured to image the retina of the user's eyes.

[0265] 7. The system according to any one of the above examples, further comprising an eyepiece disposed on the frame, the eyepiece being configured to direct light into the eyes of the user to display augmented reality image content in the user's field of view, at least a portion of the eyepiece being transparent and disposed at a position in front of the user's eyes when the user wears the head-mounted display, so that the transparent portion transmits light from the environment in front of the user to the user's eyes to provide a view of the environment in front of the user.

[0266] 8. The system according to any one of examples 7-8, wherein the eyepiece is configured to receive light from the image projector and direct the light into the eyes of the user to display augmented reality image content in the user's field of view.

[0267] 9. The system according to any one of examples 7-9, wherein the eyepiece includes the at least one waveguide.

[0268] 10. The system according to example 7, wherein the image projector is configured to direct light into the edge of the eyepiece.

[0269] 11. The system according to example 9 or 10, wherein the image projector is configured to direct light into the edge of the at least one waveguide.

[0270] 12. The system according to any one of the above examples, further comprising at least one coupling-in optical element, which is configured to couple light from the image projector into the at least one waveguide, so as to guide the light from the image projector to provide the image content to the eyes of the user.

[0271] 13. The system according to any one of the above examples, wherein the at least one coupling optical element is further configured to couple out light from the image projector guided in the waveguide from the at least one waveguide such that the image content can be viewed by the user's eyes.

[0272] 14. The system according to any one of the above examples, wherein the same coupling optical element is configured to couple out light from the image projector guided in the waveguide from the waveguide such that the image content can be viewed by the user's eyes, and to couple light into the at least one waveguide to be guided therein to the camera.

[0273] 15. The system according to any one of the above examples, further comprising at least one image content coupling-out optical element configured to couple out light from the image projector guided in the waveguide from the at least one waveguide such that the image content can be viewed by the user's eyes.

[0274] 16. The system according to any one of the above examples, wherein the at least one coupling optical element faces the eyes of the user wearing the head-mounted imaging system to receive light from the eyes.

[0275] 17. The system according to any one of the above examples, wherein the at least one coupling optical element is configured such that light from the environment in front of the user wearing the head-mounted imaging system is coupled into the at least one waveguide and guided therein such that an image of the environment can be captured by the camera.

[0276] 18. The system according to any one of the above examples, wherein the at least one coupling optical element faces the environment in front of the user wearing the head-mounted imaging system to receive light from the environment.

[0277] 19. The system according to Example 15, wherein the at least one image content coupling-out optical element is configured to couple out light from the image projector guided in the waveguide from the at least one waveguide, and the at least one coupling optical element is configured such that light is coupled into the waveguide and guided therein to the camera, wherein the at least one image content coupling-out optical element and the at least one coupling optical element are superimposed on each other.

[0278] 20. The system according to Example 15, wherein the at least one image content extraction optical element is configured to extract light from the image projector guided in the waveguide from the at least one waveguide, and the at least one coupling optical element is configured to couple light into the waveguide and guide it therein to the camera, wherein the at least one image content extraction optical element and the at least one coupling optical element are stacked on top of each other.

[0279] 21. The system according to Example 15, wherein the at least one image content extraction optical element is configured to extract light from the image projector guided in the waveguide from the at least one waveguide, and the at least one coupling optical element is configured to couple light into the waveguide and guide it therein to the camera, wherein the at least one image content extraction optical element and the at least one coupling optical element are integrated in the same diffractive optical element.

[0280] 22. The system according to Example 15, wherein the at least one coupling optical element is configured to couple light into a first waveguide and guide it therein to the camera, and the at least one image content extraction optical element is configured to extract light from the image projector guided in a second waveguide from the second waveguide.

[0281] 23. The system according to any one of the above examples, wherein the at least one coupling optical element is configured to couple light into a first waveguide and guide it therein to the camera, and the image projector is configured to couple light into a second waveguide to provide image content to the eye.

[0282] 24. The system according to any one of the above examples, wherein the image projector includes a light source, a modulator, and projection optics.

[0283] 25. The system according to any one of the above examples, wherein the image projector includes a scanning optical fiber.

[0284] 26. The system according to any one of Examples 24 or 25, wherein the modulator includes an optical modulator.

[0285] 27. The system according to Example 26, wherein the optical modulator includes a spatial light modulator.

[0286] 28. The system according to any one of the above examples, wherein the camera includes a detector array and imaging optics.

[0287] 29. The system according to Example 28, wherein the imaging optics are configured to focus collimated light onto the detector array.

[0288] 30. The system according to any one of the above examples, wherein the at least one waveguide comprises a material transparent to visible light, the material having a refractive index sufficient to guide light in the waveguide by total internal reflection.

[0289] 31. The system according to any one of the above examples, wherein the at least one waveguide comprises a stack of waveguides.

[0290] 32. The system according to example 31, wherein different waveguides in the waveguide stack are configured to output light having different wavefront divergences, as if projected from different distances from the user's eyes.

[0291] 33. The system according to example 31 or 32, wherein different waveguides in the waveguide stack are configured to output light having different colors.

[0292] 34. The system according to any one of examples 31, 32 or 33, wherein the different waveguides comprise first, second and third waveguides, and the system is configured such that the first waveguide is for red light, the second waveguide is for green light, and the third is for blue light.

[0293] 35. The system according to any one of examples 12 to 34, wherein the optical coupling-in element comprises a diffractive optical element or a reflector.

[0294] 36. The system according to any one of examples 12 to 34, wherein the optical coupling-in element comprises a diffractive optical element.

[0295] 37. The system according to any one of the above examples, wherein the optical coupling element comprises a diffractive optical element.

[0296] 38. The system according to any one of the above examples, wherein the optical coupling element comprises a liquid crystal.

[0297] 39. The system according to any one of the above examples, wherein the optical coupling element comprises a liquid crystal polarization grating.

[0298] 40. The system according to any one of the above examples, wherein the optical coupling-out element comprises a diffractive optical element.

[0299] 41. The system according to any one of the above examples, wherein the optical coupling-out element comprises a liquid crystal.

[0300] 42. The system according to any one of the above examples, wherein the optical coupling-out element comprises a liquid crystal polarization grating.

[0301] 43. The system according to any one of the above examples, wherein the coupling element is configured to increase the size of the eye box along at least one axis.

[0302] 44. The system according to example 43, further comprising an orthogonal pupil expander, which includes at least one light redirecting element located in or on the at least one waveguide, the at least one light redirecting element being configured to increase the size of the eye box along an axis orthogonal to the at least one axis.

[0303] 45. The system according to example 44, wherein the at least one light redirecting element includes a diffractive optical element.

[0304] 46. The system according to any one of the above examples, wherein the same coupling element is configured to (a) couple light into the at least one waveguide for reception by the camera, and (b) couple light from the image projector out of the at least one waveguide to the user's eye.

[0305] 47. The system according to any one of the above examples, wherein the same coupling element is configured to (a) couple light from the environment into the at least one waveguide for reception by the camera, and (b) couple light from the image projector out of the at least one waveguide to the user's eye.

[0306] 48. The system according to any one of the above examples, wherein the same coupling element is configured to (a) couple light from the eye into the at least one waveguide for reception by the camera, and (b) couple light from the image projector out of the at least one waveguide to the user's eye.

[0307] 49. The system according to any one of the above examples, further comprising a reflective surface having a focal power, the reflective surface being arranged to receive light reflected from the user's eye passing through the eyepiece and to direct the light back to the eyepiece.

[0308] 50. The system according to example 49, wherein the at least one coupling element is configured such that light from the user's eye passing through the at least one waveguide and reflected back from the reflective surface to the at least one waveguide is coupled into the at least one waveguide and guided therein.

[0309] 51. The system according to any one of examples 49 to 50, wherein the camera is arranged in the optical path relative to the at least one output optical element to receive at least a portion of the light from the user's eye, at least a portion of which is reflected from the reflective surface and coupled into the waveguide via the coupling element and guided therein and coupled out of the waveguide through the output coupling element.

[0310] 52. The system according to any one of Examples 49 to 51, wherein the reflective surface reflects infrared light but transmits visible light.

[0311] 53. The system according to any one of Examples 49 to 52, wherein the reflective surface is curved.

[0312] 54. The system according to any one of Examples 49 to 53, wherein the reflective surface is disposed on a curved optical element.

[0313] 55. The system according to any one of Examples 49 to 54, wherein the reflective surface is disposed on a concave mirror.

[0314] 56. The system according to any one of Examples 49 to 55, wherein the reflective surface has a positive optical power with respect to reflection and a negligible optical power with respect to transmission.

[0315] 57. The system according to any one of Examples 49 to 56, wherein the reflective surface is configured to collimate light from the user's eye.

[0316] 58. The system according to any one of Examples 49 to 57, wherein the reflective surface is configured to collimate light from the retina of the user's eye.

[0317] 59. The system according to any one of Examples 49 to 58, wherein the reflective surface is configured to collimate light from the anterior region of the user's eye.

[0318] 60. The system according to any one of Examples 49 to 59, wherein the reflective surface is configured to collimate light from the cornea of the user's eye.

[0319] 61. The system according to any one of Examples 49 to 60, wherein the reflective surface is formed on a curved optical element and includes an infrared reflective coating.

[0320] 62. The system according to Example 61, wherein the curved optical element has a negligible optical power with respect to the light transmitted therethrough.

[0321] 63. The system according to Example 61 or 62, wherein the curved optical element has a first curved surface and a second curved surface located on opposite sides of the curved optical element, and the first curved surface and the second curved surface have the same curvature.

[0322] 64. The system according to any one of Examples 49 to 63, further comprising a retarder disposed relative to the reflective surface and the coupling optical element to rotate the polarization of light that passes through at least one waveguide and is reflected from the reflective surface back to at least one waveguide and the coupling optical element.

[0323] 65. The system according to any one of the above examples, wherein at least one coupling element comprises a polarization-selective steering element.

[0324] 66. The system according to any one of the above examples, wherein at least one coupling element comprises a polarization grating.

[0325] 67. The system according to any one of the above examples, wherein at least one coupling element is configured to turn out light guided in at least one waveguide as collimated light directed towards a user's eye.

[0326] 68. The system according to any one of the above examples, wherein at least one coupling element is configured to turn collimated light from a reflective surface into at least one waveguide.

[0327] 69. The system according to any one of the above examples, wherein at least one outcoupling element comprises an off-axis reflector.

[0328] 70. The system according to any one of the above examples, wherein at least one outcoupling element comprises a polarization-selective steering element.

[0329] 71. The system according to any one of the above examples, wherein at least one outcoupling element comprises a polarization grating.

[0330] 72. The system according to any one of the above examples, wherein at least one outcoupling element comprises liquid crystal.

[0331] 73. The system according to any one of the above examples, wherein at least one outcoupling element comprises a liquid crystal polarization grating.

[0332] 74. The system according to any one of the above examples, further comprising a circular polarizer.

[0333] 75. The system according to any one of the above examples, wherein the in-coupling element comprises a polarization-selective steering element.

[0334] 76. The system according to any one of the above examples, wherein the in-coupling element comprises a polarization grating.

[0335] 77. The system according to any one of the above examples, wherein at least one in-coupling element comprises a diffractive optical element.

[0336] 78. The system according to any one of the above examples, wherein at least one in-coupling element comprises a diffraction grating.

[0337] 79. The system according to any one of the above examples, wherein the coupling-in element includes an off-axis reflector.

[0338] 80. The system according to any one of examples 49 to 79, wherein the reflective surface includes a liquid crystal reflector.

[0339] 81. The system according to any one of examples 49 to 80, wherein the reflective surface includes a cholesteric liquid crystal reflective lens.

[0340] 82. The system according to any one of the above examples, wherein the same waveguide (a) guides light coupled from the user's eye into the at least one waveguide to be received by the camera so as to capture an image of at least a part of the user's eye, and (b) guides light coupled from the image projector such that light from the projector can be guided to the user's eye to place the image from the image projector in the user's field of view.

[0341] 83. The system according to any one of the above examples, wherein the same coupling element (a) couples light from the user's eye into the at least one waveguide to be received by the camera, and (b) couples out light from the image projector from the at least one waveguide to the user's eye.

[0342] 84. The system according to any one of examples 49 to 83, further comprising electronics configured to cause the camera to capture a first image when light reflected from the reflective surface is blocked.

[0343] 85. The system according to example 84, wherein the electronics are configured to cause the camera to capture a second image when light reflected from the reflective surface is not blocked.

[0344] 86. The system according to example 85, wherein the electronics are configured to modify the second image using the first image.

[0345] 87. The system according to example 85 or 86, wherein the electronics are configured to subtract from the second image based on the first image.

[0346] 88. The system according to any one of the above examples, wherein the system is configured to perform eye tracking based on an image of the eye.

[0347] 89. The system according to example 88, wherein performing eye tracking based on the image of the eye includes storing an image of the retina of the eye.

[0348] 90. The system according to any one of the above examples, wherein the system is configured to:

[0349] Obtain an image of a portion of the retina of the eye using the camera;

[0350] Compare one or more stored images of the retina with the image of the portion of the retina; and

[0351] Determine the user's gaze based on the comparison of one or more stored images with the image of the portion of the retina obtained from the camera.

[0352] 91. The system according to example 90, wherein determining the user's gaze includes determining which portion of the retina corresponds to the image of the portion of the retina.

[0353] 92. The system according to any one of examples 90 to 91, wherein determining the user's gaze includes determining the orientation of the eye.

[0354] 93. The system according to any one of the above examples, wherein the system is configured to obtain biometric data based on one or more images of the user's eyes obtained using the camera.

[0355] 94. The system according to any one of the above examples, wherein the system is configured to identify the user via biometric sensing based on one or more images of the eye obtained using the camera.

[0356] 95. The system according to any one of the above examples, wherein the system is configured to provide illumination of a first polarization and preferentially capture an image using light of a second polarization different from the first polarization with the camera.

[0357] 96. The system according to any one of the above examples, wherein the system is configured to illuminate the user's eyes with light of a first polarization and preferentially capture an image of the user's eyes using light of a second polarization different from the first polarization with the camera.

[0358] 97. The system according to example 95 or 96, wherein the first polarization and the second polarization are orthogonal.

[0359] 98. The system according to any one of the above examples, further comprising a light source configured to provide illumination for capturing an image using the camera.

[0360] 99. The system according to any one of the above examples, further comprising a light source configured to illuminate the user's eyes.

[0361] 100. The system according to example 98 or 99, wherein the light source comprises one or more infrared light sources.

[0362] 101. The system according to any one of Examples 98 to 100, wherein the light source comprises one or more infrared light emitting diodes (LEDs).

[0363] 102. The system according to any one of Examples 98 to 101, wherein the light source is pulsed.

[0364] 103. The system according to any one of Examples 98 to 102, further comprising an off-axis reflector configured to receive light from the light source and irradiate the user's eyes with the light.

[0365] 104. The system according to any one of Examples 98 to 103, wherein the light source is configured to input light into a waveguide to provide the irradiation.

[0366] 105. The system according to any one of Examples 98 to 104, wherein the light source is configured to input light into a waveguide disposed relative to the eye to provide irradiation to the eye.

[0367] 106. The system according to Example 104 or 105, further comprising an irradiation coupling optical element configured to couple light from the light source into the waveguide.

[0368] 107. The system according to any one of Examples 98 to 103, wherein the light source is configured to input light into the at least one waveguide to provide irradiation.

[0369] 108. The system according to Example 107, further comprising an irradiation coupling optical element configured to couple light from the light source into the at least one waveguide to provide irradiation.

[0370] 109. The system according to any one of Examples 98 to 103, wherein the light source is configured to input light into the same waveguide as the waveguide used to project image content onto the user's eyes.

[0371] 110. The system according to any one of Examples 98 to 104, wherein the light source is configured to provide irradiation to the user's eyes to input light into the same waveguide as the waveguide used to guide the light to a camera.

[0372] 111. The system according to any one of Examples 98 to 105, wherein the light source is configured to input light into the same waveguide as the waveguide used to guide light from the user's eyes to a camera.

[0373] 112. The system according to any one of Examples 109 to 111 further includes an illumination coupling optical element configured to couple light from the light source into the waveguide.

[0374] 113. The system according to any one of Examples 106, 108, or 112, wherein the illumination coupling optical element is polarization selective and couples in light having a first polarization.

[0375] 114. The system according to Examples 98 to 113, wherein the light source is a polarization light source configured to output polarized light having a first polarization.

[0376] 115. The system according to any one of Examples 98 to 114, wherein the light source is configured to direct polarized light having a first polarization onto the eye.

[0377] 116. The system according to Examples 98 to 115 further includes an illumination polarizer having a first polarization, which is disposed in the optical path between the light source and the eye to polarize the light directed to the eye.

[0378] 117. The system according to Example 116, wherein the illumination polarizer is disposed in the optical path between the light source and the waveguide and is configured to provide illumination.

[0379] 118. The system according to any one of Examples 98 to 117 further includes an image acquisition polarizer in the optical path between the eye and the camera.

[0380] 119. The system according to Example 118, wherein the image acquisition polarizer is proximal to the camera.

[0381] 120. The system according to Example 118 or 119, wherein the image acquisition polarizer is disposed in the optical path between (a) the at least one waveguide configured to direct light to the camera and (b) the camera.

[0382] 121. The system according to any one of Examples 118 to 120, wherein the image acquisition polarizer reduces the amount of the first polarized light reaching the camera.

[0383] 122. The system according to Examples 118 to 121, wherein the image acquisition polarizer includes a polarizer configured to selectively couple light having a second polarization different from the first polarization into the camera.

[0384] 123. The system according to any one of the above examples further includes at least one light combining element disposed on the optical path between the at least one coupling element and the at least one light extracting optical element to reduce the lateral spatial extent of the light from the at least one coupling element before reaching the at least one light extracting optical element.

[0385] 124. The system according to any one of the above examples, wherein the at least one light combining element includes a diffractive optical element.

[0386] 125. The system according to any one of the above examples, wherein the at least one light combining element includes a holographic or diffractive grating.

[0387] 126. The system according to any one of the above examples, wherein the at least one waveguide includes a material transparent to infrared light and having a refractive index sufficient to guide infrared light in the waveguide by total internal reflection.

[0388] 127. The system according to any one of the above examples, wherein the at least one coupling optical element includes an exit pupil expander.

[0389] 128. The system according to any one of the above examples, wherein the system includes a focal power to increase the collimation of the light reflected from the eye and coupled into the waveguide to be guided to the camera.

[0390] 129. The system according to any one of the above examples, wherein the system includes a focal power to increase the collimation of the light reflected from the front part of the eye and coupled into the waveguide to be guided to the camera.

[0391] 130. The system according to any one of the above examples, wherein the system includes a focal power to increase the collimation of the light reflected from the cornea of the eye and coupled into the waveguide to be guided to the camera.

[0392] 131. The system according to any one of Examples 128 to 130, wherein the focal power includes a positive focal power.

[0393] 132. The system according to any one of Examples 128 to 131, wherein the focal power is provided by a lens.

[0394] 133. The system according to any one of Examples 88 to 132, wherein one or more stored images of the retina of the eye include a synthetic image of the retina of the eye generated using multiple images of different parts of the retina of the eye.

[0395] 134. The system according to any one of Examples 88 to 133, wherein the synthetic image of the retina includes multiple images of the retina stitched together.

[0396] 135. The system according to any one of Examples 88 to 134, wherein the plurality of images of the retina stitched together include images obtained when a fixation target is displayed in the user's field of view at respective positions.

[0397] 136. The system according to any one of Examples 88 to 135, wherein the one or more stored images of the retina include images obtained when a fixation target is displayed in the user's field of view at respective positions.

[0398] 137. The system according to any one of Examples 88 to 136, wherein the system is further configured to update the composite image using the obtained images of portions of the retina of the eye.

[0399] 138. The system according to any one of Examples 88 to 137, wherein updating the composite image of the retina using the obtained images of portions of the retina includes stitching the obtained images into a portion of the composite image corresponding to the portions of the retina shown in the obtained images.

[0400] 139. The system according to any one of Examples 88 to 138, wherein the system is further configured to apply a digital filter to the obtained images of portions of the retina of the eye to obtain filtered images of the portions of the retina.

[0401] 140. The system according to Example 139, wherein the system is further configured to compare one or more stored images of the retina with the filtered images of the portions of the retina.

[0402] 141. The system according to any one of Examples 139 to 140, wherein the digital filter includes a Frangi filter.

[0403] 142. The system according to any one of Examples 88 to 139, wherein the system is configured to apply edge enhancement to the obtained images of portions of the retina.

[0404] 143. The system according to any of the above examples, wherein the system is configured to perform user identification verification using the images of the retina.

[0405] 144. The system according to any of the above examples, wherein the system is configured to:

[0406] obtain images of portions of the retina of the eye using the camera;

[0407] Compare one or more stored images of the retina with an image of the portion in the retina.

[0408] 145. The system according to example 144, wherein one or more stored images of the retina of an eye include a synthetic image of the retina of the eye generated using a plurality of images of different portions of the retina of the eye.

[0409] 146. The system according to any one of examples 144 to 145, wherein the synthetic image of the retina includes a plurality of images of the retina spliced together.

[0410] 147. The system according to any one of examples 144 to 146, wherein the plurality of images of the retina spliced together include images obtained when a fixation target is displayed at respective positions in the user's field of view.

[0411] 148. The system according to any one of examples 144 to 146, wherein one or more stored images of the retina include images obtained when a fixation target is displayed at respective positions in the user's field of view.

[0412] 149. The system according to any one of examples 144 to 148, wherein the system is further configured to update the synthetic image using the obtained image of the portion in the retina of the eye.

[0413] 150. The system according to any one of examples 144 to 149, wherein updating the synthetic image of the retina using the obtained image of the portion in the retina includes splicing the obtained image into a portion of the synthetic image corresponding to the portion in the retina shown in the obtained image.

[0414] 151. The system according to any one of examples 144 to 150, wherein the system is further configured to apply a digital filter to the obtained image of the portion in the retina of the eye to obtain a filtered image of the portion in the retina.

[0415] 152. The system according to example 151, wherein the system is further configured to compare one or more stored images of the retina with the filtered image of the portion in the retina.

[0416] 153. The system according to any one of examples 144 to 152, wherein the digital filter includes a Frangi filter.

[0417] 154. The system according to any one of examples 144 to 153, wherein the system is configured to apply edge enhancement to the obtained image of the portion in the retina.

[0418] Example Part Ⅱ

[0419] 1. A head-mounted display system configured to project light into a user's eyes to display augmented reality image content in the user's field of view and configured to image at least a portion of the environment in front of a user wearing the head-mounted display system, the head-mounted display system comprising:

[0420] A frame configured to be supported on the user's head;

[0421] An image projector configured to project an image;

[0422] A camera;

[0423] An eyepiece disposed on the frame, the eyepiece configured to direct light into the user's eyes to display augmented reality image content in the user's field of view, at least a portion of the eyepiece being transparent and being disposed in front of the user's eyes when the user wears the head-mounted display such that the transparent portion transmits light from the environment in front of the user to the user's eyes to provide a view of the environment in front of the user, the eyepiece comprising:

[0424] (a) At least one waveguide;

[0425] (b) At least one coupling-in optical element configured to couple light from the image projector into the at least one waveguide so as to guide the light from the image projector therein;

[0426] (c) At least one coupling optical element configured to couple the light from the image projector guided in the waveguide out of the waveguide and direct the light to the user's eyes; and

[0427] (d) At least one coupling-out element configured to couple the light guided within the waveguide out of the waveguide and direct the light to the camera,

[0428] wherein the image projector is disposed in the optical path relative to the at least one coupling-in optical element to couple light from the image projector into the waveguide to be guided therein such that the light is coupled out of the waveguide to the user's eyes through the at least one coupling element so that the image from the projector is located within the user's field of view,

[0429] wherein the coupling element is configured such that light from the environment in front of the user wearing the head-mounted display is coupled into the waveguide and guided therein,

[0430] Wherein, a camera is disposed in an optical path relative to the at least one output optical element to receive at least a portion of light from the environment in front of the user, at least a portion of the light is coupled into the waveguide via a coupling element and guided therein, and is coupled out of the waveguide by the output coupling element such that an image can be captured by the camera, and

[0431] Wherein, the same waveguide (a) guides the light coupled from the environment into the waveguide to be received by the camera for capturing an image of at least a portion of the environment in front of the user, and (b) guides the light coupled from the projector such that the light from the projector is guided to the eyes of the user, so that the image from the projector is located within the field of view of the user.

[0432] 2. The system according to Example 1, wherein the image projector includes a light source, a modulator, and projection optics.

[0433] 3. The system according to Example 1 or 2, wherein the image projector includes a scanning optical fiber.

[0434] 4. The system according to Example 2 or 3, wherein the modulator includes an optical modulator.

[0435] 5. The system according to Example 4, wherein the optical modulator includes a spatial light modulator.

[0436] 6. The system according to any one of the above examples, wherein the camera includes a detector array and imaging optics.

[0437] 7. The system according to Example 6, wherein the imaging optics are configured to focus collimated light onto the detector array.

[0438] 8. The system according to any one of the above examples, wherein the at least one waveguide includes a material transparent to visible light, the material having a refractive index sufficient to guide light in the waveguide by total internal reflection.

[0439] 9. The system according to any one of the above examples, wherein the at least one waveguide includes a stack of waveguides.

[0440] 10. The system according to Example 9, wherein different waveguides in the waveguide stack are configured to output light having different wavefront divergences, as if projected from different distances from the user's eyes.

[0441] 11. The system according to Example 9 or 10, wherein different waveguides in the waveguide stack are configured to output light having different colors.

[0442] 12. The system according to any one of Examples 9, 10, or 11, wherein the different waveguides include first, second, and third waveguides, and the system is configured such that the first waveguide is for red light, the second waveguide is for green light, and the third waveguide is for blue light.

[0443] 13. The system according to any one of the above examples, wherein the optical coupling-in element includes a diffractive optical element or a reflector.

[0444] 14. The system according to any one of the above examples, wherein the optical coupling element includes a diffractive optical element.

[0445] 15. The system according to any one of the above examples, wherein the optical coupling-out element includes a diffractive optical element.

[0446] 16. The system according to any one of the above examples, wherein the coupling element is configured to increase the size of the eyebox along at least one axis.

[0447] 17. The system according to Example 16, further comprising an orthogonal pupil expander, which includes at least one light redirecting element located within or on the at least one waveguide, and the at least one light redirecting element is configured to increase the size of the eyebox along an axis orthogonal to the at least one axis.

[0448] 18. The system according to Example 17, wherein the at least one light redirecting element includes a diffractive optical element.

[0449] 19. The system according to any one of the above examples, wherein the same coupling element is configured to (a) couple light from the environment into the at least one waveguide for reception by the camera, and (b) couple light from the image projector out of the at least one waveguide to the user's eye.

[0450] 20. The system according to any one of the above examples, wherein the same coupling element is configured to (a) couple light from the environment into the at least one waveguide for reception by the camera, and (b) couple light from the image projector out of the at least one waveguide to the user's eye.

[0451] 21. The system according to Example 20, wherein the at least one coupling element is configured such that light from the user's eye passing through the eyepiece and reflected back to the eyepiece from the reflective surface is coupled into the waveguide and guided therein.

[0452] 22. The system according to any one of Examples 20 to 21, wherein the camera is disposed in the optical path relative to the at least one output optical element to receive at least a portion of the light from the user's eye, at least a portion of which is reflected from the reflective surface and coupled into the waveguide via the coupling element and guided therein and coupled out of the waveguide through the output coupling element.

[0453] 23. The system according to any one of Examples 20 to 22, wherein the reflective surface reflects infrared light but transmits visible light.

[0454] 24. The system according to any one of Examples 20 to 23, wherein the reflective surface is curved.

[0455] 25. The system according to any one of Examples 20 to 24, wherein the reflective surface is disposed on a curved optical element.

[0456] 26. The system according to any one of Examples 20 to 25, wherein the reflective surface is disposed on a concave mirror.

[0457] 27. The system according to any one of Examples 20 to 26, wherein the reflective surface has a positive optical power with respect to reflection and a negligible optical power with respect to transmission.

[0458] 28. The system according to any one of Examples 20 to 27, wherein the reflective surface is configured to collimate the light from the user's eye.

[0459] 29. The system according to any one of Examples 20 to 28, wherein the reflective surface is configured to collimate the light from the retina of the user's eye.

[0460] 30. The system according to any one of Examples 20 to 29, wherein the reflective surface is configured to collimate the light from the anterior region of the user's eye.

[0461] 31. The system according to any one of Examples 20 to 30, wherein the reflective surface is configured to collimate the light from the cornea of the user's eye.

[0462] 32. The system according to any one of Examples 20 to 31, wherein the reflective surface is formed on a curved optical element having an infrared reflective coating disposed on the reflective surface.

[0463] 33. The system according to Example 33, wherein the curved optical element has a negligible optical power with respect to the light transmitted therethrough.

[0464] 34. The system according to example 32 or 33, wherein the curved optical element has a first curved surface and a second curved surface located on opposite sides of the curved optical element, and the first curved surface and the second curved surface have the same curvature.

[0465] 35. The system according to any one of the above examples, further comprising a retarder disposed relative to the reflective surface and the coupling optical element to rotate the polarization of light that passes through the eyepiece and is reflected back from the reflective surface to the eyepiece and the coupling optical element.

[0466] 36. The system according to any one of the above examples, wherein at least one coupling element comprises a polarization-selective steering element.

[0467] 37. The system according to any one of the above examples, wherein at least one coupling element comprises a polarization grating.

[0468] 38. The system according to any one of the above examples, wherein at least one coupling element is configured to turn out light guided in at least one waveguide as collimated light directed towards the user's eye.

[0469] 39. The system according to any one of the above examples, wherein at least one coupling element is configured to steer collimated light from the reflective surface into at least one waveguide.

[0470] 40. The system according to any one of the above examples, wherein at least one outcoupling element comprises an off-axis reflector.

[0471] 41. The system according to any one of the above examples, wherein at least one outcoupling element comprises a polarization-selective steering element.

[0472] 42. The system according to any one of the above examples, wherein at least one outcoupling element comprises a polarization grating.

[0473] 43. The system according to any one of the above examples, further comprising a circular polarizer.

[0474] 44. The system according to any one of the above examples, wherein the in-coupling element comprises a polarization-selective steering element.

[0475] 45. The system according to any one of the above examples, wherein the in-coupling element comprises a polarization grating.

[0476] 46. The system according to any one of the above examples, wherein the in-coupling element comprises an off-axis reflector.

[0477] 47. The system according to any one of examples 20 to 34, wherein the reflective surface comprises a liquid crystal reflector.

[0478] 48. The system according to any one of Examples 20 to 34 or 37, wherein the reflective surface includes a cholesteric liquid crystal reflective lens.

[0479] 49. The system according to any one of the above examples, wherein the same waveguide (a) guides light coupled from the user's eye into the at least one waveguide to be received by the camera in order to capture an image of at least a part of the user's eye, and (b) guides light coupled from the image projector, such that light from the projector can be guided to the user's eye to place the image from the image projector in the user's field of view.

[0480] 50. The system according to any one of the above examples, wherein the same coupling element (a) couples light from the user's eye into the at least one waveguide to be received by the camera, and (b) couples out light from the image projector from the at least one waveguide to the user's eye.

[0481] 51. The system according to any one of the above examples, further comprising electronics configured to cause the camera to capture a first image when light reflected from the reflective surface is blocked.

[0482] 52. The system according to Example 51, wherein the electronics are configured to cause the camera to capture a second image when light reflected from the reflective surface is not blocked.

[0483] 53. The system according to Example 52, wherein the electronics are configured to modify the second image using the first image.

[0484] 54. The system according to Example 53, wherein the electronics are configured to subtract from the second image based on the first image.

[0485] 55. The system according to any one of the above examples, wherein the system is configured to perform eye tracking based on the image of the eye.

[0486] 56. The system according to Example 55, wherein performing eye tracking based on the image of the eye includes storing an image of the retina of the eye.

[0487] 57. The system according to any one of the above examples, wherein the system is configured to:

[0488] store an image of the retina of the eye;

[0489] capture an image of a part of the retina of the eye;

[0490] Compare the stored image of the retina with the image of the portion of the retina; and

[0491] Determine the user's gaze based on the comparison of the stored image with the image of the portion in the retina.

[0492] 58. The system according to example 57, wherein determining the user's gaze includes determining which portion of the retina corresponds to the image of the portion of the retina.

[0493] 59. The system according to any one of examples 57 to 58, wherein determining the user's gaze includes determining the orientation of the eye.

[0494] 60. The system according to any one of the above examples, further comprising a light source configured to irradiate the user's eyes.

[0495] 61. The system according to example 60, wherein the light source includes one or more infrared light sources configured to direct infrared light to the user's eyes.

[0496] 62. The system according to example 60 or 61, wherein the light source includes one or more infrared light-emitting diodes (LEDs).

[0497] 63. The system according to any one of examples 60 to 62, wherein the light source is pulsed.

[0498] 64. The system according to any one of the above examples, further comprising an off-axis reflector configured to receive light from the light source and irradiate the user's eyes with the light.

[0499] 65. A head-mounted imaging system configured to image at least a portion of an environment in front of a user wearing the head-mounted imaging system, the head-mounted imaging system comprising:

[0500] A frame configured to be supported on the user's head;

[0501] A camera;

[0502] An eyepiece disposed on the frame, at least a portion of the eyepiece being transparent and being disposed in a position in front of the user's eyes when the user wears the head-mounted imaging system, such that the transparent portion transmits light from the environment in front of the user to the user's eyes to provide a view of the environment in front of the user, the eyepiece comprising:

[0503] (a) At least one waveguide;

[0504] (b) At least one light-coupling optical element configured to couple light from the environment in front of a user wearing the head-mounted imaging system into the waveguide and direct it therein; and

[0505] (c) At least one light-extracting element configured to extract light guided within the waveguide from the waveguide and direct the light to the camera,

[0506] wherein the camera is disposed in the optical path with respect to the at least one light-extracting optical element to receive at least a portion of the light from the environment in front of the user, the at least a portion of the light being coupled into the waveguide via a coupling element and directed therein, and being extracted from the waveguide by the light-extracting coupling element such that an image can be captured by the camera.

[0507] 66. The system according to example 65, wherein the camera includes a detector array and imaging optics.

[0508] 67. The system according to example 66, wherein the imaging optics are configured to focus collimated light onto the detector array.

[0509] 68. The system according to any one of examples 65 to 67, wherein the at least one waveguide includes a material transparent to visible light having a refractive index sufficient to guide light in the waveguide by total internal reflection.

[0510] 69. The system according to any one of examples 65 to 68, wherein the at least one waveguide includes a stack of waveguides.

[0511] 70. The system according to example 69, wherein different waveguides in the waveguide stack are configured to output light having different wavefront divergences as if projected from different distances from the user's eyes.

[0512] 71. The system according to example 69 or 70, wherein different waveguides in the waveguide stack are configured to output light having different colors.

[0513] 72. The system according to any one of examples 69 to 71, wherein the different waveguides include a first, a second, and a third waveguide, and the system is configured such that the first waveguide is for red light, the second waveguide is for green light, and the third waveguide is for blue light.

[0514] 73. The system according to any one of examples 65 to 72, wherein the light-coupling optical element includes a diffractive optical element.

[0515] 74. The system according to any one of examples 65 to 73, wherein the light-extracting optical element includes a diffractive optical element.

[0516] 75. The system according to any one of Examples 65 to 74, wherein the coupling element is configured to increase the size of the eye box along at least one axis.

[0517] 76. The system according to Example 75, further comprising an orthogonal pupil expander, which includes at least one light redirecting element located within or on the at least one waveguide, the at least one light redirecting element being configured to increase the size of the eye box along an axis orthogonal to the at least one axis.

[0518] 77. The system according to Example 76, wherein the at least one light redirecting element includes a diffractive optical element.

[0519] Example Part Ⅲ

[0520] 1. A head-mounted display system configured to project light into a user's eyes to display augmented reality image content in the user's field of view and configured to image at least a portion of the eyes of a user wearing the head-mounted display system, the head-mounted display system comprising:

[0521] A frame configured to be supported on the user's head;

[0522] An image projector configured to project an image;

[0523] A camera;

[0524] An eyepiece disposed on the frame, the eyepiece being configured to direct light into the user's eyes to display augmented reality image content in the user's field of view, at least a portion of the eyepiece being transparent and being disposed at a position in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from the environment in front of the user to the user's eyes to provide a view of the environment in front of the user, the eyepiece comprising:

[0525] (a) At least one waveguide;

[0526] (b) At least one light coupling-in optical element configured to couple light from the image projector into the at least one waveguide for guiding the light from the image projector therein;

[0527] (c) At least one coupling optical element configured to couple out the light from the image projector guided in the waveguide and direct the light to the user's eyes; and

[0528] (d) At least one light coupling-out element configured to couple out the light guided within the waveguide from the waveguide and direct the light to the camera;

[0529] A reflective surface having a focal power, which is configured to receive light reflected from a user's eye passing through the eyepiece and guide the light back to the eyepiece;

[0530] Wherein, an image projector is disposed in the optical path with respect to the at least one light-coupling optical element to couple light from the image projector into the waveguide for guiding therein, such that the light is coupled out of the waveguide to the user's eye through the at least one coupling element, so that the image from the projector is located within the user's field of view,

[0531] Wherein, the at least one coupling element is configured such that light from the user's eye passing through the eyepiece and reflected back to the eyepiece from the reflective surface is coupled into the waveguide and guided therein, and

[0532] Wherein, a camera is disposed in the optical path with respect to the at least one light-coupling-out optical element to receive at least a portion of the light from the user's eye, at least a portion of which is reflected from the reflective surface and coupled into the waveguide and guided therein and then coupled out of the waveguide through the light-coupling-out coupling element.

[0533] 2. The system according to Example 1, further comprising a light source configured to irradiate the user's eye.

[0534] 3. The system according to Example 2, wherein the light source includes one or more infrared light sources configured to direct infrared light to the user's eye.

[0535] 4. The system according to Example 2 or 3, wherein the light source includes one or more infrared light-emitting diodes (LEDs).

[0536] 5. The system according to any one of Examples 2 to 4, wherein the light source is pulsed.

[0537] 6. The system according to any one of the above examples, further comprising an off-axis reflector configured to receive light from the light source and irradiate the eye with the light.

[0538] 7. The system according to any one of the above examples, wherein the reflective surface reflects infrared light but transmits visible light.

[0539] 8. The system according to any one of the above examples, wherein the reflective surface is curved.

[0540] 9. The system according to any one of the above examples, wherein the reflective surface is disposed on a curved optical element.

[0541] 10. The system according to any one of the above examples, wherein the reflective surface is provided on a concave mirror.

[0542] 11. The system according to any one of the above examples, wherein the reflective surface has a positive optical power with respect to reflection and a negligible optical power with respect to transmission.

[0543] 12. The system according to any one of the above examples, wherein the reflective surface is configured to collimate light from the user's eye.

[0544] 13. The system according to any one of the above examples, wherein the reflective surface is configured to collimate light from the retina of the user's eye.

[0545] 14. The system according to any one of the above examples, wherein the reflective surface is configured to collimate light from the anterior region of the user's eye.

[0546] 15. The system according to any one of the above examples, wherein the reflective surface is configured to collimate light from the cornea of the user's eye.

[0547] 16. The system according to any one of the above examples, wherein the reflective surface is formed on a curved optical element having an infrared reflective coating located on the reflective surface.

[0548] 17. The system according to example 9 or 16, wherein the curved optical element has a negligible optical power with respect to light transmitted therethrough.

[0549] 18. The system according to any one of examples 9 or 16 or 17, wherein the curved optical element has a first curved surface and a second curved surface located on opposite sides of the curved optical element, and the first curved surface and the second curved surface have the same curvature.

[0550] 19. The system according to any one of the above examples, further comprising a retarder disposed relative to the reflective surface and the coupling optical element to rotate the polarization of light that passes through the eyepiece and is reflected back from the reflective surface to the eyepiece and the coupling optical element.

[0551] 20. The system according to any one of the above examples, wherein at least one coupling element includes a polarization-selective steering element.

[0552] 21. The system according to any one of the above examples, wherein at least one coupling element includes a polarization grating.

[0553] 22. The system according to any one of the above examples, wherein at least one coupling element is configured to turn out light guided in at least one waveguide as collimated light directed towards a user's eye.

[0554] 23. The system according to any one of the above examples, wherein at least one coupling element is configured to turn collimated light from a reflective surface into at least one waveguide.

[0555] 24. The system according to any one of the above examples, wherein at least one outcoupling element includes an off-axis reflector.

[0556] 25. The system according to any one of the above examples, wherein at least one outcoupling element includes a polarization-selective steering element.

[0557] 26. The system according to any one of the above examples, wherein at least one outcoupling element includes a polarization grating.

[0558] 27. The system according to any one of the above examples, further comprising a circular polarizer.

[0559] 28. The system according to any one of the above examples, wherein the in-coupling element includes a polarization-selective steering element.

[0560] 29. The system according to any one of the above examples, wherein the in-coupling element includes a polarization grating.

[0561] 30. The system according to any one of the above examples, wherein the in-coupling element includes an off-axis reflector.

[0562] 31. The system according to any one of the above examples, wherein the reflective surface includes a liquid crystal reflector.

[0563] 32. The system according to any one of the above examples, wherein the reflective surface includes a cholesteric liquid crystal reflective lens.

[0564] 33. The system according to any one of the above examples, wherein the image projector includes a light source, a modulator, and projection optics.

[0565] 34. The system according to any one of the above examples, wherein the image projector includes a scanning optical fiber.

[0566] 35. The system according to any one of the above examples, wherein the modulator includes an optical modulator.

[0567] 36. The system according to Example 34, wherein the optical modulator includes a spatial light modulator.

[0568] 37. The system according to any one of the above examples, wherein the camera includes a detector array and imaging optics.

[0569] 38. The system according to example 36, wherein the imaging optics are configured to focus collimated light onto the detector array.

[0570] 39. The system according to any one of the above examples, wherein the at least one waveguide includes a material transparent to visible light, the material having a refractive index sufficient to guide light in the waveguide by total internal reflection.

[0571] 40. The system according to any one of the above examples, wherein the at least one waveguide includes a stack of waveguides.

[0572] 41. The system according to example 40, wherein different waveguides in the waveguide stack are configured to output light having different wavefront divergences, as if projected from different distances from the user's eye.

[0573] 42. The system according to example 40 or 41, wherein different waveguides in the waveguide stack are configured to output light having different colors.

[0574] 43. The system according to any one of examples 40, 41, or 42, wherein the different waveguides include first, second, and third waveguides, and the system is configured such that the first waveguide is for red light, the second waveguide is for green light, and the third waveguide is for blue light.

[0575] 44. The system according to any one of the above examples, wherein the optical coupling-in element includes a diffractive optical element or a reflector.

[0576] 45. The system according to any one of the above examples, wherein the optical coupling element includes a diffractive optical element.

[0577] 46. The system according to any one of the above examples, wherein the optical coupling-out element includes a diffractive optical element.

[0578] 47. The system according to any one of the above examples, wherein the coupling element is configured to increase the size of the eye box along at least one axis.

[0579] 48. The system according to example 47, further comprising an orthogonal pupil expander, which includes at least one light redirecting element located within or on the at least one waveguide, the at least one light redirecting element being configured to increase the size of the eye box along an axis orthogonal to the at least one axis.

[0580] 49. The system according to Example 48, wherein the at least one light redirecting element includes a diffractive optical element.

[0581] 50. The system according to any one of the above examples, wherein the same waveguide (a) guides light coupled from the user's eye into the at least one waveguide to be received by the camera so as to capture an image of at least a part of the user's eye, and (b) guides light coupled from the image projector such that light from the projector can be guided to the user's eye to place the image from the projector in the user's field of view.

[0582] 51. The system according to any one of the above examples, wherein the same coupling element (a) couples light from the user's eye into the at least one waveguide to be received by the camera, and (b) couples light from the image projector out of the at least one waveguide to the user's eye.

[0583] 52. The system according to any one of the above examples, further comprising electronics configured to cause the camera to capture a first image when light reflected from a reflective surface is blocked.

[0584] 53. The system according to Example 52, wherein the electronics is configured to cause the camera to capture a second image when light reflected from a reflective surface is not blocked.

[0585] 54. The system according to Example 53, wherein the electronics is configured to modify the second image using the first image.

[0586] 55. The system according to Example 54, wherein the electronics is configured to subtract the first image from the second image.

[0587] 56. The system according to any one of the above examples, wherein the system is configured to perform eye tracking based on the image of the eye.

[0588] 57. The system according to Example 56, wherein performing eye tracking based on the image of the eye includes storing an image of the retina of the eye.

[0589] 58. The system according to any one of the above examples, wherein the system is configured to:

[0590] Store an image of the retina of the eye;

[0591] Capture an image of a part of the retina of the eye;

[0592] Compare the stored image of the retina with the image of the portion of the retina; and

[0593] Determine the user's gaze based on the comparison of the stored image with the image of the portion in the retina.

[0594] 59. The system according to example 58, wherein determining the user's gaze includes determining which portion of the retina corresponds to the image of the portion of the retina.

[0595] 60. The system according to any one of examples 58 to 59, wherein determining the user's gaze includes determining the orientation of the eye.

[0596] 61. The system according to any one of the above examples, wherein the coupling element is configured such that light from the environment in front of the user wearing the head-mounted display is coupled into the waveguide and guided therein.

[0597] 62. The system according to any one of the above examples, wherein a camera is disposed in the optical path relative to the at least one outcoupling optical element to receive at least a portion of the light from the environment in front of the user, the at least a portion of the light is coupled into the waveguide via the coupling element and guided therein, and is outcoupled from the waveguide through the outcoupling coupling element such that an image of the environment can be captured by the camera.

[0598] 63. The system according to any one of the above examples, wherein the same waveguide (a) guides the light coupled from the environment into the waveguide to be received by the camera for capturing an image of at least a portion of the environment in front of the user, and (b) guides the light coupled from the projector such that the light from the projector is guided to the user's eyes so that the image from the projector is located within the user's field of view.

[0599] 64. The system according to any one of the above examples, wherein the same coupling element (a) couples the light from the environment into the at least one waveguide to be received by the camera, and (b) outcouples the light from the image projector from the at least one waveguide to the user's eyes.

[0600] 65. A head-mounted display system configured to project light onto a user's eyes to display augmented reality image content in the user's field of view and configured to image at least a portion of the eyes of a user wearing the head-mounted display system, the head-mounted display system comprising:

[0601] A frame configured to be supported on the user's head;

[0602] An image projector configured to project an image;

[0603] A camera;

[0604] An eyepiece disposed on the frame, the eyepiece being configured to direct light into the user's eyes to display augmented reality image content in the user's field of view, at least a portion of the eyepiece being transparent and being disposed at a position in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from the environment in front of the user to the user's eyes to provide a view of the environment in front of the user, the eyepiece comprising:

[0605] (a) At least one waveguide;

[0606] (b) At least one coupling-in optical element configured to couple light from the image projector into the at least one waveguide so as to guide the light from the image projector therein;

[0607] (c) At least one coupling optical element configured to couple the light from the image projector guided in the waveguide out of the waveguide and direct the light to the user's eyes; and

[0608] (d) At least one coupling-out element configured to couple the light guided within the waveguide out of the waveguide and direct the light to the camera; and

[0609] A positive lens having a positive optical power, which is disposed in the optical path between the user's eyes and the eyepiece such that light reflected from the user's eyes is projected through the lens onto the eyepiece; and

[0610] A negative lens having a negative optical power, which is disposed on the other side of the eyepiece with the positive lens to offset the optical power of the positive lens for light from the environment in front of the user,

[0611] wherein the image projector is disposed in the optical path relative to the at least one coupling-in optical element to couple light from the image projector into the waveguide to be guided therein, such that the light is coupled out of the waveguide to the user's eyes through the at least one coupling element, so that the image from the image projector is located within the user's field of view,

[0612] wherein the at least one coupling element is configured such that light from the user's eyes that reaches the eyepiece through the lens is coupled into the waveguide and guided therein, and

[0613] Wherein, the camera is disposed in the optical path with respect to the at least one output optical element to receive at least a portion of the light from the user's eye, and at least a portion of the light is reflected from the reflective surface and coupled into the waveguide via the coupling element and guided therein and coupled out of the waveguide through the output coupling element.

[0614] 66. The system according to example 65, wherein the positive lens comprises a Fresnel lens.

[0615] 67. The system according to example 65 or 66, wherein the positive lens is configured to collimate light from the anterior region of the user's eye.

[0616] 68. The system according to any one of examples 65, 66 or 67 above, wherein the positive lens is configured to collimate light from the cornea of the user's eye.

[0617] 69. The system according to any one of examples 65 to 68, wherein the system is configured to perform eye tracking based on the image of the eye.

[0618] 70. The system according to any one of examples 65 to 69, further comprising a light source configured to irradiate the user's eye.

[0619] 71. The system according to example 70, wherein the light source comprises one or more infrared light sources configured to direct infrared light to the user's eye.

[0620] 72. The system according to example 70 or 71, wherein the light source comprises one or more infrared light-emitting diodes (LEDs).

[0621] 73. The system according to any one of the above examples, wherein the system is configured to identify the user via biometric sensing based on the image of the eye.

Claims

1. A head-mounted display system configured to project light into a user's eyes to display augmented reality image content in the user's field of view and configured to image at least a portion of the environment in front of the user wearing the head-mounted display system, the head-mounted display system comprising: A frame configured to be supported on the user's head; An image projector configured to project an image; A camera; And An eyepiece disposed on the frame, the eyepiece being configured to direct light into the user's eyes to display augmented reality image content in the user's field of view, at least a portion of the eyepiece being transparent and disposed at a position in front of the user's eyes when the user wears the head-mounted display, such that the transparent portion transmits light from the environment in front of the user to the user's eyes to provide a view of the environment in front of the user. The eyepiece includes: At least one waveguide; At least one coupling-in optical element configured to couple light from the image projector into the at least one waveguide for guiding the light from the image projector therein; At least one coupling optical element configured to couple out the light from the image projector guided in the waveguide from the waveguide and direct the light to the user's eyes; and At least one coupling-out element configured to couple out the light guided within the waveguide from the waveguide and direct the light to the camera, wherein the image projector is disposed in the optical path relative to the at least one coupling-in optical element to couple light from the image projector into the waveguide for guiding therein, such that the light is coupled out from the waveguide to the user's eyes through the at least one coupling element, so that the image from the projector is located within the user's field of view. Wherein the coupling element is configured such that light from the environment in front of the user wearing the head-mounted display is coupled into the waveguide and guided therein, wherein the camera is disposed in the optical path relative to the at least one coupling-out optical element to receive at least a portion of the light from the environment in front of the user, the at least a portion of the light being coupled into the waveguide and guided therein via the coupling element, and being coupled out from the waveguide by the coupling-out coupling element such that an image of the environment can be captured by the camera, and wherein the same waveguide (a) guides the light coupled from the environment into the waveguide to be received by the camera for capturing an image of at least a portion of the environment in front of the user, and (b) guides the light coupled from the projector such that the light from the projector is directed to the user's eyes, so that the image from the projector is located within the user's field of view.

2. The system according to claim 1, wherein, The image projector includes a light source, a modulator, and projection optics.

3. The system according to claim 1 or 2, wherein, The image projector includes a scanning optical fiber.

4. The system according to claim 2 or 3, wherein, The modulator includes an optical modulator.

5. The system according to claim 4, wherein, The optical modulator includes a spatial light modulator.

6. The system according to any one of claims 1-5, wherein, The camera includes a detector array and imaging optics.

7. The system according to claim 6, wherein, The imaging optics is configured to focus collimated light onto the detector array.

8. The system according to any one of claims 1-7, wherein, The at least one waveguide includes a material transparent to visible light, the material having a refractive index sufficient to guide light in the waveguide by total internal reflection.

9. The system according to any one of claims 1-8, wherein, The at least one waveguide includes a waveguide stack.

10. The system according to claim 9, wherein, The different waveguides in the waveguide stack are configured to output light having different wavefront divergences as if projected from different distances from the user's eyes.

Images