Method and system for eye-tracking using an off-axis mirror with distortion correction

Abstract

Techniques for tracking an eye of a user of an augmented reality headset having an eye tracking system. The eye tracking system can reflect light from the eye of the user and propagate the light through a first refractive optical element. The eye tracking system can then propagate the light through a second refractive optical element, impinge the light on a reflective optical element, and reflect the light from the reflective optical element. The eye tracking system can then propagate the reflected light through the second refractive optical element to a lateral aperture of the second refractive optical element and detect the reflected light at a camera. The eye tracking system can output an eye tracking signal from the camera to track the eye.

Description

METHOD AND SYSTEM FOR EYE-TRACKING USING AN OFF-AXIS MIRROR WITH DISTORTION CORRECTIONBACKGROUND OF THE INVENTION

[0001] Modem computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, wherein digitally reproduced images or portions thereof are presented to a viewer in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or "VR," scenario typically involves the presentation of digital or virtual image information without transparency to other actual real -world visual input; an augmented reality, or "AR," scenario typically involves the presentation of digital or virtual image information as an augmentation to visualization of the actual world around the viewer.

[0002] Referring to FIG. 1, an augmented reality scene 100 is depicted. The user of an AR technology sees a real -world park-like setting 106 featuring people, trees, buildings in the background, and a real-world concrete platform 120. The user also perceives that he / she "sees" "virtual content" such as a robot statue 110 standing upon the real -world concrete platform 120, and a flying cartoon-like avatar character 102 which seems to be a personification of a bumble bee. The robot statue 110 and the flying cartoon-like avatar character 102 are "virtual" in that they do not exist in the real world. Because the human visual perception system is complex, it is challenging to produce AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements.

[0003] Despite the progress made in these display technologies, there is a need in the art for improved methods and systems related to augmented reality systems, particularly, eye tracking in augmented reality systems.SUMMARY OF THE INVENTION

[0004] The present invention relates generally to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the present invention provide methods and systems useful for eye tracking when additional refractive optical elements are present with the wearable displays. Merely by way of example, an augmented reality system can include prescription lenses to correct a user's vision. An eye tracking system used with the augmented reality system can image the user's eye to determine, for example, the user's gaze with respect to virtual content presented by the augmented reality system. The eye tracking system can be configured in accordance with embodiments of the present invention to accurately account for the effects of the prescription lenses on illumination light used to image the user's eyes for eye tracking. The invention applies to a variety of applications in computer vision and image display systems.

[0005] Numerous benefits are achieved by way of the present invention over conventional techniques. For example, embodiments of the present invention provide methods and systems that can improve the imaging of a user's eye in the presence of prescription lenses. Conventional eye tracking systems include cameras that directly image the user's eyes. If the user requires prescription lenses, then the prescription lenses can affect the light reflected from the user's eyes, introducing distortion to the image. If the path of the light reflected from the user's eyes also reflects off an optical element in the system before reaching the camera, the reflected light can pass through the prescription lens a second time, adding additional distortion. The techniques described herein avoid the second pass of light through the prescription lens, reducing distortion in the image of the eyes and improving the determination of eye gaze. Moreover, the techniques described herein can include positioning the eye tracking cameras to the side of the eyepiece of an augmented reality system, which can provide a larger clear viewing area and allow the user to have a clearer view of the virtual content produced by the augmented reality system. These and other embodiments of the invention along with many of its advantages and features are described in more detail in conjunction with the text below and attached figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 illustrates a user's view of augmented reality (AR) through an AR device.

[0007] FIG. 2A illustrates a cross-sectional, side view of an example of a set of stacked waveguides that each includes an incoupling optical element.

[0008] FIG. 2B illustrates a perspective view of an example of the one or more stacked waveguides of FIG. 2A.

[0009] FIG. 2C illustrates a top-down, plan view of an example of the one or more stacked waveguides of FIGS. 2 A and 2B.

[0010] FIG. 3 is a simplified illustration of an eyepiece waveguide having a combined pupil expander according to an embodiment of the present invention.

[0011] FIG. 4 illustrates an example of a wearable display system according to an embodiment of the present invention.

[0012] FIG. 5 shows a perspective view of a wearable device according to an embodiment of the present invention.

[0013] FIG. 6 is an exploded perspective view of elements of an eye tracking system for an augmented reality display according to an embodiment of the present invention.

[0014] FIG. 7 is a simplified plan view of elements of an AR headset including eye tracking according to an embodiment of the present invention.

[0015] FIG. 8 A is a simplified plan view of an AR headset including eye tracking using a reflective optical element according to an embodiment of the present invention.

[0016] FIG. 8B is a simplified plan view of an AR headset including eye tracking and a prescription lens according to an embodiment of the present invention.

[0017] FIG. 9 is a simplified plan view of an AR headset with eye tracking using an off- axis mirror, according to an embodiment of the present invention.

[0018] FIG. 10A is a simplified plan view of elements of an eye tracking system using an off-axis mirror, according to an embodiment of the present invention.

[0019] FIG. 10B is an example optical schematic including elements of an eye tracking system using an off-axis mirror, according to an embodiment of the present invention.

[0020] FIG. 11 A is an example optical schematic including modeled ray tracing for imaging at a decentered camera sensor, according to an embodiment of the present invention.

[0021] FIG. 1 IB illustrates an example image formed at the decentered camera sensor using an off-axis mirror having a linear diffraction grating.

[0022] FIG. 11C illustrates another example image formed at the decentered camera sensor using an off-axis mirror having a diffraction grating characterized by fourth-order polynomial phase parameters.

[0023] FIG. 12A is an example optical schematic including modeled ray tracing for imaging at a camera sensor, according to an embodiment of the present invention.

[0024] FIG. 12B illustrates an example image formed at the camera sensor using an off- axis mirror having a linear diffraction grating.

[0025] FIG. 12C illustrates another example image formed at the camera sensor using an off-axis mirror having a diffraction grating characterized by fourth-order polynomial phase parameters.

[0026] FIG. 13 is a simplified flowchart illustrating a method of eye tracking according to an embodiment of the present invention.

[0027] FIG. 14 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention.DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

[0028] The present invention relates generally to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the present invention provide methods and systems for tracking the eyes of a user (e.g., for determining eye orientation metrics, including the center of eye rotation, interpupillary distance, pupil diameter, gaze angle, and the like) when the wearable displays include prescription or other refractive optical elements that can distort illumination light used to illuminate the eyes during eye tracking. By positioning eye tracking cameras laterally adjacent to an off-axis mirror, the illumination light that reflects from the eyes may only pass through the prescription lens once, preventing a "double pass" effect in which the prescription lens affects the illumination light twice.

[0029] As described herein, embodiments of the present invention provide off-axis mirrors (e.g., reflective diffraction gratings; one each per eyepiece of the wearable display corresponding to the user’s eyes) disposed on the world side (e.g., side of the optical elements away from the user's eye) of a set of refractive optical elements of the wearable display. The prescription lenses can be positioned between the refractive optical elements and the user's eyes. An eye tracking system can include cameras, at least one per eye, that are positioned laterally adjacent to the refractive optical element having the off-axis mirror. As a result, the illumination light reflecting off the user's eyes may pass through the prescription lens and through the refractive optical element, reflect from the off-axis mirror, and then propagate out from a lateral side of the refractive optical element to the eye tracking cameras. In doing so, the illumination light reflected from the eye only passes through the prescription lenses once. However, positioning the eye tracking cameras laterally adjacent to the refractive optical elements results in a large angle of incidence for the light at the off-axis mirror, introducing perspective distortion to an image of the eye formed at the cameras. Embodiments of the present invention reduce the perspective distortion by configuring the off-axis mirror with suitable parameters, including fourth-order polynomial phase parameters. Appropriate configuration of the off-axis mirror enables the wearable devices of the present invention to achieve improved eye tracking quality due to both avoiding the double-pass effect through the prescription lenses and reducing the perspective distortion from the increased angle of incidence of the reflected light from the eye.

[0030] Reference will now be made to the drawings, in which like reference numerals refer to like parts throughout. Unless indicated otherwise, the drawings are schematic not necessarily drawn to scale.

[0031] With reference now to FIG. 2 A, in some embodiments, light impinging on a waveguide may need to be redirected to incouple that light into the waveguide. An incoupling optical element may be used to redirect and incouple the light into its corresponding waveguide. Although referred to as "incoupling optical element" through the specification, the incoupling optical element need not be an optical element and may be a non-optical element. FIG. 2A illustrates a cross-sectional, side view of an example of a set of stacked waveguides 200 that each includes an incoupling optical element. The waveguides may each be configured to output light of one or more different wavelengths, or one or moredifferent ranges of wavelengths. Light from a projector is injected into the set of stacked waveguides 200 and outcoupled to a user as described more fully below.

[0032] The illustrated set of stacked waveguides 200 includes waveguide 202, waveguide204, and waveguide 206. Each waveguide includes an associated incoupling optical element (which may also be referred to as a light input area on the waveguide), with, e.g., incoupling optical element 203 disposed on a major surface (e.g., an upper major surface) of waveguide 202, incoupling optical element 205 disposed on a major surface (e.g., an upper major surface) of waveguide 204, and incoupling optical element 207 disposed on a major surface (e.g., an upper major surface) of waveguide 206. In some embodiments, one or more of the incoupling optical elements may be disposed on the bottom major surface of the respective waveguide (particularly where the one or more incoupling optical elements are reflective, deflecting optical elements). As illustrated, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be disposed on the upper major surface of waveguide 202, waveguide 204, and waveguide 206, respectively (or the top of the next lower waveguide), particularly where those incoupling optical elements are transmissive, deflecting optical elements. In some embodiments, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be disposed in the body of the waveguide 202, waveguide 204, and waveguide 206, respectively. In some embodiments, as discussed herein, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 are wavelength- selective, such that they selectively redirect one or more wavelengths of light, while transmitting other wavelengths of light. While illustrated on one side or comer of waveguide 202, waveguide 204, and waveguide 206, respectively, it will be appreciated that the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be disposed in other areas of waveguide 202, waveguide 204, and waveguide 206, respectively, in some embodiments.

[0033] As illustrated, the incoupling optical element 203, the incoupling optical element205, and the incoupling optical element 207 may be laterally offset from one another. In some embodiments, each incoupling optical element may be offset such that it receives light without that light passing through another incoupling optical element. For example, each of the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be configured to receive light from a different projector and may beseparated (e.g., laterally spaced apart) from other incoupling optical elements such that it substantially does not receive light from the other ones of the incoupling optical elements.

[0034] Each waveguide also includes associated light distributing elements, with, e.g., light distributing elements 210 disposed on a major surface (e.g., a top major surface) of waveguide 202, light distributing elements 212 disposed on a major surface (e.g., a top major surface) of waveguide 204, and light distributing elements 214 disposed on a major surface (e.g., a top major surface) of waveguide 206. In some other embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be disposed on a bottom major surface of associated waveguide 202, waveguide 204, and waveguide 206, respectively. In some other embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be disposed on both top and bottom major surfaces of associated waveguide 202, waveguide 204, and waveguide 206, respectively; or the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be disposed on different ones of the top and bottom major surfaces in different associated waveguide 202, waveguide 204, and waveguide 206, respectively.

[0035] Waveguide 202, waveguide 204, and waveguide 206 may be spaced apart and separated by, e.g., gas, liquid, and / or solid layers of material. For example, as illustrated in FIG. 2A, layer 208 may separate waveguide 202 and waveguide 204 and layer 209 may separate waveguide 204 and waveguide 206. In some embodiments, layer 208 and layer 209 are formed of low refractive index materials (that is, materials having a lower refractive index than the material forming the immediately adjacent one of waveguide 202, waveguide 204, or waveguide 206). Preferably, the refractive index of the material forming layer 208 and / or layer 209 is 0.05 or more, or 0.10 or less than the refractive index of the material forming the waveguide 202, the waveguide 204, or the waveguide 206. Advantageously, layer 208 and layer 209 having the lower refractive index may function as cladding layers that facilitate total internal reflection (TIR) of light through the waveguide 202, the waveguide 204, and the waveguide 206 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layer 208 and the layer 209 are formed of air. While not illustrated, it will be appreciated that the top and bottom of the illustrated set of stacked waveguides 200 may include immediately neighboring cladding layers.

[0036] Preferably, for ease of manufacturing and other considerations, the material forming the waveguide 202, the waveguide 204, and the waveguide 206 are similar or the same, and the material forming the layer 208 and the layer 209 are similar or the same. In some embodiments, the material forming the waveguide 202, the waveguide 204, and the waveguide 206 may be different between one or more waveguides, and / or the material forming the layer 208 and the layer 209 may be different, while still holding to the various refractive index relationships noted above.

[0037] With continued reference to FIG. 2 A, light ray 218, light ray 219, and light ray 220 are incident on the set of stacked waveguides 200. It will be appreciated that the light ray 218, the light ray 219, and the light ray 220 may be injected into the waveguide 202, the waveguide 204, and the waveguide 206 by one or more projectors (not shown).

[0038] In some embodiments, light ray 218, the light ray 219, and the light ray 220 have different properties, e.g., different wavelengths or different ranges of wavelengths, which may correspond to different colors. The incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 each deflect the incident light such that the light propagates through a respective one of the waveguide 202, the waveguide 204, or the waveguide 206 by TIR. In some embodiments, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 each selectively deflect one or more particular wavelengths of light, while transmitting other wavelengths to an underlying waveguide and associated incoupling optical element.

[0039] For example, incoupling optical element 203 may be configured to deflect light ray 218, which has a first wavelength or range of wavelengths, while transmitting light ray 219 and light ray 220, which have different second and third wavelengths or ranges of wavelengths, respectively. The light ray 219 transmitted through the waveguide 202 impinges on and is deflected by the incoupling optical element 205, which is configured to deflect light of a second wavelength or range of wavelengths. The light ray 220 is deflected by the incoupling optical element 207, which is configured to selectively deflect light of a third wavelength or range of wavelengths.

[0040] With continued reference to FIG. 2 A, the light ray 218, the light ray 219, and the light ray 220 are deflected such that they propagate through corresponding waveguide 202, waveguide 204, and waveguide 206, respectively; that is, the incoupling optical element 203,the incoupling optical element 205, and the incoupling optical element 207 of each waveguide deflects light into the corresponding waveguide 202, waveguide 204, or waveguide 206 to incouple light into that corresponding waveguide. The light ray 218, the light ray 219, and the light ray 220 are deflected at angles that cause the light to propagate through the respective waveguide 202, waveguide 204, and waveguide 206 by TIR. The light ray 218, the light ray 219, and the light ray 220 propagate through the respective waveguide202, waveguide 204, and waveguide 206 by TIR until impinging on the waveguide's corresponding light distributing elements: the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214, where they are outcoupled to provide out-coupled light rays 216.

[0041] With reference now to FIG. 2B, a perspective view of an example of the set of stacked waveguides 200 of FIG. 2A is illustrated. As noted above, the light ray 218, the light ray 219, and the light ray 220 are incoupled and deflected by the incoupling optical element203, the incoupling optical element 205, and the incoupling optical element 207, respectively, and then propagate by TIR within the waveguide 202, the waveguide 204, and the waveguide 206, respectively. The light ray 218, the light ray 219, and the light ray 220 then impinge on the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214, respectively. The light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 deflect the light ray 218, the light ray 219, and the light ray 220 so that they propagate towards the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226, respectively.

[0042] In some embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or distribute light to the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 and, in some embodiments, may also increase the beam or spot size of this light as it propagates to the outcoupling optical elements. In some embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be omitted and the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be configured to deflect light directly to the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling opticalelements 226. For example, with reference to FIG. 2A, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be replaced with the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226, respectively. In some embodiments, the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light to the eye of the user. It will be appreciated that the OPEs may be configured to increase the dimensions of the eye box in at least one axis and the EPEs may be configured to increase the eye box in an axis crossing, e.g., orthogonal to, the axis of the OPEs. For example, each OPE may be configured to redirect a portion of the light striking the OPE to an EPE of the same waveguide, while allowing the remaining portion of the light to continue to propagate down the waveguide. Upon impinging on the OPE again, another portion of the remaining light is redirected to the EPE, and the remaining portion of that portion continues to propagate further down the waveguide, and so on. Similarly, upon striking the EPE, a portion of the impinging light is directed out of the waveguide towards the user, and a remaining portion of that light continues to propagate through the waveguide until it strikes the EPE again, at which time another portion of the impinging light is directed out of the waveguide, and so on. Consequently, a single beam of incoupled light may be "replicated" each time a portion of that light is redirected by an OPE or EPE, thereby forming a field of cloned beams of light. In some embodiments, the OPE and / or EPE may be configured to modify a size of the beams of light. In some embodiments, the functionality of the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 and the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 are combined in a combined pupil expander as discussed in relation to FIG. 2E.

[0043] Accordingly, with reference to FIGS. 2 A and 2B, in some embodiments, the set of stacked waveguides 200 includes the waveguide 202, the waveguide 204, and the waveguide 206; the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207; the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 (e.g., OPEs); and the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 (e.g., EPs) for each component color. The waveguide 202, the waveguide 204, and the waveguide 206 may be stacked with an air gap / cladding layer between each one. Theincoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 redirect or deflect incident light (with different incoupling optical elements receiving light of different wavelengths) into its waveguide. The light then propagates at an angle which will result in TIR within the waveguide 202, the waveguide 204, and the waveguide 206, respectively. In the example shown, light ray 218 (e.g., blue light) is deflected by the incoupling optical element 203, and then continues to bounce down the waveguide, interacting with the light distributing element 210 (e.g., OPEs) and then the outcoupling optical element 222 (e.g., EPs), in a manner described earlier. The light ray 219 and the light ray 220 (e.g., green and red light, respectively) will pass through the waveguide202, with light ray 219 impinging on and being deflected by incoupling optical element 205. The light ray 219 then bounces down the waveguide 204 via TIR, proceeding on to its light distributing element 212 (e.g., OPEs) and then the outcoupling optical element 224 (e.g., EPs). Finally, light ray 220 (e.g., red light) passes through the waveguide 206 to impinge on the incoupling optical element 207 of the waveguide 206. The incoupling optical element 207 deflects the light ray 220 such that the light ray propagates to light distributing element 214 (e.g., OPEs) by TIR, and then to the outcoupling optical element 226 (e.g., EPs) by TIR. The outcoupling optical element 226 then finally out-couples the light ray 220 to the viewer, who also receives the outcoupled light from the other waveguides: the waveguide 202 and the waveguide 204.

[0044] FIG. 2C illustrates a top-down, plan view of an example of the set of stacked waveguides 200 of FIGS. 2A and 2B. As illustrated, the waveguide 202, the waveguide 204, and the waveguide 206, along with each waveguide's associated light distributing element: the light distributing element 210, light distributing element 212, and light distributing element 214 and the associated outcoupling optical elements: the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226, may be vertically aligned. However, as discussed herein, the incoupling optical element203, the incoupling optical element 205, and the incoupling optical element 207 are not vertically aligned; rather, the incoupling optical elements are preferably non-overlapping (e.g., laterally spaced apart as seen in the top-down or plan view). As discussed further herein, this non-overlapping spatial arrangement facilitates the injection of light from different resources into different waveguides on a one-to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments,arrangements including non-overlapping spatially separated incoupling optical elements may be referred to as a shifted pupil system, and the incoupling optical elements within these arrangements may correspond to sub pupils.

[0045] FIG. 3 is a simplified illustration of an eyepiece waveguide having a combined pupil expander according to an embodiment of the present invention. In the example illustrated in FIG. 3, the eyepiece 310 utilizes a combined OPEZEPE region in a single-side configuration. Referring to FIG. 3, the eyepiece 310 includes a substrate 320 in which incoupling optical element 322 and a combined OPEZEPE region 324, also referred to as a combined pupil expander (CPE), are provided. Incident light ray 330 is incoupled via the incoupling optical element 322 and outcoupled as output light rays 332 via the combined OPEZEPE region 324.

[0046] The combined OPEZEPE region 324 includes gratings corresponding to both an OPE and an EPE that spatially overlap in the x-direction and the y-direction. In some embodiments, the gratings corresponding to both the OPE and the EPE are located on the same side of a substrate 320 such that either the OPE gratings are superimposed onto the EPE gratings or the EPE gratings are superimposed onto the OPE gratings (or both). In other embodiments, the OPE gratings are located on the opposite side of the substrate 320 from the EPE gratings such that the gratings spatially overlap in the x-direction and the y-direction but are separated from each other in the z-direction (i.e., in different planes). Thus, the combined OPEZEPE region 324 can be implemented in either a single-sided configuration or in a two- sided configuration.

[0047] FIG. 4 illustrates an example of wearable display system 430 into which the various waveguides and related systems disclosed herein may be integrated. With reference to FIG.4, the wearable display system 430 includes a display 432, and various mechanical and electronic modules and systems to support the functioning of the display 432. The display 432 may be coupled to a frame 434, which is wearable by a user 440 (also referred to as a viewer or a display system user) and which is configured to position the display 432 in front of the eyes of the user 440. The display 432 may be considered eyewear in some embodiments. In some embodiments, a speaker 436 is coupled to the frame 434 and configured to be positioned adjacent to the ear canal of the user 440 (in some embodiments, another speaker, not shown, may optionally be positioned adjacent to the other ear canal ofthe user to provide stereo / shapeable sound control). The wearable display system 430 may also include one or more microphones or other devices to detect sound. In some embodiments, the microphone is configured to allow the user to provide inputs or commands to the wearable display system 430 (e.g., the selection of voice menu commands, natural language questions, etc.), and / or may allow audio communication with other persons (e.g., with other users of similar display systems). The microphone may further be configured as a peripheral sensor to collect audio data (e.g., sounds from the user and / or environment). In some embodiments, the wearable display system 430 may further include one or more outwardly directed environmental sensors configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user. For example, environmental sensors may include one or more cameras, which may be located, for example, facing outward so as to capture images similar to at least a portion of an ordinary field of view of the user 440. In some embodiments, the wearable display system may also include a peripheral sensor, which may be separate from the frame 434 and attached to the body of the user 440 (e.g., on the head, torso, an extremity, etc. of the user 440). The peripheral sensor may be configured to acquire data characterizing a physiological state of the user 440 in some embodiments. For example, the sensor may be an electrode.

[0048] The display 432 is operatively coupled by a communications link, such as by a wired lead or wireless connectivity, to a local data processing module which may be mounted in a variety of configurations, such as fixedly attached to the frame 434, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 440 (e.g., in a backpack-style configuration, in a belt-coupling style configuration). Similarly, the sensor may be operatively coupled by a communications link, e.g., a wired lead or wireless connectivity, to the local processor and data module. The local processing and data module may comprise a hardware processor, as well as digital memory, such as nonvolatile memory (e.g., flash memory or hard disk drives), both of which may be utilized to assist in the processing, caching, and storage of data. Optionally, the local processor and data module may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on. The data may include data a) captured from sensors (which may be, e.g., operatively coupled to the frame 434 or otherwise attached to the user 440), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, gyros, and / or othersensors disclosed herein; and / or b) acquired and / or processed using remote processing module 452 and / or remote data repository 454 (including data relating to virtual content), possibly for passage to the display 432 after such processing or retrieval. The local processing and data module may be operatively coupled by communication links 438 such as via wired or wireless communication links, to the remote processing and data module 450, which can include the remote processing module 452, the remote data repository 454, and a battery 460. The remote processing module 452 and the remote data repository 454 can be coupled by communication links 456 and communication links 458 to remote processing and data module 450 such that these remote modules are operatively coupled to each other and available as resources to the remote processing and data module 450. In some embodiments, the remote processing and data module 450 may include one or more of the image capture devices, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyros. In some other embodiments, one or more of these sensors may be attached to the frame 434 or may be standalone structures that communicate with the remote processing and data module 450 by wired or wireless communication pathways.

[0049] With continued reference to FIG. 4, in some embodiments, the remote processing and data module 450 may comprise one or more processors configured to analyze and process data and / or image information, for instance including one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on. In some embodiments, the remote data repository 454 may comprise a digital data storage facility, which may be available through the internet or other networking configuration in a "cloud" resource configuration. In some embodiments, the remote data repository 454 may include one or more remote servers, which provide information, e.g., information for generating augmented reality content, to the local processing and data module and / or the remote processing and data module 450. In some embodiments, all data is stored, and all computations are performed in the local processing and data module, allowing fully autonomous use from a remote module. Optionally, an outside system (e.g., a system of one or more processors, one or more computers) that includes CPUs, GPUs, and so on, may perform at least a portion of processing (e.g., generating image information, processing data) and provide information to, and receive information from, the illustrated modules, for instance, via wireless or wired connections.

[0050] FIG. 5 shows a perspective view of a wearable device 500 according to an embodiment of the present invention. Wearable device 500 includes a frame 502 configured to support one or more projectors 504 at various positions along an interior-facing surface of frame 502, as illustrated. In some embodiments, projectors 504 can be attached at positions near temples 506. Alternatively, or in addition, another projector could be placed in position 508. Such projectors may, for instance, include or operate in conjunction with one or more liquid crystal on silicon (LCoS) modules, micro-LED displays, or fiber scanning devices. In some embodiments, light from projectors 504 or projectors disposed in position 508 could be guided into eyepieces 510 for display to the eyes of a user. Projectors placed at positions 512 can be somewhat smaller on account of the proximity this gives the projectors to the waveguide system. The closer proximity can reduce the amount of light lost as the waveguide system guides light from the projectors to eyepiece 510. In some embodiments, the projectors at positions 512 can be utilized in conjunction with projectors 504 or projectors disposed in position 508. While not depicted, in some embodiments, projectors could also be located at positions beneath eyepieces 510. Wearable device 500 is also depicted including sensors 514 and sensors 516. Sensors 514 and sensors 516 can take the form of forwardfacing and lateral-facing optical sensors configured to characterize the real-world environment surrounding wearable device 500.

[0051] Embodiments of the present invention utilize an eye tracking system to determine the eye gaze location of the user and utilize the eye gaze location for image compression processes. Referring to FIG. 5, eye tracking cameras 505 are located on the frame 502 and can be utilized to track the eye gaze location of the user using the wearable device 500. In other embodiments, other eye tracking systems are utilized to determine the eye gaze location and the eye tracking cameras 505 illustrated in FIG. 5 are merely exemplary. As described more fully herein, the image compression processes utilized to compress and decompress virtual content for storage in memory, internal communications, and display, among other functions, can be modified depending on the eye gaze location, for example, portions of an image or video stream corresponding to the eye gaze location can be compressed using a higher quality compression process compared to other portions of the image or video stream that are located more distant from the eye gaze location. Since these more distant portions of the image or video stream are in the user's peripheral vision, any impact on the user experience resulting from the reduction in compression quality can be less than the benefitsachieved in terms of memory and processing efficiency and / or requirements. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0052] FIG. 6 is an exploded perspective view of elements of an eye tracking system for an augmented reality display according to an embodiment of the present invention. Viewable optics assembly 600 includes a rear portion 610, a merged illumination and refractive layer (MILR) structure 620, and a camera flex cable 630. The rear portion 610 may include a rear carrier 612 and define a virtual content output region 614 of an eyepiece. In some embodiments, spacers or spacing components (not shown) may be disposed between the rear carrier 612 and the eyepiece (not shown). The spacers or spacing components may space a front Extended Depth Of Field (EDOF) lens a predetermined distance from the rear carrier 612. The rear carrier 612 may be any material as deemed appropriate. In some embodiments, the rear carrier is magnesium.

[0053] As illustrated in FIG. 6, the MILR structure 620 may have a combined structure that includes an optical element 622, a substrate 624, an illumination structure 626, and a passivation coating 628 applied on the illumination structure 626 along an axis 601. The optical element 622 (e.g., the rear EDOF lens) may include a front (e.g., a world side) planar surface, and the substrate 624 can be bonded to the front planar surface of the optical element 622. In some embodiments, the substrate 624 may be a polyethylene terephthalate (PET) film. The illumination structure 626 may include eye tracking illumination sources (e.g., infrared LEDs), metal traces, and a flex cable surrounding the illumination structure 626 and in electrical communications with the eye tracking illumination sources. The MILR structure 620 implemented in this laminated form thus includes the illumination structure 626, which includes the eye tracking illumination sources, as well as the optical element 622. Light emitted from the eye tracking illumination sources in the illumination structure 626 passes through optical element 622 before reflecting from the eye of the user.

[0054] The optical element 622 may have an optical power and can also be referred to as an EDOF refractive element since it moves the virtual content plane a predetermined distance away from the user's eye, thereby extending the depth of field. In some embodiments, optical element 622 may move the virtual content by a distance on the order of tens of centimeters. In the embodiment illustrated in FIG. 6, the optical element 622 has negative optical power, i.e., it is a negative lens that diverges collimated light received from the eyepiece. Althoughnot shown in FIG. 6, a corresponding front EDOF lens with the opposite optical power (e.g., positive optical power) may be positioned on the world side of the eyepiece in order to counteract the action of optical element 622 with respect to world light.

[0055] The illumination structure 626 may be fabricated by a lamination process. In the lamination process, the substrate 624 (e.g., a PET film substrate) is provided and metal traces (e.g., copper traces) and contact pads may be deposited on the substrate 624. The eye tracking illumination sources (e.g., infrared LEDs) may be mounted and electrically connected to the contact pads. Subsequently, the passivation coating 628 may be applied upon the illumination structure 626. In some embodiments, the passivation coating 628 may be applied by a roll-to-roll process. The substrate 624, the illumination structure 626, and the passivation coating 628 may form a combined structure. The combined structure may then be bonded to the front (e.g., a world side) planar surface of the optical element 622 (e.g., the rear EDOF lens). Accordingly, the eye tracking illumination sources are positioned on the world side of the optical element 622 (e.g., rear EDOF lens) opposite the user's eye and propagate through the rear EDOF lens before reflection from the user's eye.

[0056] Although in FIG. 6 the eye tracking illumination sources are illustrated as infrared (IR) LEDs that are laminated to the substrate 624 (e.g., PET film substrate) and emit light toward the eye side, the MILR structure 620 may have other configurations as desired. In other embodiments, the eye tracking illumination sources may be disposed at locations peripheral to the metal traces of the illumination structure 626, and a waveguiding layer may be utilized in the plane in which the metal traces are disposed. In these embodiments, light from the eye tracking illumination sources propagates in the plane of the waveguiding layer and is outcoupled to provide the eye tracking illumination. The light can be outcoupled using reflective structures, for example, a mirrored surface tilted at -45° to the plane of the waveguiding layer, or diffractive structures, for example, vertical outcoupling gratings disposed in or on the plane of the waveguiding layer. Thus, the IR LEDs illustrated in FIG. 6 can be replaced with illumination regions in which eye tracking illumination is output and directed toward the user's eye. One of ordinary skill in the art would recognize many variations, modifications, and alternatives. Additional information related to waveguiding layers is provided in U.S. Patent No. 11,106,033 and International Patent Application No. PCT / US22 / 71988, the disclosures of which are hereby incorporated by reference in their entirety for all purposes.

[0057] In some embodiments, a set of cameras 632 may be disposed on the camera flex cable 630 to detect illumination light reflected from the eye. The camera flex cable 630 and the set of cameras 632 may be disposed on the eye side of optical element 622. In some embodiments, the set of cameras 632 may be incorporated into a wearable device similar to how the eye tracking cameras 505 are incorporated into the wearable device 500 illustrated in FIG. 5.

[0058] FIG. 7 is a simplified plan view of elements of an AR headset including eye tracking according to an embodiment of the present invention. In FIG. 7, elements of a Visible Optics Assembly (VOA) are illustrated. The VOA 700 includes a front Extended Depth Of Field (EDOF) refractive element 710 and front optics 712 that receive world light propagating from left to right toward the eye 770 of the user. The VOA 700 also includes a dimmer 702, an eyepiece 704, rear EDOF and eye tracking (ET) structure 706, and optional prescription lens insert 760. Dimmer 702 includes a world side linear polarizer 720, a first quarter waveplate 722, a liquid crystal panel 724, a second quarter waveplate 726, and an eye side linear polarizer 728, for example, a hard coated linear polarizer (HC-LP) with a surface open to air hard coated for handling purposes. Eyepiece 704 includes three eyepiece waveguide layers: blue active layer 730, green active layer 732, and red active layer 734.Although a three-layer eyepiece (i.e., an eyepiece including three eyepiece waveguide layers) is illustrated in FIG. 7, this is not required and in other embodiments, a six-layer eyepiece structure can be utilized with, for example, two depth planes.

[0059] Rear EDOF and ET structure 706 includes rear EDOF 752, illumination layer 750 and imaging layer 754 utilized in eye tracking. Eye-tracking cameras 505 illustrated in FIG. 5 can be utilized in the imaging layer 754 to capture images of the eye 770. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0060] FIGS. 8 A and 8B show plan views of an arrangement of optical elements for eye tracking. FIG. 8A is a simplified plan view of an AR headset 800 including eye tracking using a reflective optical element 806 according to an embodiment of the present invention. In the arrangement shown in FIG. 8A, an eye tracking camera 802 may be placed on a temporal arm 804 of the AR headset 800. The reflective optical element 806 can be included as part of the optical assembly 808 of the AR headset 800. As described above, the optical assembly 808 can be an example of the VOA 700 of FIG. 7, so that the reflective opticalelement 806 can be incorporated into the optical assembly 808. For example, the reflective optical element can be disposed on the world side of the rearmost refractive optical element (e.g., rear EDOF 752 of FIG. 7) in the optical assembly 808.

[0061] As shown in FIG. 8 A, light 812 reflected from the eye 810 can propagate through eye side optics of the optical assembly 808 and impinge on the reflective optical element 806. The reflected light 814 from the reflective optical element 806 can then propagate back through the eye side optics of the optical assembly 808 and be received at the eye tracking camera 802. Although not shown in FIGS. 8A or 8B, the AR headset 800 can include a similar arrangement of components on the other side for tracking the user's other eye.

[0062] FIG. 8B is a simplified plan view of an AR headset 820 including eye tracking using a reflective optical element 806 and a prescription lens 822 according to an embodiment of the present invention. The AR headset 820 can be the same as AR headset 800 except for the inclusion of the prescription lens 822. In some examples, the prescription lens 822 may be a component of a user's eyeglasses worn in conjunction with the AR headset 820. For example, the AR headset 820 may be worn over and enclose the user's eyeglasses including the prescription lens 822. In other examples, including several of the embodiments described herein, the prescription lens 822 can be a component of the AR headset 820 (e.g., prescription lens insert 760 of FIG. 7). For example, the prescription lens 822 can be included in a frame that is removably couplable to the frame of the AR headset 820, so that the prescription lens 822 is positioned between the eye 810 and the optical assembly 808.

[0063] By comparison with FIG. 8A, light 824 that reflects off the eye 810 can propagate through the prescription lens 822, through the eye side optics of the optical assembly 808 and impinge on the reflective optical element 806. The reflected light 826 can then propagate back through the eye side optics of the optical assembly 808 and again through the prescription lens 822. The eye tracking camera 802 mounted on the temporal arm 804 can receive the reflected light 826 from the reflective optical element 806.

[0064] The prescription lens 822 can have positive or negative optical power depending on the correction utilized by the user. For example, the prescription lens 822 can have a negative optical power to correct for a user's nearsightedness or a positive optical power to correct for a user's farsightedness. The prescription lens 822 can also be configured to correct for astigmatism and have parameters that characterize the optical power differently fordifferent positions around the optical axis of the prescription lens 822. Because of the variation in optical power for different prescription lenses used with an AR headset 820, propagating the light 824 and the reflected light 826 through the prescription lens 822 can introduce significant distortion to an image of the eye 810 formed at the eye tracking camera 802.

[0065] FIG. 9 is a simplified plan view of an AR headset 900 with eye tracking using an off-axis mirror (e.g., reflective optical element 906), according to an embodiment of the present invention. The AR headset 900 may be similar to AR headset 820 of FIG. 8. For example, AR headset 900 can include a frame having a temporal arm 904 and holding an optical assembly 908 (one for each eye, e.g., eye 916 of the user). The optical assembly 908 can be an example of the VOA 700 of FIG. 7A. A prescription lens 910 can be positioned between the eye 916 and the optical assembly 908 (e.g., at the eye side of the optical assembly 908). In some examples, the prescription lens 910 can be mounted in an insert that is coupled to (e.g., removably connected to by inserting into the eye side of the optical assembly 908) the frame of the AR headset 900. As used herein, the prescription lens can be referred to as a first refractive optical element to distinguish from other refractive optical elements within the optical assembly 908. For example, optical assembly 908 can include a rear EDOF at the eye side of the optical assembly 908 that is a refractive optical element of the optical assembly 908.

[0066] As shown in FIG. 9, an eye tracking camera 902 can be positioned laterally adjacent to eye side optics of the optical assembly 908. In some embodiments, the reflective optical element 906 can include an off-axis mirror disposed on the world side of the second refractive optical element in the optical assembly 908. For example, the second refractive optical element can be the rear EDOF (e.g., rear EDOF 752 of FIG. 7). The off-axis mirror can be disposed on the world side of the rear EDOF. In some examples, the reflective optical element 906 can be a separate optical component coupled with the second refractive optical element. In other examples, the reflective optical element 906 can be formed from a portion of the first refractive optical element. For example, the reflective optical element 906 can be an off-axis mirror including a diffraction grating, wherein the diffraction grating is formed onto the world side of the rear EDOF.

[0067] In embodiments in which the reflective optical element 906 includes a diffraction grating, the diffraction grating can be a linear grating (e.g., a blazed grating or an echelette). For example, the linear diffraction grating can be characterized by a constant pitch in one direction of the periodic grating rulings, with the ruling profile configured for reflecting the particular wavelength (e.g., infrared) of light 912 from the eye to the eye tracking camera 902 while allowing world light and other light within the system (e.g., light emitted from eyepiece 704 of FIG. 7) to pass through the optical assembly 908. In some examples, the diffraction grating can be a non-linear grating characterized by grating parameters (e.g., grating ruling profile dimensions, pitch, etc.) varying in one or more directions (e.g., x- and y-dimensions of the grating surface). The parameters characterizing the diffraction grating can be configured by specifying the phase of reflected light at each point on the grating with respect to light impinging on the surface (e.g., light 912 from eye 916). One skilled in the art would recognize how to construct a grating from such phase parameters. Additional details about the grating parameters suitable for embodiments of the present invention are provided below with respect to FIG 10A. It is advantageous to construct a grating that has narrow band response which only reflects the near infrared light (e.g., 850 nm) while being transparent to visible light. Specific examples of this type of grating include a liquid crystal polarization grating (LCPG) or a volume phase hologram (VPH) grating.

[0068] As depicted in FIG. 9, light 912 from the eye 916 of the user of the AR headset 900 can pass through the prescription lens 910 and impinge on the reflective optical element 906. Before impinging on the reflective optical element, the light 912 can propagate through the rear EDOF. Reflected light 914 can reflect from the reflective optical element 906 to the eye tracking camera 902. The reflected light 914 can also propagate through a portion of the rear EDOF. Advantageously, the reflected light 914 may avoid propagating through the prescription lens 910, preventing the prescription lens from affecting the light 912 from the eye 916 a second time.

[0069] As described above with respect to FIG. 6, light 912 may originate as illumination light produced by eye tracking illumination sources from an eye tracking illumination structure (e.g., eye tracking illumination structure 626). The eye tracking illumination sources can be infrared (IR) LEDs, so that the illumination light can be infrared (IR) light projected toward the eye 916 from the eye tracking illumination structure in the optical assembly 908. As shown in embodiments below with respect to FIG. 10B, the reflectiveoptical element 906 may only be disposed over a portion of the rear EDOF so that illumination light can be freely transmitted through the rear EDOF toward the eye 916 but can be reflected toward the eye tracking camera when the light 912 impinges on the reflective optical element 906. In some embodiments, the reflective optical element 906 may be configured to transmit illumination light from the world side to the eye side at a central portion of the reflective optical element 906 but configured to reflect the light 912 at a peripheral portion of the reflective optical element 906. The configuration of the reflective optical element 906 can include modifying the parameters of the diffraction grating at the central portion and the peripheral portion to achieve suitable transmission and reflection characteristics for the wavelength of illumination light used.

[0070] In comparison with the set of eye tracking cameras 632 of FIG. 6, the eye tracking camera 902 can be oriented to receive reflected light 914 reflected from the reflective optical element 906 disposed on the world side of the rear EDOF. Such positioning and orientation can allow for the eye tracking camera 902 to be positioned further outward from the center of the optical axis (e.g., axis 601 of FIG. 6) of the optical assembly 908. Positioning the eye tracking camera 902 laterally adjacent to the rear EDOF with the reflective optical element 906 can provide a larger clear viewing area for the optical assembly 908 than even the notched configuration of the eye tracking components of VOA 600. For example, the cameras can be embedded in the frame structure (e.g., in temporal arm 904) to not protrude into the eye side region between the rear EDOF and the eye 916 of the user. By enlarging the clear viewing area of the optical system, the user's visual region is less obstructed by components of the optical assembly 908 and can have a clearer view of the virtual content produced by the AR headset 900 (e.g., virtual content in virtual content output region 614 from an eyepiece).

[0071] As used herein, the term "laterally adjacent" can include positions for the eye tracking camera 902 as depicted in FIG. 9 and FIGS. 10A & 10B, 11 A, and 12A shown below. Laterally adjacent positions can include positions radially further from an optical axis of the rear EDOF (second refractive optical element) and / or of the reflective optical element than the radial edge of the rear EDOF and / or reflective optical element. With respect to position along the optical axis, a laterally adjacent eye tracking camera 902 can be positioned in a plane defining the position along the optical axis of the rear EDOF, positioned on the eye side of the plane defining the position along the optical axis of the rear EDOF. In someembodiments, a laterally adjacent eye tracking camera 902 can be positioned so that reflected light 914 from the reflective optical element 906 propagates out through an aperture at the edge of the rear EDOF and forms an image of the eye 916 at the eye tracking camera 902 without the reflected light 914 propagating through the prescription lens 910. In these embodiments, a "laterally adjacent" position for the eye tracking camera 902 can be any position where the eye tracking camera 902 is coupled to the frame of the AR headset 900 so that the reflected light 914 can form an image of the eye 916 without the reflected light passing through the prescription lens 910. In some embodiments, the laterally adjacent positions could be above or below the optical axis of the optical assembly 908 (e.g., in an x- dimension of the optical system). For example, the eye tracking camera 902 could be positioned laterally adjacent within the bottom portion of the frame holding the optical assembly 908.

[0072] FIG. 10A is a simplified plan view of elements of an eye tracking system 1000 using an off-axis mirror 1004, according to an embodiment of the present invention. The eye tracking system 1000 can include a rear EDOF 1002, an off-axis mirror 1004, and an eye tracking camera 1006. The off-axis mirror 1004 and the eye tracking camera 1006 may be examples of the reflective optical element 906 and the eye tracking camera 902 described above with respect to FIG. 9. The eye tracking system 1000 may be used as part of an AR headset (e.g., AR headset 900 of FIG. 9) for eye tracking when a prescription lens 1010 is present. The prescription lens 1010 may be similar to prescription lens 910 of FIG. 9 and may be referred to as a first refractive optical element. The rear EDOF 1002 may be an example of the rear EDOF in the optical assembly of an AR headset (e.g., rear EDOF 752 of VOA 700 of FIG. 7). The rear EDOF 1002 may be referred to as a second refractive optical element. Light 1014 reflected from the eye 1012 of a user of an AR headset including the eye tracking system 1000 can pass through the first refractive optical element (prescription lens 1010) and then through the second refractive optical element (rear EDOF 1002) before impinging on a reflective optical element (off-axis mirror 1004).

[0073] The eye tracking camera 1006 can be positioned laterally adjacent to the rear EDOF 1002. Reflected light 1016 from the off-axis mirror 1004 can propagate through the rear EDOF 1002 and be received at the eye tracking camera 1006. As shown in FIG. 10A, light 1014 from the eye 1012 to be reflected to the eye tracking camera 1006 may impinge at the off-axis mirror 1004 with an incidence angle greater than 0° (hence, "off-axis" from theprincipal optical axis of the optical assembly). The off-axis mirror 1004 can be configured to reflect the reflected light 1016 toward the eye tracking camera 1006. In some embodiments, the off-axis mirror 1004 can be disposed on the world side of the rear EDOF 1002. The off- axis mirror 1004 can be disposed over substantially all of the world side of the rear EDOF 1002.

[0074] The reflected light 1016 can exit the rear EDOF 1002 through a lateral aperture 1008. The lateral aperture 1008 can be formed as a window at the edge of the rear EDOF 1002. For example, the rear EDOF 1002 can be a circular or elliptical shaped lens with an edge around the perimeter. A portion of the edge of the rear EDOF 1002 can be the lateral aperture 1008. In some embodiments, the lateral aperture 1008 can be oriented to transmit the reflected light 1016 out from the rear EDOF 1002. For example, the lateral aperture 1008 can be a surface parallel to the optical axis of the rear EDOF 1002. As another example, the lateral aperture 1008 can be a surface parallel to the image plane of the eye tracking camera 1006 (e.g., as shown with lateral aperture 1028 of FIG. 10B, below). The lateral aperture 1008 can be disposed at a lateral side of the rear EDOF 1002 between the eye side and the world side of the refractive optical element. The lateral side of the rear EDOF 1002 can be the side of the lens at the periphery away from the center of the lens defined by the optical axis. In some embodiments, the lateral aperture 1008 may be a portion of the lateral side of the rear EDOF 1002. For example, a window may be formed in the edge of the rear EDOF 1002 lens near the eye tracking camera 1006 to allow the reflected light 1016 to pass through the lateral aperture 1008. One skilled in the art would recognize many variations for a window in a lens as lateral aperture 1008 for transmitting reflected light.

[0075] The off-axis mirror 1004 can be configured to reflect the light 1014 impinging on the off-axis mirror 1004 from the eye 1012 to produce reflected light 1016. The off-axis mirror 1004 may be a reflective diffraction grating. Because the light 1014 may have a characteristic wavelength (e.g., 850 nm) and may impinge on the off-axis mirror 1004 with a relatively large incidence angle, the diffraction grating of the off-axis mirror 1004 can be configured to optimally reflect the characteristic wavelength of the light 1014 to produce reflected light 1016 that propagates through the rear EDOF 1002 and out from the lateral aperture 1008 to form an image of the eye 1012 at the eye tracking camera 1006. The diffraction grating of the off-axis mirror 1004 may be a linear grating having rulings characterized by a uniform profile and uniform pitch (ruling spacing) in the y-direction (asindicated by the coordinate axes). However, due to the off-axis incidence of the light 1014, particularly from an extended source like the eye 1012 that is relatively close to the off-axis mirror 1004, a linear diffraction grating may produce reflected light 1016 that forms a distorted image at the eye tracking camera 1006. To produce an image of the eye 1012 that is suitable for eye tracking, the diffraction grating of the off-axis mirror can be configured using higher-order parameters. For example, the diffraction grating may be a non-linear grating characterized by grating phase parameters given by the following polynomial equation:where (p is the phase added to the incident light at each point of the grating and Cm nare phase parameters defining the polynomial equation. The polynomial equation can be defined for an x-y coordinate system in the plane of the off-axis mirror 1004 (e.g., the x-y coordinate system defined by the y-z axes shown in FIG 10A, with the x-axis oriented into the page).

[0076] As a particular example of parameters for the off-axis mirror 1004, the light 1014 may be 850 nm infrared light. For light diffracted into the first diffraction order, the grating parameters can be given as shown in Table 1, below:Table 1 : Example Grating Parameters

[0077] As shown in Table 1, for a grating that is symmetric in the x-direction, odd orders in x vanish. Then, parameters up to the fourth order in y can determine a surface profile for the diffraction grating. The surface profile for the fourth-order parameters given in Table 1 characterizes a grating having a pitch for rulings that varies in both the x and y directions. One skilled in the art would recognize several methods for producing a reflective diffraction grating characterized by a polynomial phase equation. The parameters given in Table 1 can be generated by modeling the optical system of the eye tracking system 1000 in FIG. 10 A. The position and orientation of the eye tracking camera 1006 can determine the reflection angle, and therefore, the diffraction order m, for reflected light 1016 from the surface of the off-axis mirror 1004.

[0078] For the linear grating case, only the parameters C0 1(e.g., y-direction pitch) or C1 0(e.g., x-direction pitch) characterize the grating. For example, a phase parameter of C0,i=—0.607 indicates a grating having a constant pitch of 1.4 pm in the y-direction.

[0079] FIG. 10B is an example optical schematic including elements of an eye tracking system 1020 using an off-axis mirror 1024, according to an embodiment of the present invention. Components of the eye tracking system 1020 may be similar to the eye tracking system 1000 described above with respect to FIG. 10 A. For example, rear EDOF 1022 may be an example of rear EDOF 1002. In the embodiment shown in FIG. 10B, the off-axis mirror 1024 may be disposed at a lateral portion of the world side surface of the rear EDOF 1002. For example, the off-axis mirror 1024 may be disposed at the periphery away from the optical axis of the rear EDOF 1002.

[0080] In the example optical schematic of FIG. 10B, the eye 1012 shown in FIG. 10A has been replaced by an object plane 1030 representing an extended source object at the location of the eye. Similarly, the eye tracking camera is represented by image plane 1026. The prescription lens has been omitted for clarity in the optical schematic; however, the refractive effects of the prescription lens can be accounted for by the object plane 1030 (e.g., the object plane 1030 can represent the image of the eye 1012 produced from light through the prescription lens).

[0081] Marginal rays from the object plane 1030 (represented by the dashed and dot- dashed lines) propagate through the rear EDOF 1022 and impinge on the off-axis mirror 1024. The reflected light from the off-axis mirror 1024 then propagates through the rearEDOF 1022 and out lateral aperture 1028 and forms an image at the image plane 1026. As shown in FIG. 10B, the lateral aperture 1028 may form a surface of the edge of the rear EDOF 1022 so that the surface is parallel to the image plane 1026. The off-axis mirror 1024 can be characterized by parameters (e.g., diffraction grating phase parameters for a polynomial phase equation as described with respect to FIG. 10 A) to produce the image at image plane 1026 from the object plane 1030. As described in more detail below in FIGS. 11 A-12C, modeling of the optical system using known parameters for the position and / or orientation of the object plane 1030, the position and / or orientation of the image plane 1026, the geometric path lengths of the light rays, and the material parameters of the rear EDOF 1022 can produce parameters characterizing the off-axis mirror 1024.

[0082] FIG. 11 A is an example optical schematic of a portion of an eye tracking system 1100 including modeled ray tracing for imaging at a decentered camera sensor (represented by image plane 1106), according to an embodiment of the present invention. The eye tracking system 1100 can include a refractive optical element 1102 (e.g., rear EDOF 1002 of FIG. 10A) and a reflective optical element 1104 (e.g., reflective optical element 1104 of FIG. 10A). As with FIG. 10B, the object to be imaged (e.g., an eye 1012 of a user) can be represented by object plane 1112, and the camera sensor on which the image is to be formed (e.g., eye tracking camera 1006) can be represented by image plane 1106. The refractive optical element 1102 can include a lateral aperture 1108 through which reflected light from the reflective optical element 1104 exits the refractive optical element 1102.

[0083] For modeling the eye tracking system 1100, the object can be represented by a grid array of point sources located at the object plane 1112. The grid array can be defined in the x-y plane of the object plane with reference to the y-z coordinate system shown in FIG. 11 A, with the x direction into the page. The rays from the grid array can include marginal rays 1114 and marginal rays 1116 as well as interior rays 1118. The marginal rays 1114 are represented by a "x," the marginal rays 1116 are represented by a "+," and the interior rays 1118 are represented by a "o." Additional marginal rays for the extent of the grid array in the x-direction are not shown in the view of FIG. 11 A but are represented byin the views of FIGS. 11B and 11C.

[0084] The image plane 1106 may represent the camera sensor for a decentered camera. For example, the eye tracking camera can include one or more optical elements (e.g., lenses,mirrors) that are configured to correct for off-axis propagation of light through the camera optics. The decentered camera sensor may be tilted in the y-z plane as shown in FIG. 11 A, and / or the optical elements of the eye tracking camera may be configured with one or more asymmetries with respect to the optical axis. For example, a lens of the camera may be positioned off center with respect to the optical axis of incoming light. The decentered eye tracking camera can therefore be characterized by a decenter parameter. A decentered eye tracking camera may help correct distortions to an image formed at the camera sensor due to off-axis propagation of light (e.g., light from the object plane 1112 having a high angle of incidence on the refractive optical element 1102 and / or reflective optical element 1104). As a trade-off, decentered eye tracking cameras can be expensive for use in eye tracking systems for AR headsets.

[0085] FIG. 1 IB illustrates an example image 1120 formed at the decentered camera sensor having image plane 1106 in FIG. 11 A. The image 1120 is formed from the grid array in the object plane 1112 of FIG. 11 A. The image 1120 is formed using an off-axis mirror (e.g., reflective optical element 1104) having a linear diffraction grating. For example, the diffraction grating can be characterized by a phase parameter Co,i, so that the diffraction grating has a constant pitch in the y direction. The image 1120 may be formed at image plane 1106 of FIG. 11 A.

[0086] As shown in FIG. 1 IB, the image 1120 has noticeable perspective distortion (keystone distortion). Image points formed by marginal rays 1114 show spread in the x- direction and y-direction with respect to image points formed by marginal rays 1116.

[0087] FIG. 11C illustrates another example image 1130 formed at the decentered camera sensor image plane 1106 in FIG. 11 A. The image 1130 is formed using an off-axis mirror (e.g., off-axis mirror 1004) having a diffraction grating characterized by fourth-order grating parameters from a polynomial phase equation (e.g., Equation (1) described above with respect to FIG. 10A). For example, the diffraction grating can be characterized by phase parameters similar to those of Table 1. As a result, the grating pitch and / or ruling profiles can vary in both the x and y directions. The image 1130 may be formed at image plane 1106 of FIG. 11 A. The grating parameters used with a decentered camera sensor may be referred to as first grating parameters.

[0088] In contrast with image 1120, image 1130 shows significantly reduced perspective distortion due to the fourth-order parameters. The marginal rays 1114 still exhibit spread in the y-direction as compared to other points in the grid array. However, the groupings of image points are approximately arranged in a regular grid as expected for an accurate image of a grid array of point sources.

[0089] FIG. 12A is an example optical schematic of a portion of an eye tracking system 1200 including modeled ray tracing for imaging at a camera sensor (represented by image plane 1206), according to an embodiment of the present invention. In comparison with the decentered camera sensor of FIG. 11 A, the camera sensor of the eye tracking system 1200 may not include decentered optical elements. The camera sensor can therefore be an off-the- shelf component that may be suitable for use in AR headset eye tracking systems.

[0090] The eye tracking system 1200 can include a refractive optical element 1202 (e.g., rear EDOF 1002 of FIG. 10A) and a reflective optical element 1204 (e.g., off-axis mirror 1004 of FIG. 10A). The object to be imaged (e.g., an eye 1012 of a user) can be represented by object plane 1212, and the camera sensor on which the image is to be formed (e.g., eye tracking camera 1006) can be represented by image plane 1206. The refractive optical element 1202 can include a lateral aperture 1208 through which reflected light from the reflective optical element 1204 exits the refractive optical element 1202.

[0091] Similar to FIG. 11 A, for modeling the eye tracking system 1200, the object can be represented by a grid array of point sources located at the object plane 1212. The grid array can be defined in the x-y plane of the object plane with reference to the y-z coordinate system shown in FIG. 12 A, with the x direction into the page. The rays from the grid array can include marginal rays 1214 and marginal rays 1216 as well as interior rays 1218. The marginal rays 1214 are represented by a "x," the marginal rays 1216 are represented by a "+," and the interior rays 1218 are represented by a "o." Additional marginal rays for the extent of the grid array in the x-direction are not shown in the view of FIG. 12A but are represented by the views of FIGS. 12B and 12C.

[0092] FIG. 12B illustrates an example image 1220 formed at the camera sensor having image plane 1206 in FIG. 12A. The image 1220 is formed from the grid array in the object plane 1212 of FIG. 12A. The image 1220 is formed using an off-axis mirror (e.g., reflective optical element 1204) having a linear diffraction grating. For example, the diffraction gratingcan be characterized by a phase parameter Co.i, so that the diffraction grating has a constant pitch in the y direction. The image 1220 may be formed at image plane 1206 of FIG. 11 A. As with image 1120 of FIG. 1 IB, image 1220 exhibits significant perspective distortion.

[0093] FIG. 12C illustrates another example image 1230 formed at the camera sensor image plane 1206 in FIG. 12A. The image 1130 is formed using an off-axis mirror (e.g., reflective optical element 1204) having a diffraction grating characterized by fourth-order grating parameters from a polynomial phase equation (e.g., Equation (1) described above with respect to FIG. 10A). For example, the diffraction grating can be characterized by a phase parameters similar to those of Table 1. As a result, the grating pitch and / or ruling profiles can vary in both the x and y directions. The image 1230 may be formed at image plane 1206 of FIG. 12 A. The grating parameters characterizing the reflective optical element 1204 may be referred to as second grating parameters.

[0094] The image 1230 shows reduced perspective distortion due to the fourth-order parameters. In comparison with image 1130 of FIG. 11C, which uses a decentered camera sensor, the perspective distortion is not reduced as much when imaging with a standard camera sensor even with a diffraction grating characterized by fourth-order parameters. The image points from marginal rays 1214 in image 1230 exhibit more x-direction spread than the image points from marginal rays 1114 in image 1130.

[0095] FIG. 13 is a simplified flowchart illustrating a method 1300 of performing eye tracking according to an embodiment of the present invention. The method 1300 includes reflecting light from an eye of a user (1310). The light may be illumination light produced by illumination sources, including illumination structure 626 of FIG. 6. The illumination light may be infrared light (e.g., 850 nm). Referring to FIG. 9, the light 912 may be the illumination light reflected from the eye 916.

[0096] The method 1300 can also include propagating the light through a first refractive optical element (1312) and then propagating the light through a second refractive optical element (1314). The first refractive optical element can be a prescription lens. The second refractive optical element can be a rear EDOF in an optical assembly of an AR headset. Referring to FIG. 10 A, the first refractive optical element can be a prescription lens 1010 and the second refractive optical element can be the rear EDOF 1002. The light 1014 reflects from the eye 1012 and propagates through the prescription lens 1010 and the rear EDOF1002. The first refractive optical element can have a positive optical power or a negative optical power. For example, a prescription lens to correct a user's vision can have either a positive optical power or a negative optical power. The second refractive lens (e.g., rear EDOF 1002) can move a virtual content plane a predetermined distance away from the user's eye, thereby extending the depth of field. In some embodiments, the second refractive optical element can move the virtual content by a distance on the order of tens of centimeters. In some embodiments, the second refractive optical element (e.g., rear EDOF 1002) may have negative optical power, i.e., it is a negative lens that diverges collimated light received from the eyepiece. Although not shown, a corresponding front EDOF lens with the opposite optical power (e.g., positive optical power) may be positioned on the world side of the eyepiece in order to counteract the action of the second refractive optical element (e.g., rear EDOF 1002) with respect to world light.

[0097] The method 1300 can also include impinging the light on a reflective optical element (1316) and reflecting the light from the reflective optical element (1318). Still referring to FIG. 10A, the reflective optical element may be off-axis mirror 1004. The reflective optical element can include a diffraction grating (e.g., a reflective diffraction grating). In some embodiments, the reflective optical element is configured to reduce a perspective distortion in an image formed with the reflected light. Referring to FIGS. 11 A- 12C, in some embodiments, the reflective optical element can be reflective optical element 1104 or reflective optical element 1204 and characterized by grating parameters for a reflective diffraction grating. For embodiments in which a decentered camera sensor is used in the eye tracking system, the grating parameters can include first grating parameters specified by a fourth-order polynomial equation modeled with the decentered camera sensor. For embodiments in which an on-axis camera sensor is used in the eye tracking system, the grating parameters can include second grating parameters specified by a fourth-order polynomial equation modeled with the on-axis camera sensor. Reflecting the light from the reflective optical element can include diffracting the characteristic wavelength of the light (e.g., 850 nm) into a diffraction order (e.g., n=l order) of the diffraction grating.

[0098] After the reflected light reflects from the reflective optical element, the method 1300 can include propagating the reflected light through the second refractive optical element to a lateral aperture of the second refractive optical element (1320). Referring to FIG. 10A, the lateral aperture can be an example of lateral aperture 1008, which may be a windowformed on or at the edge of the rear EDOF 1002. The lateral aperture may be formed at a portion of the rear EDOF 1002 facing the eye tracking camera 1006.

[0099] The method 1300 also includes detecting the reflected light at a camera (or a set of cameras) (1322). The camera may be the eye tracking camera 1006 that is incorporated into a wearable device including an AR headset. The eye tracking camera 1006 captures images that include the eye as well as reflected light produced by eye tracking illumination sources (not shown) after reflection from the user's eye. In some embodiments, the camera (e.g., eye tracking camera 1006) can be disposed laterally adjacent to the second refractive element. The reflected light from the reflective optical element may be received at the camera after exiting the lateral aperture and without propagating through the first refractive optical element.

[0100] The method 1300 also includes outputting an eye tracking signal from the camera (1324). The eye tracking signal may be in any suitable form including electrical signals produced by the camera sensor. The method 1300 also includes tracking the eye based on the eye tracking signal from the camera (1326).

[0101] In some embodiments, the illumination sources (e.g., illumination structure 626 of FIG. 6) may be disposed on a side of the second refractive optical element opposite the user (e.g., the world side of the second refractive optical element. The illumination light may be propagated through the second refractive optical element prior to reflecting the light from the eye of the user. The illumination light can then be propagated through the first refractive optical element. Referring to FIG. 6, illumination light from illumination structure 626 can propagate toward the eye side through the optical element 622 (the second refractive optical element, which can be a rear EDOF as described above) and through a prescription lens (not shown) in front of the user's eye. Rather than the eye tracking cameras 632 detecting the illumination light reflected from the eye, the eye tracking cameras positioned laterally adjacent to the second refractive optical element can detect reflected light from the reflective optical element disposed on the world side of the second refractive optical element.

[0102] The method 1300 can also include producing virtual content from an output region of an eyepiece, propagating the virtual content through a plane in which the plurality of illumination sources are disposed, and impinging the virtual content on the eye of the user. Still referring to FIG. 6, the rear carrier 612 can hold an eyepiece (not shown) from whichvirtual content can be projected (as from virtual content output region 614) toward the user's eye. The virtual content can pass through the plane having the illumination sources (e.g., the plane of the illumination structure 626). After the light of the virtual content passes through optical element 622 (rear EDOF), the virtual content impinges on the user's eye. The effect of the rear EDOF is to project the virtual content as coming from the virtual content output region 614. The virtual content can also propagate through the first refractive optical element (e.g., prescription lens 910 of FIG. 9).

[0103] It should be appreciated that the specific steps illustrated in FIG. 13 provide a particular method of eye tracking according to an embodiment of the present invention. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in FIG. 13 may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0104] FIG. 14 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention. AR system 1400 as illustrated in FIG. 14 may be incorporated into the AR devices as described herein. FIG. 14 provides a schematic illustration of one embodiment of AR system 1400 that can perform some or all of the steps of the methods provided by various embodiments. It should be noted that FIG. 14 is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. FIG. 14, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

[0105] AR system 1400 is shown comprising hardware elements that can be electrically coupled via a bus 1405, or may otherwise be in communication, as appropriate. The hardware elements may include one or more processors 1410, including without limitation one or more general-purpose processors and / or one or more special-purpose processors such as digital signal processing chips, graphics acceleration processors, and / or the like; one or more input devices 1430, which can include without limitation a mouse, a keyboard, a camera, and / or the like; and one or more output devices 1440, which can include withoutlimitation a display device, a printer, and / or the like. Additionally, AR system 1400 includes an eye tracking system 1470 that can provide the user's eye gaze location to the AR system. Utilizing one or more processors 1410, the eye tracking techniques discussed herein can be implemented.

[0106] AR system 1400 may further include and / or be in communication with storage device(s) 1420 (e.g., one or more non-transitory storage devices), which can comprise, without limitation, local and / or network accessible storage, and / or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device, such as a random access memory (RAM), and / or a read-only memory (ROM), which can be programmable, flash-updateable, and / or the like. Such storage devices may be configured to implement any appropriate data stores, including without limitation, various file systems, database structures, and / or the like.

[0107] AR system 1400 might also include a communications subsystem 1450, which can include without limitation a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc., and / or the like. Communications subsystem 1450 may include one or more input and / or output communication interfaces to permit data to be exchanged with a network such as the network described below to name one example, other computer systems, television, and / or any other devices described herein. Depending on the desired functionality and / or other implementation concerns, a portable electronic device or similar device may communicate image and / or other information via communications subsystem 1450. In other embodiments, a portable electronic device, e.g., the first electronic device, may be incorporated into AR system 1400, e.g., an electronic device as an input device 1430. In some embodiments, AR system 1400 will further comprise a working memory 1460, which can include a RAM or ROM device, as described above.

[0108] AR system 1400 also can include software elements, shown as being currently located within working memory 1460, including an operating system 1462, device drivers, executable libraries, and / or other code, such as one or more application programs 1464, which may comprise computer programs provided by various embodiments, and / or may be designed to implement methods, and / or configure systems, provided by other embodiments,as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above might be implemented as code and / or instructions executable by a computer and / or a processor within a computer; in an aspect, then, such code and / or instructions can be used to configure and / or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.

[0109] A set of these instructions and / or code may be stored on a non-transitory computer- readable storage medium, such as storage device(s) 1420 described above. In some cases, the storage medium might be incorporated within a computer system, such as AR system 1400. In other embodiments, the storage medium might be separate from a computer system e.g., a removable medium, such as a compact disc, and / or provided in an installation package, such that the storage medium can be used to program, configure, and / or adapt a general-purpose computer with the instructions / code stored thereon. These instructions might take the form of executable code, which is executable by AR system 1400 and / or might take the form of source and / or installable code, which, upon compilation and / or installation on AR system 1400, e.g., using any of a variety of generally available compilers, installation programs, compression / decompression utilities, etc., then takes the form of executable code.

[0110] It will be apparent to those skilled in the art that substantial variations may be made in accordance with specific requirements. For example, customized hardware might also be used, and / or particular elements might be implemented in hardware, software including portable software, such as applets, etc., or both. Further, connection to other computing devices such as network input / output devices may be employed.

[0111] As mentioned above, in one aspect, some embodiments may employ a computer system such as AR system 1400 to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all of the procedures of such methods are performed by AR system 1400 in response to one or more processors 1410 executing one or more sequences of one or more instructions, which might be incorporated into operating system 1462 and / or other code, such as an application program 1464, contained in working memory 1460. Such instructions may be read into working memory 1460 from another computer-readable medium, such as one or more of storage device(s) 1420. Merely by way of example, execution of the sequences of instructions contained in working memory 1460 might cause one or more processors 1410 to perform oneor more procedures of the methods described herein. Additionally or alternatively, portions of the methods described herein may be executed through specialized hardware.

[0112] The terms machine-readable medium and computer-readable medium, as used herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using AR system 1400, various computer-readable media might be involved in providing instructions / code to one or more processors 1410 for execution and / or might be used to store and / or carry such instructions / code. In many implementations, a computer-readable medium is a physical and / or tangible storage medium. Such a medium may take the form of a non-volatile media or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device(s) 1420. Volatile media include, without limitation, dynamic memory, such as working memory 1460.

[0113] Common forms of physical and / or tangible computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0114] Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to one or more processors 1410 for execution. Merely by way of example, the instructions may initially be carried on a magnetic disk and / or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by AR system 1400.

[0115] Communications subsystem 1450 and / or components thereof generally will receive signals, and bus 1405 then might carry the signals and / or the data, instructions, etc. carried by the signals to working memory 1460, from which one or more processors 1410 retrieves and executes the instructions. The instructions received by working memory 1460 may optionally be stored on storage device(s) 1420, e.g., a non-transitory storage device, either before or after execution by one or more processors 1410.

[0116] Various examples of the present disclosure are provided below. As used below, any reference to a series of examples is to be understood as a reference to each of those examples disjunctively (e.g., "Examples 1-4" is to be understood as "Examples 1, 2, 3, or 4").

[0117] Example 1 is a method for performing eye tracking, the method comprising: reflecting light from an eye of a user; propagating the light through a first refractive optical element; propagating the light through a second refractive optical element; impinging the light on a reflective optical element; reflecting the light from the reflective optical element; propagating the reflected light through the second refractive optical element to a lateral aperture of the second refractive optical element; detecting the reflected light at a camera; outputting an eye tracking signal from the camera; and tracking the eye based on the eye tracking signal from the camera.

[0118] Example 2 is the method of Example 1 wherein the first refractive optical element comprises a prescription lens having positive optical power.

[0119] Example 3 is the method of Example 1 wherein the first refractive optical element comprises a prescription lens having negative optical power.

[0120] Example 4 is the method of Examples 1-3 wherein the second refractive optical element comprises a lens having negative optical power.

[0121] Example 5 is the method of Examples 1-4 wherein the reflective optical element comprises an off-axis mirror.

[0122] Example 6 is the method of Example 5 wherein the off-axis mirror is configured to reduce a perspective distortion of an image formed by the reflected light at the camera.

[0123] Example 7 is the method of Examples 1-6 wherein the light reflected from the eye of the user comprises illumination light generated using a plurality of illumination sources disposed on a side of the second refractive optical element opposite the user, and the method further comprising, prior to reflecting the light from the eye of the user, propagating the illumination light through the second refractive optical element; and propagating the illumination light through the first refractive optical element.

[0124] Example 8 is the method of Example 7 further comprising producing virtual content from an output region of an eyepiece; propagating the virtual content through a plane inwhich the plurality of illumination sources are disposed; and impinging the virtual content on the eye of the user.

[0125] Example 9 is an augmented reality (AR) system comprising a remote computing device including a processor, a memory, and a communication system coupled to the processor; and a wearable device including a frame, a communication device coupled to the frame, a projector coupled to the frame, a display optically coupled to the projector, a first refractive optical element coupled to the frame; and an eye tracking system comprising a second refractive optical element, a reflective optical element coupled to the first refractive optical element, and a camera disposed laterally adjacent to the second refractive optical element.

[0126] Example 10 is the AR system of Example 9 wherein the frame comprises a temporal arm, and wherein the camera is coupled to the temporal arm of the frame.

[0127] Example 11 is the AR system of Examples 9-10 wherein the camera is oriented to receive reflected light from the reflective optical element.

[0128] Example 12 is the AR system of Example 11 wherein the second refractive optical element comprises a lateral aperture, and the reflected light propagates through the second refractive optical element to the lateral aperture.

[0129] Example 13 is the AR system of Examples 9-12 wherein the reflective optical element comprises an off-axis mirror having a diffraction grating characterized by fourthorder polynomial phase parameters.

[0130] Example 14 is the AR system of Example 9-13 wherein the eye tracking system further comprises a plurality of illumination sources disposed on a world side of the second refractive optical element, wherein the plurality of illumination sources are configured to provide illumination light through the first refractive optical element and the second refractive optical element to an eye of a user.

[0131] Example 15 is a wearable device for projecting image light to an eye of a user, the wearable device comprising a frame; a set of projectors coupled to the frame; a set of displays coupled to the frame, wherein each of the set of displays is optically coupled to one of the set of projectors; a first set of refractive optical elements coupled to the frame; and an eye tracking system comprising a second set of refractive optical elements disposed on a worldside of the first set of refractive optical elements; a set of reflective optical elements, wherein each of the set of reflective optical elements is coupled to one of the second set of refractive optical elements; and a set of cameras, wherein each of the set of cameras is disposed laterally adjacent to one of the second set of refractive optical elements and oriented to receive reflected light from the reflective optical element coupled to the one of the second set of refractive optical elements.

[0132] Example 16 is the wearable device of Example 15 wherein the set of cameras comprises sensors characterized by a decenter parameter.

[0133] Example 17 is the wearable device of Example 16 wherein each of the set of reflective optical elements comprises an off-axis mirror having a diffraction grating characterized by first grating parameters, and wherein the decenter parameter and the first grating parameters are configured to reduce a perspective distortion in an image formed from the received reflected light at each of the sensors.

[0134] Example 18 is the wearable device of Examples 15-17 wherein the set of cameras comprises on-axis sensors.

[0135] Example 19 is the wearable device of Example 18 wherein each of the set of reflective optical elements comprises an off-axis mirror having a diffraction grating characterized by second grating parameters, and wherein the second grating parameters are configured to reduce a perspective distortion in an image formed from the received reflected light at each of the on-axis sensors.

[0136] Example 20 is an eye tracking system comprising a refractive optical element; a reflective optical element coupled to the refractive optical element; and a camera disposed laterally adjacent to the refractive optical element.

[0137] Example 21 is the eye tracking system of Example 20 wherein the reflective optical element is coupled to the refractive optical element at a lateral portion of a world side of the refractive optical element.

[0138] Example 22 is the eye tracking system of Examples 20-21 wherein the reflective optical element comprises an off-axis mirror having a diffraction grating.

[0139] Example 23 is the eye tracking system of Example 22 wherein the diffraction grating comprises a linear diffraction grating.

[0140] Example 24 is the eye tracking system of Example 22 wherein the diffraction grating is characterized by fourth-order polynomial phase parameters.

[0141] Example 25 is the eye tracking system of Examples 20-24 wherein the refractive optical element comprises a lens having negative optical power.

[0142] Example 26 is the eye tracking system of Examples 20-25 wherein the refractive optical element comprises a lateral aperture disposed at a lateral side of the refractive optical element between an eye side and a world side of the refractive optical element.

[0143] Example 27 is the eye tracking system of Example 26 wherein the camera disposed laterally adjacent to the refractive optical element is oriented to receive light reflected from the reflective optical element and propagated through the refractive optical element to the lateral aperture.

[0144] Example 28 is the eye tracking system of Examples 20-27 further comprising a plurality of illumination sources disposed on a world side of the optical element.

[0145] Example 29 is the eye tracking system of Example 28, wherein illumination light from the plurality of illumination sources is configured to propagate through the refractive optical element before reaching an eye of a user.

[0146] Example 30 is the eye tracking system of Examples 20-29, wherein the refractive optical element is a first refractive optical element, and further comprising a second refractive optical element disposed between an eye side of the first refractive optical element and an eye of a user.

[0147] Example 31 is an augmented reality (AR) system comprising a wearable device including a frame, a set of projectors coupled to the frame, a set of displays coupled to the frame, wherein each of the set of displays is optically coupled to one of the set of projectors; a set of eye tracking devices coupled to the frame and comprising a first set of refractive optical elements, a set of reflective optical elements, wherein each of the set of reflective optical elements is coupled to one of the first set of refractive optical elements, and a set of cameras; and a second set of refractive optical elements coupled to the frame; a memory; anda processor coupled to the memory, wherein the processor is configured to reflect light from eyes of a user; propagate the light through the first set of refractive optical elements; propagate the light through the second set of refractive optical element; impinge the light on the set of reflective optical elements; reflect light from the set of reflective optical elements; propagate the reflected light through the second set of refractive optical elements to a lateral aperture of each of the second set of refractive optical elements; detect the reflected light at the set of cameras; output eye tracking signals from the set of cameras; and track the eyes based on the eye tracking signals from the set of cameras.

[0148] Example 32 is the AR system of Example 31 wherein the first set of refractive optical elements comprises a prescription lens having positive optical power.

[0149] Example 33 is the AR system of Examples 31-32 wherein the first set of refractive optical elements comprises a prescription lens having negative optical power.

[0150] Example 34 is the AR system of Examples 31-33 wherein the second set of refractive optical elements comprises a lens having negative optical power.

[0151] Example 35 is the AR system of Examples 31-34 wherein the set of reflective optical elements comprises an off-axis mirror.

[0152] Example 36 is the AR system of Example 35 wherein the off-axis mirror is configured to reduce a perspective distortion of an image formed by the reflected light at the set of cameras.

[0153] Example 37 is the AR system of Examples 31-36 further comprising a plurality of illumination sources disposed on a side of the second set of refractive optical elements opposite the user, wherein the light reflected from the eye of the user comprises illumination light generated using the plurality of illumination sources, and wherein the processor is configured to further, prior to reflecting the light from the eye of the user, propagate the illumination light through the second set of refractive optical elements; and propagate the illumination light through the first set of refractive optical elements.

[0154] Example 38 is the AR system of Example 37 wherein the processor is configured to further produce virtual content from an output region of an eyepiece; propagate the virtual content through a plane in which the plurality of illumination sources are disposed; and impinge the virtual content on the eye of the user.

[0155] Example 39 is a non-transitory computer-readable medium comprising program code that is executable by a processor of a device that is wearable by a user, the program code being executable by the processor to reflect light from an eye of a user; propagate the light through a first refractive optical element; propagate the light through a second refractive optical element; impinge the light on a reflective optical element; reflect reflected light from the reflective optical element; propagate the reflected light through the refractive optical element to a lateral aperture of the second refractive optical element; detect the reflected light at a camera; output an eye tracking signal from the camera; and track the eye based on the eye tracking signal from the camera.

[0156] Example 40 is the non-transitory computer-readable medium of Example 39 wherein the first refractive optical element comprises a prescription lens having positive optical power.

[0157] Example 41 is the non-transitory computer-readable medium of Examples 39-40 wherein the first refractive optical element comprises a prescription lens having negative optical power.

[0158] Example 42 is the non-transitory computer-readable medium of Examples 39-41 wherein the second refractive optical element comprises a lens having negative optical power.

[0159] Example 43 is the non-transitory computer-readable medium of Examples 39-42 wherein the reflective optical element comprises an off-axis mirror.

[0160] Example 44 is the non-transitory computer-readable medium of Example 43 wherein the off-axis mirror is configured to reduce a perspective distortion of an image formed by the reflected light at the camera.

[0161] Example 45 is the non-transitory computer-readable medium of Examples 39-44 wherein the light reflected from the eye of the user comprises illumination light generated using a plurality of illumination sources disposed on a side of the second refractive optical element opposite the user, and further comprising program code executable by the processor to, prior to reflecting the light from the eye of the user, propagate the illumination light through the second refractive optical element; and propagate the illumination light through the first refractive optical element.

[0162] Example 46 is the non-transitory computer-readable medium of Example 45 further comprising program code executable by the processor to produce virtual content from an output region of an eyepiece; propagate the virtual content through a plane in which the plurality of illumination sources are disposed; and impinge the virtual content on the eye of the user.

[0163] In the foregoing specification, the disclosure has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.

[0164] Indeed, it will be appreciated that the systems and methods of the disclosure each have several innovative aspects, no single one of which is solely responsible or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure.

[0165] Certain features that are described in this specification in the context of separate embodiments also may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment also may be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination. No single feature or group of features is necessary or indispensable to each and every embodiment.

[0166] It will be appreciated that conditional language used herein, such as, among others, "can," "could," "might," "may," "e.g.," and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps. Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author inputor prompting, whether these features, elements and / or steps are included or are to be performed in any particular embodiment. The terms "comprising," "including," "having," and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the" as used in this application and the appended claims are to be construed to mean "one or more" or "at least one" unless specified otherwise. Similarly, while operations may be depicted in the drawings in a particular order, it is to be recognized that such operations need not be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Further, the drawings may schematically depict one or more example processes in the form of a flowchart. However, other operations that are not depicted may be incorporated in the example methods and processes that are schematically illustrated. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. Additionally, the operations may be rearranged or reordered in other embodiments. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0167] Accordingly, the claims are not intended to be limited to the embodiments shown herein but are to be accorded the widest scope consistent with this disclosure, the principles, and the novel features disclosed herein. Thus, it is also understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and scope of the appended claims.

Claims

WHAT IS CLAIMED IS:

1. A method for performing eye tracking, the method comprising: reflecting light from an eye of a user; propagating the light through a first refractive optical element; propagating the light through a second refractive optical element; impinging the light on a reflective optical element; reflecting the light from the reflective optical element; propagating the reflected light through the second refractive optical element to a lateral aperture of the second refractive optical element; detecting the reflected light at a camera; outputting an eye tracking signal from the camera; and tracking the eye based on the eye tracking signal from the camera.

2. The method of claim 1 wherein the first refractive optical element comprises a prescription lens having positive optical power.

3. The method of claim 1 wherein the first refractive optical element comprises a prescription lens having negative optical power.

4. The method of claim 1 wherein the second refractive optical element comprises a lens having negative optical power.

5. The method of claim 1 wherein the reflective optical element comprises an off-axis mirror.

6. The method of claim 5 wherein the off-axis mirror is configured to reduce a perspective distortion of an image formed by the reflected light at the camera.

7. The method of claim 1 wherein the light reflected from the eye of the user comprises illumination light generated using a plurality of illumination sources disposed on a side of the second refractive optical element opposite the user, and the method further comprising, prior to reflecting the light from the eye of the user: propagating the illumination light through the second refractive optical element; and propagating the illumination light through the first refractive optical element.

8. The method of claim 7 further comprising: producing virtual content from an output region of an eyepiece; propagating the virtual content through a plane in which the plurality of illumination sources are disposed; and impinging the virtual content on the eye of the user.

9. An augmented reality (AR) system comprising: a remote computing device including: a processor; a memory; and a communication system coupled to the processor; and a wearable device including: a frame; a communication device coupled to the frame; a projector coupled to the frame; a display optically coupled to the projector; a first refractive optical element coupled to the frame; and an eye tracking system comprising: a second refractive optical element; a reflective optical element coupled to the first refractive optical element; and a camera disposed laterally adjacent to the second refractive optical element.

10. The AR system of claim 9 wherein the frame comprises a temporal arm, and wherein the camera is coupled to the temporal arm of the frame.

11. The AR system of claim 9 wherein the camera is oriented to receive reflected light from the reflective optical element.

12. The AR system of claim 11 wherein the second refractive optical element comprises a lateral aperture, and the reflected light propagates through the second refractive optical element to the lateral aperture.

13. The AR system of claim 9 wherein the reflective optical element comprises an off-axis mirror having a diffraction grating characterized by fourth-order polynomial phase parameters.

14. The AR system of claim 9 wherein the eye tracking system further comprises a plurality of illumination sources disposed on a world side of the second refractive optical element, wherein the plurality of illumination sources are configured to provide illumination light through the first refractive optical element and the second refractive optical element to an eye of a user.

15. A wearable device for projecting image light to an eye of a user, the wearable device comprising: a frame; a set of projectors coupled to the frame; a set of displays coupled to the frame, wherein each of the set of displays is optically coupled to one of the set of projectors; a first set of refractive optical elements coupled to the frame; and an eye tracking system comprising: a second set of refractive optical elements disposed on a world side of the first set of refractive optical elements; a set of reflective optical elements, wherein each of the set of reflective optical elements is coupled to one of the second set of refractive optical elements; and a set of cameras, wherein each of the set of cameras is disposed laterally adjacent to one of the second set of refractive optical elements and oriented to receive reflected light from the reflective optical element coupled to the one of the second set of refractive optical elements.

16. The wearable device of claim 15 wherein the set of cameras comprises sensors characterized by a decenter parameter.

17. The wearable device of claim 16 wherein each of the set of reflective optical elements comprises an off-axis mirror having a diffraction grating characterized by first grating parameters, and wherein the decenter parameter and the first grating parameters are configured to reduce a perspective distortion in an image formed from the received reflected light at each of the sensors.

18. The wearable device of claim 15 wherein the set of cameras comprises on-axis sensors.

19. The wearable device of claim 18 wherein each of the set of reflective optical elements comprises an off-axis mirror having a diffraction grating characterized by second grating parameters, and wherein the second grating parameters are configured to reduce a perspective distortion in an image formed from the received reflected light at each of the on-axis sensors.

20. An eye tracking system comprising: a refractive optical element; a reflective optical element coupled to the refractive optical element; and a camera disposed laterally adjacent to the refractive optical element.

21. The eye tracking system of claim 20 wherein the reflective optical element is coupled to the refractive optical element at a lateral portion of a world side of the refractive optical element.

22. The eye tracking system of claim 20 wherein the reflective optical element comprises an off-axis mirror having a diffraction grating.

23. The eye tracking system of claim 22 wherein the diffraction grating comprises a linear diffraction grating.

24. The eye tracking system of claim 22 wherein the diffraction grating is characterized by fourth-order polynomial phase parameters.

25. The eye tracking system of claim 20 wherein the refractive optical element comprises a lens having negative optical power.

26. The eye tracking system of claim 20 wherein the refractive optical element comprises a lateral aperture disposed at a lateral side of the refractive optical element between an eye side and a world side of the refractive optical element.

27. The eye tracking system of claim 26 wherein the camera disposed laterally adjacent to the refractive optical element is oriented to receive light reflected fromthe reflective optical element and propagated through the refractive optical element to the lateral aperture.

28. The eye tracking system of claim 20 further comprising a plurality of illumination sources disposed on a world side of the optical element.

29. The eye tracking system of claim 28, wherein illumination light from the plurality of illumination sources is configured to propagate through the refractive optical element before reaching an eye of a user.

30. The eye tracking system of claim 20, wherein the refractive optical element is a first refractive optical element, and further comprising a second refractive optical element disposed between an eye side of the first refractive optical element and an eye of a user.

31. An augmented reality (AR) system comprising: a wearable device including: a frame; a set of projectors coupled to the frame; a set of displays coupled to the frame, wherein each of the set of displays is optically coupled to one of the set of projectors; a set of eye tracking devices coupled to the frame and comprising: a first set of refractive optical elements; a set of reflective optical elements, wherein each of the set of reflective optical elements is coupled to one of the first set of refractive optical elements; and a set of cameras; and a second set of refractive optical elements coupled to the frame; a memory; and a processor coupled to the memory, wherein the processor is configured to: reflect light from eyes of a user; propagate the light through the first set of refractive optical elements; propagate the light through the second set of refractive optical element; impinge the light on the set of reflective optical elements; reflect light from the set of reflective optical elements;propagate the reflected light through the second set of refractive optical elements to a lateral aperture of each of the second set of refractive optical elements; detect the reflected light at the set of cameras; output eye tracking signals from the set of cameras; and track the eyes based on the eye tracking signals from the set of cameras.

32. The AR system of claim 31 wherein the first set of refractive optical elements comprises a prescription lens having positive optical power.

33. The AR system of claim 31 wherein the first set of refractive optical elements comprises a prescription lens having negative optical power.

34. The AR system of claim 31 wherein the second set of refractive optical elements comprises a lens having negative optical power.

35. The AR system of claim 31 wherein the set of reflective optical elements comprises an off-axis mirror.

36. The AR system of claim 35 wherein the off-axis mirror is configured to reduce a perspective distortion of an image formed by the reflected light at the set of cameras.

37. The AR system of claim 31 further comprising a plurality of illumination sources disposed on a side of the second set of refractive optical elements opposite the user, wherein the light reflected from the eye of the user comprises illumination light generated using the plurality of illumination sources, and wherein the processor is configured to further: prior to reflecting the light from the eye of the user: propagate the illumination light through the second set of refractive optical elements; and propagate the illumination light through the first set of refractive optical elements.

38. The AR system of claim 37 wherein the processor is configured to further:produce virtual content from an output region of an eyepiece; propagate the virtual content through a plane in which the plurality of illumination sources are disposed; and impinge the virtual content on the eye of the user.

39. A non-transitory computer-readable medium comprising program code that is executable by a processor of a device that is wearable by a user, the program code being executable by the processor to: reflect light from an eye of a user; propagate the light through a first refractive optical element; propagate the light through a second refractive optical element; impinge the light on a reflective optical element; reflect reflected light from the reflective optical element; propagate the reflected light through the refractive optical element to a lateral aperture of the second refractive optical element; detect the reflected light at a camera; output an eye tracking signal from the camera; and track the eye based on the eye tracking signal from the camera.

40. The non-transitory computer-readable medium of claim 39 wherein the first refractive optical element comprises a prescription lens having positive optical power.

41. The non-transitory computer-readable medium of claim 39 wherein the first refractive optical element comprises a prescription lens having negative optical power.

42. The non-transitory computer-readable medium of claim 39 wherein the second refractive optical element comprises a lens having negative optical power.

43. The non-transitory computer-readable medium of claim 39 wherein the reflective optical element comprises an off-axis mirror.

44. The non-transitory computer-readable medium of claim 43 wherein the off-axis mirror is configured to reduce a perspective distortion of an image formed by the reflected light at the camera.

45. The non-transitory computer-readable medium of claim 39 wherein the light reflected from the eye of the user comprises illumination light generated using a plurality of illumination sources disposed on a side of the second refractive optical element opposite the user, and further comprising program code executable by the processor to, prior to reflecting the light from the eye of the user: propagate the illumination light through the second refractive optical element; and propagate the illumination light through the first refractive optical element.

46. The non-transitory computer-readable medium of claim 45 further comprising program code executable by the processor to: produce virtual content from an output region of an eyepiece; propagate the virtual content through a plane in which the plurality of illumination sources are disposed; and impinge the virtual content on the eye of the user.

Images