Method and system for eye tracking with reduced keystone distortion

Abstract

An eye tracking system operable to image an eye of a user includes an illumination source operable to illuminate the eye and an optical system including a first imaging surface, an aperture stop, and a second imaging surface. The optical system is operable to image an object plane corresponding to the eye. The eye tracking system also includes an image sensor having an image plane parallel to the first imaging surface, the second imaging surface, and the object plane.

Description

METHOD AND SYSTEM FOR EYE TRACKING WITH REDUCED KEYSTONE DISTORTIONBACKGROUND OF THE INVENTION

[0001] Modem computing and display technologies have facilitated the development of systems for so called "virtual reality" or "augmented reality" experiences, wherein digitally reproduced images or portions thereof are presented to a viewer in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or "VR," scenario typically involves presentation of digital or virtual image information without transparency to other actual real -world visual input; an augmented reality, or "AR," scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the viewer.

[0002] Referring to FIG. 1, an augmented reality scene 100 is depicted. The user of an AR technology sees a real -world park-like setting 106 featuring people, trees, buildings in the background, and a real-world concrete platform 120. The user also perceives that he / she "sees" "virtual content" such as a robot statue 110 standing upon the real -world concrete platform 120, and a flying cartoon-like avatar character 102 which seems to be a personification of a bumble bee. The robot statue 110 and the flying cartoon-like avatar character 102 are "virtual" in that they do not exist in the real world. Because the human visual perception system is complex, it is challenging to produce AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements.

[0003] Despite the progress made in these display technologies, there is a need in the art for improved methods and systems related to augmented reality systems, particularly, display systems.SUMMARY OF THE INVENTION

[0004] The present invention relates generally to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the presentinvention provide methods and systems useful for tracking eye position and movement in AR systems. Merely by way of example, a low complexity, off-axis imaging system is provided that performs AR eye tracking with reduced distortion, including keystone distortion, while keeping the image in focus. The invention is applicable to a variety of applications in computer vision and image display systems.

[0005] As described more fully herein, embodiments of the present invention utilize an off- center image produced using an imaging system that is oriented parallel to the object plane in contrast with systems using a tilted camera. The imaging system employs diffractive, e.g., holographic optics or metasurface-based optics, enabling wafer level fabrication and low complexity optical systems.

[0006] Numerous benefits are achieved by way of the present invention over conventional techniques. For example, embodiments of the present invention provide methods and systems that reduce keystone distortion in augmented reality systems. Moreover, in some embodiments, an off-axis imaging system is utilized in an eye tracking system with reduced optical distortion. These and other embodiments of the invention along with many of its advantages and features are described in more detail in conjunction with the text below and attached figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 illustrates a user's view of augmented reality (AR) through an AR device.

[0008] FIG. 2A illustrates a cross-sectional, side view of an example of a set of stacked waveguides that each includes an incoupling optical element.

[0009] FIG. 2B illustrates a perspective view of an example of the one or more stacked waveguides of FIG. 2A.

[0010] FIG. 2C illustrates a top-down, plan view of an example of the one or more stacked waveguides of FIGS. 2 A and 2B.

[0011] FIG. 3 is a simplified illustration of an eyepiece waveguide having a combined pupil expander according to an embodiment of the present invention.

[0012] FIG. 4 illustrates an example of wearable display system according to an embodiment of the present invention.

[0013] FIG. 5 shows a perspective view of a wearable device according to an embodiment of the present invention.

[0014] FIG. 6 is an exploded perspective view of elements of an eye tracking system for an augmented reality display according to an embodiment of the present invention.

[0015] FIG. 7 is a simplified plan view of elements of an AR headset including eye tracking according to an embodiment of the present invention.

[0016] FIG. 6 is an exploded perspective view of elements of an eye tracking system for an augmented reality display according to an embodiment of the present invention.

[0017] FIG. 7 is a simplified plan view of elements of an AR headset including eye tracking according to an embodiment of the present invention.

[0018] FIG. 8 A is a simplified schematic diagram illustrating an object plane and an imaging system.

[0019] FIG. 8B is a plot illustrating keystone distortion exhibited by the imaging system shown in FIG. 8A.

[0020] FIG. 9A is a simplified schematic diagram illustrating a parallel orientation object plane and imaging system according to an embodiment of the present invention.

[0021] FIG. 9B is a schematic diagram illustrating the imaging system shown in FIG. 9A according to an embodiment of the present invention.

[0022] FIG. 9C is a plot illustrating keystone distortion exhibited by the imaging system shown in FIG. 9B.

[0023] FIG. 9D is a plot showing the optical phase function for the first imaging surface illustrated in FIG. 9B according to an embodiment of the present invention.

[0024] FIG. 9E is a plot showing the optical phase function for the first imaging surface for the illuminated portion of the phase profile according to an embodiment of the present invention.

[0025] FIG. 9F is a plot showing the optical phase function for the second imaging surface illustrated in FIG. 9B according to an embodiment of the present invention.

[0026] FIG. 9G is a plot showing the optical phase function for the second imaging surface for the illuminated portion of the phase profile according to an embodiment of the present invention.

[0027] FIG. 10 is a simplified schematic diagram illustrating a folded imaging system according to an embodiment of the present invention.

[0028] FIG. 11 is a simplified schematic diagram illustrating an imaging system according to an alternative embodiment of the present invention.

[0029] FIG. 12 is a simplified schematic diagram illustrating a multi-element imaging system according to an embodiment of the present invention.

[0030] FIG. 13 is a simplified schematic diagram illustrating a multi-element imaging system according to an alternative embodiment of the present invention.

[0031] FIG. 14 is a simplified schematic diagram illustrating a multi-element imaging system according to yet another alternative embodiment of the present invention.

[0032] FIG. 15 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention.DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

[0033] The present invention relates generally to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the present invention provide methods and systems useful for tracking eye position and movement in AR systems. Merely by way of example, a low complexity, off-axis imaging system is provided that performs AR eye tracking with reduced distortion, including keystone distortion, while keeping the image in focus. The invention is applicable to a variety of applications in computer vision and image display systems.

[0034] Reference will now be made to the drawings, in which like reference numerals refer to like parts throughout. Unless indicated otherwise, the drawings are schematic not necessarily drawn to scale.

[0035] With reference now to FIG. 2A, in some embodiments, light impinging on a waveguide may need to be redirected to incouple that light into the waveguide. An incoupling optical element may be used to redirect and incouple the light into itscorresponding waveguide. Although referred to as "incoupling optical element" through the specification, the incoupling optical element need not be an optical element and may be a non-optical element. FIG. 2A illustrates a cross-sectional, side view of an example of a set of stacked waveguides 200 that each includes an incoupling optical element. The waveguides may each be configured to output light of one or more different wavelengths, or one or more different ranges of wavelengths. Light from a projector is injected into the set of stacked waveguides 200 and outcoupled to a user as described more fully below.

[0036] The illustrated set of stacked waveguides 200 includes waveguide 202, waveguide204, and waveguide 206. Each waveguide includes an associated incoupling optical element (which may also be referred to as a light input area on the waveguide), with, e.g., incoupling optical element 203 disposed on a major surface (e.g., an upper major surface) of waveguide 202, incoupling optical element 205 disposed on a major surface (e.g., an upper major surface) of waveguide 204, and incoupling optical element 207 disposed on a major surface (e.g., an upper major surface) of waveguide 206. In some embodiments, one or more of the incoupling optical elements may be disposed on the bottom major surface of the respective waveguide (particularly where the one or more incoupling optical elements are reflective, deflecting optical elements). As illustrated, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be disposed on the upper major surface of waveguide 202, waveguide 204, and waveguide 206, respectively (or the top of the next lower waveguide), particularly where those incoupling optical elements are transmissive, deflecting optical elements. In some embodiments, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be disposed in the body of the waveguide 202, waveguide 204, and waveguide 206, respectively. In some embodiments, as discussed herein, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 are wavelength- selective, such that they selectively redirect one or more wavelengths of light, while transmitting other wavelengths of light. While illustrated on one side or comer of waveguide 202, waveguide 204, and waveguide 206, respectively, it will be appreciated that the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be disposed in other areas of waveguide 202, waveguide 204, and waveguide 206, respectively, in some embodiments.

[0037] As illustrated, the incoupling optical element 203, the incoupling optical element205, and the incoupling optical element 207 may be laterally offset from one another. Insome embodiments, each incoupling optical element may be offset such that it receives light without that light passing through another incoupling optical element. For example, each of the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be configured to receive light from a different projector and may be separated (e.g., laterally spaced apart) from other incoupling optical elements such that it substantially does not receive light from the other ones of the incoupling optical elements.

[0038] Each waveguide also includes associated light distributing elements, with, e.g., light distributing elements 210 disposed on a major surface (e.g., a top major surface) of waveguide 202, light distributing elements 212 disposed on a major surface (e.g., a top major surface) of waveguide 204, and light distributing elements 214 disposed on a major surface (e.g., a top major surface) of waveguide 206. In some other embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be disposed on a bottom major surface of associated waveguide 202, waveguide 204, and waveguide 206, respectively. In some other embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be disposed on both top and bottom major surfaces of associated waveguide 202, waveguide 204, and waveguide 206, respectively; or the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be disposed on different ones of the top and bottom major surfaces in different associated waveguide 202, waveguide 204, and waveguide 206, respectively.

[0039] Waveguide 202, waveguide 204, and waveguide 206 may be spaced apart and separated by, e.g., gas, liquid, and / or solid layers of material. For example, as illustrated in FIG. 2A, layer 208 may separate waveguide 202 and waveguide 204 and layer 209 may separate waveguide 204 and waveguide 206. In some embodiments, layer 208 and layer 209 are formed of low refractive index materials (that is, materials having a lower refractive index than the material forming the immediately adjacent one of waveguide 202, waveguide 204, or waveguide 206). Preferably, the refractive index of the material forming layer 208 and / or layer 209 is 0.05 or more, or 0.10 or less than the refractive index of the material forming the waveguide 202, the waveguide 204, or the waveguide 206. Advantageously, layer 208 and layer 209 having the lower refractive index may function as cladding layers that facilitate total internal reflection (TIR) of light through the waveguide 202, the waveguide 204, and the waveguide 206 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layer 208 and the layer 209 are formed of air. While not illustrated,it will be appreciated that the top and bottom of the illustrated set of stacked waveguides 200 may include immediately neighboring cladding layers.

[0040] Preferably, for ease of manufacturing and other considerations, the material forming the waveguide 202, the waveguide 204, and the waveguide 206 are similar or the same, and the material forming the layer 208 and the layer 209 are similar or the same. In some embodiments, the material forming the waveguide 202, the waveguide 204, and the waveguide 206 may be different between one or more waveguides, and / or the material forming the layer 208 and the layer 209 may be different, while still holding to the various refractive index relationships noted above.

[0041] With continued reference to FIG. 2A, light ray 218, light ray 219, and light ray 220 are incident on the set of stacked waveguides 200. It will be appreciated that the light ray 218, the light ray 219, and the light ray 220 may be injected into the waveguide 202, the waveguide 204, and the waveguide 206 by one or more projectors (not shown).

[0042] In some embodiments, light ray 218, the light ray 219, and the light ray 220 have different properties, e.g., different wavelengths or different ranges of wavelengths, which may correspond to different colors. The incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 each deflect the incident light such that the light propagates through a respective one of the waveguide 202, the waveguide 204, or the waveguide 206 by TIR. In some embodiments, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 each selectively deflect one or more particular wavelengths of light, while transmitting other wavelengths to an underlying waveguide and associated incoupling optical element.

[0043] For example, incoupling optical element 203 may be configured to deflect light ray 218, which has a first wavelength or range of wavelengths, while transmitting light ray 219 and light ray 220, which have different second and third wavelengths or ranges of wavelengths, respectively. The light ray 219 transmitted through the waveguide 202 impinges on and is deflected by the incoupling optical element 205, which is configured to deflect light of a second wavelength or range of wavelengths. The light ray 220 is deflected by the incoupling optical element 207, which is configured to selectively deflect light of third wavelength or range of wavelengths.

[0044] With continued reference to FIG. 2 A, the light ray 218, the light ray 219, and the light ray 220 are deflected such that they propagate through corresponding waveguide 202,waveguide 204, and waveguide 206, respectively; that is, the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 of each waveguide deflects light into the corresponding waveguide 202, waveguide 204, or waveguide 206 to incouple light into that corresponding waveguide. The light ray 218, the light ray 219, and the light ray 220 are deflected at angles that cause the light to propagate through the respective waveguide 202, waveguide 204, and waveguide 206 by TIR. The light ray 218, the light ray 219, and the light ray 220 propagate through the respective waveguide202, waveguide 204, and waveguide 206 by TIR until impinging on the waveguide's corresponding light distributing elements: the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214, where they are outcoupled to provide out-coupled light rays 216.

[0045] With reference now to FIG. 2B, a perspective view of an example of the set of stacked waveguides 200 of FIG. 2A is illustrated. As noted above, the light ray 218, the light ray 219, and the light ray 220 are incoupled and deflected by the incoupling optical element203, the incoupling optical element 205, and the incoupling optical element 207, respectively, and then propagate by TIR within the waveguide 202, the waveguide 204, and the waveguide 206, respectively. The light ray 218, the light ray 219, and the light ray 220 then impinge on the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214, respectively. The light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 deflect the light ray 218, the light ray 219, and the light ray 220 so that they propagate towards the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226, respectively.

[0046] In some embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or distribute light to the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 and, in some embodiments, may also increase the beam or spot size of this light as it propagates to the outcoupling optical elements. In some embodiments, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be omitted and the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207 may be configured to deflect light directly to the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling opticalelements 226. For example, with reference to FIG. 2A, the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 may be replaced with the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226, respectively. In some embodiments, the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light to the eye of the user. It will be appreciated that the OPEs may be configured to increase the dimensions of the eye box in at least one axis and the EPEs may be configured to increase the eye box in an axis crossing, e.g., orthogonal to, the axis of the OPEs. For example, each OPE may be configured to redirect a portion of the light striking the OPE to an EPE of the same waveguide, while allowing the remaining portion of the light to continue to propagate down the waveguide. Upon impinging on the OPE again, another portion of the remaining light is redirected to the EPE, and the remaining portion of that portion continues to propagate further down the waveguide, and so on. Similarly, upon striking the EPE, a portion of the impinging light is directed out of the waveguide towards the user, and a remaining portion of that light continues to propagate through the waveguide until it strikes the EPE again, at which time another portion of the impinging light is directed out of the waveguide, and so on. Consequently, a single beam of incoupled light may be "replicated" each time a portion of that light is redirected by an OPE or EPE, thereby forming a field of cloned beams of light. In some embodiments, the OPE and / or EPE may be configured to modify a size of the beams of light. In some embodiments, the functionality of the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 and the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 are combined in a combined pupil expander as discussed in relation to FIG. 2E.

[0047] Accordingly, with reference to FIGS. 2 A and 2B, in some embodiments, the set of stacked waveguides 200 includes the waveguide 202, the waveguide 204, and the waveguide 206; the incoupling optical element 203, the incoupling optical element 205, and the incoupling optical element 207; the light distributing elements 210, the light distributing elements 212, and the light distributing elements 214 (e.g., OPEs); and the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226 (e.g., EPs) for each component color. The waveguide 202, the waveguide 204, and the waveguide 206 may be stacked with an air gap / cladding layer between each one. The incoupling optical element 203, the incoupling optical element 205, and the incouplingoptical element 207 redirect or deflect incident light (with different incoupling optical elements receiving light of different wavelengths) into its waveguide. The light then propagates at an angle which will result in TIR within the waveguide 202, the waveguide 204, and the waveguide 206, respectively. In the example shown, light ray 218 (e.g., blue light) is deflected by the incoupling optical element 203, and then continues to bounce down the waveguide, interacting with the light distributing element 210 (e.g., OPEs) and then the outcoupling optical element 222 (e.g., EPs), in a manner described earlier. The light ray 219 and the light ray 220 (e.g., green and red light, respectively) will pass through the waveguide202, with light ray 219 impinging on and being deflected by incoupling optical element 205. The light ray 219 then bounces down the waveguide 204 via TIR, proceeding on to its light distributing element 212 (e.g., OPEs) and then the outcoupling optical element 224 (e.g., EPs). Finally, light ray 220 (e.g., red light) passes through the waveguide 206 to impinge on the incoupling optical element 207 of the waveguide 206. The incoupling optical element 207 deflects the light ray 220 such that the light ray propagates to light distributing element 214 (e.g., OPEs) by TIR, and then to the outcoupling optical element 226 (e.g., EPs) by TIR. The outcoupling optical element 226 then finally out-couples the light ray 220 to the viewer, who also receives the outcoupled light from the other waveguides: the waveguide 202 and the waveguide 204.

[0048] FIG. 2C illustrates a top-down, plan view of an example of the set of stacked waveguides 200 of FIGS. 2A and 2B. As illustrated, the waveguide 202, the waveguide 204, and the waveguide 206, along with each waveguide's associated light distributing element: the light distributing element 210, light distributing element 212, and light distributing element 214 and the associated outcoupling optical elements: the outcoupling optical elements 222, the outcoupling optical elements 224, and the outcoupling optical elements 226, may be vertically aligned. However, as discussed herein, the incoupling optical element203, the incoupling optical element 205, and the incoupling optical element 207 are not vertically aligned; rather, the incoupling optical elements are preferably nonoverlapping (e.g., laterally spaced apart as seen in the top-down or plan view). As discussed further herein, this nonoverlapping spatial arrangement facilitates the injection of light from different resources into different waveguides on a one-to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, arrangements including nonoverlapping spatially separated incoupling optical elements may be referred to as a shiftedpupil system, and the incoupling optical elements within these arrangements may correspond to sub pupils.

[0049] FIG. 3 is a simplified illustration of an eyepiece waveguide having a combined pupil expander according to an embodiment of the present invention. In the example illustrated in FIG. 3, the eyepiece 310 utilizes a combined OPEZEPE region in a single-side configuration. Referring to FIG. 3, the eyepiece 310 includes a substrate 320 in which incoupling optical element 322 and a combined OPEZEPE region 324, also referred to as a combined pupil expander (CPE), are provided. Incident light ray 330 is incoupled via the incoupling optical element 322 and outcoupled as output light rays 332 via the combined OPEZEPE region 324.

[0050] The combined OPEZEPE region 324 includes gratings corresponding to both an OPE and an EPE that spatially overlap in the x-direction and the y-direction. In some embodiments, the gratings corresponding to both the OPE and the EPE are located on the same side of a substrate 320 such that either the OPE gratings are superimposed onto the EPE gratings or the EPE gratings are superimposed onto the OPE gratings (or both). In other embodiments, the OPE gratings are located on the opposite side of the substrate 320 from the EPE gratings such that the gratings spatially overlap in the x-direction and the y-direction but are separated from each other in the z-direction (i.e., in different planes). Thus, the combined OPEZEPE region 324 can be implemented in either a single-sided configuration or in a two- sided configuration.

[0051] FIG. 4 illustrates an example of wearable display system 430 into which the various waveguides and related systems disclosed herein may be integrated. With reference to FIG.4, the wearable display system 430 includes a display 432, and various mechanical and electronic modules and systems to support the functioning of the display 432. The display 432 may be coupled to a frame 434, which is wearable by a user 440 (also referred to as a viewer or a display system user) and which is configured to position the display 432 in front of the eyes of the user 440. The display 432 may be considered eyewear in some embodiments. In some embodiments, a speaker 436 is coupled to the frame 434 and configured to be positioned adjacent the ear canal of the user 440 (in some embodiments, another speaker, not shown, may optionally be positioned adjacent the other ear canal of the user to provide stereo / shapeable sound control). The wearable display system 430 may also include one or more microphones or other devices to detect sound. In some embodiments,the microphone is configured to allow the user to provide inputs or commands to the wearable display system 430 (e.g., the selection of voice menu commands, natural language questions, etc.), and / or may allow audio communication with other persons (e.g., with other users of similar display systems). The microphone may further be configured as a peripheral sensor to collect audio data (e.g., sounds from the user and / or environment). In some embodiments, the wearable display system 430 may further include one or more outwardly directed environmental sensors configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user. For example, environmental sensors may include one or more cameras, which may be located, for example, facing outward so as to capture images similar to at least a portion of an ordinary field of view of the user 440. In some embodiments, the wearable display system may also include a peripheral sensor, which may be separate from the frame 434 and attached to the body of the user 440 (e.g., on the head, torso, an extremity, etc. of the user 440). The peripheral sensor may be configured to acquire data characterizing a physiological state of the user 440 in some embodiments. For example, the sensor may be an electrode.

[0052] The display 432 is operatively coupled by a communications link, such as by a wired lead or wireless connectivity, to a local data processing module which may be mounted in a variety of configurations, such as fixedly attached to the frame 434, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 440 (e.g., in a backpack-style configuration, in a belt-coupling style configuration). Similarly, the sensor may be operatively coupled by a communications link, e.g., a wired lead or wireless connectivity, to the local processor and data module. The local processing and data module may comprise a hardware processor, as well as digital memory, such as nonvolatile memory (e.g., flash memory or hard disk drives), both of which may be utilized to assist in the processing, caching, and storage of data. Optionally, the local processor and data module may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on. The data may include data a) captured from sensors (which may be, e.g., operatively coupled to the frame 434 or otherwise attached to the user 440), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, gyros, and / or other sensors disclosed herein; and / or b) acquired and / or processed using remote processing module 452 and / or remote data repository 454 (including data relating to virtual content), possibly for passage to the display 432 after such processing or retrieval. The localprocessing and data module may be operatively coupled by communication links 438 such as via wired or wireless communication links, to the remote processing and data module 450, which can include the remote processing module 452, the remote data repository 454, and a battery 460. The remote processing module 452 and the remote data repository 454 can be coupled by communication links 456 and communication links 458 to remote processing and data module 450 such that these remote modules are operatively coupled to each other and available as resources to the remote processing and data module 450. In some embodiments, the remote processing and data module 450 may include one or more of the image capture devices, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyros. In some other embodiments, one or more of these sensors may be attached to the frame 434, or may be standalone structures that communicate with the remote processing and data module 450 by wired or wireless communication pathways.

[0053] With continued reference to FIG. 4, in some embodiments, the remote processing and data module 450 may comprise one or more processors configured to analyze and process data and / or image information, for instance including one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, and so on. In some embodiments, the remote data repository 454 may comprise a digital data storage facility, which may be available through the internet or other networking configuration in a "cloud" resource configuration. In some embodiments, the remote data repository 454 may include one or more remote servers, which provide information, e.g., information for generating augmented reality content, to the local processing and data module and / or the remote processing and data module 450. In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing fully autonomous use from a remote module. Optionally, an outside system (e.g., a system of one or more processors, one or more computers) that includes CPUs, GPUs, and so on, may perform at least a portion of processing (e.g., generating image information, processing data) and provide information to, and receive information from, the illustrated modules, for instance, via wireless or wired connections.

[0054] FIG. 5 shows a perspective view of a wearable device 500 according to an embodiment of the present invention. Wearable device 500 includes a frame 502 configured to support one or more projectors 504 at various positions along an interior-facing surface of frame 502, as illustrated. In some embodiments, projectors 504 can be attached at positions near temples 506. Alternatively, or in addition, another projector could be placed in position508. Such projectors may, for instance, include or operate in conjunction with one or more liquid crystal on silicon (LCoS) modules, micro-LED displays, or fiber scanning devices. In some embodiments, light from projectors 504 or projectors disposed in position 508 could be guided into eyepieces 510 for display to eyes of a user. Projectors placed at positions 512 can be somewhat smaller on account of the close proximity this gives the projectors to the waveguide system. The closer proximity can reduce the amount of light lost as the waveguide system guides light from the projectors to eyepiece 510. In some embodiments, the projectors at positions 512 can be utilized in conjunction with projectors 504 or projectors disposed in position 508. While not depicted, in some embodiments, projectors could also be located at positions beneath eyepieces 510. Wearable device 500 is also depicted including sensors 514 and sensors 516. Sensors 514 and sensors 516 can take the form of forwardfacing and lateral-facing optical sensors configured to characterize the real-world environment surrounding wearable device 500.

[0055] Embodiments of the present invention utilize an eye tracking system to determine the eye gaze location of the user and utilize the eye gaze location for image compression processes. Referring to FIG. 5, eye tracking cameras 505 are located on the frame 502 and can be utilized to track the eye gaze location of the user using the wearable device 500. In other embodiments, other eye tracking systems are utilized to determine the eye gaze location and the eye tracking cameras 505 illustrated in FIG. 5 are merely exemplary. As described more fully herein, the image compression processes utilized to compress and decompress virtual content for storage in memory, internal communications, and display, among other functions, can be modified depending on the eye gaze location, for example, portions of an image or video stream corresponding to the eye gaze location can be compressed using a higher quality compression process compared to other portions of the image or video stream that are located more distant from the eye gaze location. Since these more distant portions of the image or video stream are in the user's peripheral vision, any impact on the user experience resulting from the reduction in compression quality can be less than the benefits achieved in terms of memory and processing efficiency and / or requirements. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0056] FIG. 6 is an exploded perspective view of elements of an eye tracking system for an augmented reality display according to an embodiment of the present invention. Viewable optics assembly 600 includes a rear portion 610, a merged illumination and refractive layer (MILR) structure 620, and a camera flex cable 630. The rear portion 610 may include a rearcarrier 612 and define a virtual content output region 614 of an eyepiece. In some embodiments, the eyepiece may be the eyepiece 310 as illustrated in FIG. 3. In some embodiments, spacers or spacing components (not shown) may be disposed between the rear carrier 612 and the eyepiece (not shown). The spacers or spacing components may space a front Extended Depth Of Field (EDOF) lens a predetermined distance from the rear carrier 612. The rear carrier 612 may be any material as deemed appropriate. In some embodiments, the rear carrier is magnesium.

[0057] As illustrated in FIG. 6, the MILR structure 620 may have a combined structure that includes an optical element 622, a substrate 624, an illumination structure 626, and a passivation coating 628 applied on the illumination structure 626 along an axis 601. The geometry illustrated in FIG. 6 includes a lateral plane defined by the x-axis and the y-axis and a longitudinal direction aligned with the z-axis. As illustrated in FIG. 6, and discussed more fully below, each of the set of cameras 632 are laterally offset from axis 601, i.e., positioned at non-zero positions along the x-axis and / or the y-axis. The optical element 622 (e.g., the rear EDOF lens) may include a front (e.g., a world side) planar surface and the substrate 624 can be bonded to the front planar surface of the optical element 622. In some embodiments, the substrate 624 may be a polyethylene terephthalate (PET) film. The illumination structure 626 may include eye tracking illumination sources (e.g., infrared LEDs), metal traces and a flex cable surrounding the illumination structure 626 and in electrical communications with the eye tracking illumination sources. The MILR structure 620 implemented in this laminated form thus includes the illumination structure 626, which includes the eye tracking illumination sources, as well as the optical element 622. Light emitted from the eye tracking illumination sources in the illumination structure 626 passes through optical element 622 before reflecting from the eye of the user.

[0058] The optical element 622 may have an optical power and can also be referred to as an EDOF refractive element since it moves the virtual content plane a predetermined distance away from the user's eye, thereby extending the depth of field. In some embodiments, optical element 622 may move the virtual content by a distance on the order of tens of centimeters. In the embodiment illustrated in FIG. 6, the optical element 622 has negative optical power, i.e., it is a negative lens that diverges collimated light received from the eyepiece. Although not shown in FIG. 6, a corresponding front EDOF lens with the opposite optical power (e.g., positive optical power) may be positioned on the world side of the eyepiece in order to counteract the action of optical element 622 with respect to world light.

[0059] The illumination structure 626 may be fabricated by a lamination process. In the lamination process, the substrate 624 (e.g., a PET film substrate) is provided and metal traces (e.g., copper traces) and contact pads may be deposited on the substrate 624. The eye tracking illumination sources (e.g., infrared LEDs) may be mounted and electrically connected to the contact pads. Subsequently, the passivation coating 628 may be applied upon the illumination structure 626. In some embodiments, the passivation coating 628 may be applied by a roll-to-roll process. The substrate 624, the illumination structure 626, and the passivation coating 628 may form a combined structure. The combined structure may then be bonded to the front (e.g., a world side) planar surface of the optical element 622 (e.g., the rear EDOF lens). Accordingly, the eye tracking illumination sources are positioned on the world side of the optical element 622 (e.g., rear EDOF lens) opposite the user's eye and propagate through the rear EDOF lens before reflection from the user's eye.

[0060] Although in FIG. 6 the eye tracking illumination sources are illustrated as infrared (IR) LEDs that are laminated to the substrate 624 (e.g., PET film substrate) and emit light toward the eye side, the MILR structure 620 may have other configurations as desired. In other embodiments, the eye tracking illumination sources may be disposed at locations peripheral to the metal traces of the illumination structure 626, and a waveguiding layer may be utilized in the plane in which the metal traces are disposed. In these embodiments, light from the eye tracking illumination sources propagates in the plane of the waveguiding layer and is outcoupled to provide the eye tracking illumination. The light can be outcoupled using reflective structures, for example, a mirrored surface tilted at -45° to the plane of the waveguiding layer, or diffractive structures, for example, vertical outcoupling gratings disposed in or on the plane of the waveguiding layer. Thus, the IR LEDs illustrated in FIG. 6 can be replaced with illumination regions in which eye tracking illumination is output and directed toward the user's eye. One of ordinary skill in the art would recognize many variations, modifications, and alternatives. Additional information related to waveguiding layers is provided in U.S. Patent No. 11,106,033 and International Patent Application No. PCT / US22 / 71988, the disclosures of which are hereby incorporated by reference in their entirety for all purposes.

[0061] In some embodiments, a set of cameras 632 may be disposed on the camera flex cable 630 to detect illumination light reflected from the eye. The camera flex cable 630 and the set of cameras 632 may be disposed on the eye side of optical element 622. In some embodiments, the set of cameras 632 may be incorporated into a wearable device similar tothe manner in which the eye tracking cameras 505 are incorporated into the wearable device 500 illustrated in FIG. 5. As illustrated in FIG. 6, in some systems, the set of cameras 632 used to perform eye tracking are positioned below the center of the eye at positions that are laterally offset from the optical axis of the optical system. Moreover, the cameras in the set of camera 632 can be oriented so that the image plane of the camera is oriented perpendicular to axis 601, resulting in the object plane, which corresponds to the location of the eye and the image plane being parallel to each other. As described more fully in relation to FIGS. 8 A and 8B, this offset positioning of the eye tracking cameras can result in distortion that is reduced by embodiments of the present invention.

[0062] During operation of an AR device, the set of cameras 632, which may be referred to as off-axis cameras, may detect information (e.g., capture images) leading to the estimation of a gaze vector corresponding to the eye of the user. The gaze vector may be computed for each image frame and may, in various embodiments, be expressed as a two-dimensional (2D) or three-dimensional (3D) value. For example, a gaze vector may be expressed using a spherical coordinate system by a polar angle 9 and an azimuthal angle cp. Alternatively, or additionally, a gaze vector may be expressed using a 3D Cartesian coordinate system by X, Y, and Z values. The gaze vector may intersect with an eyepiece waveguide of the AR device at an intersection point that may be calculated based on the location of the eye of the user, the location of the eyepiece waveguide, and the gaze vector. In some instances, the projector of the AR device may adjust the virtual image to improve image brightness and / or clarity around the intersection point in favor of other areas of the field of view. Additional description related to the gaze vector is provided in U.S. Patent No. 11,775,058, the disclosure of which is hereby incorporated by reference in its entirety for all purposes.

[0063] FIG. 7 is a simplified plan view of elements of an AR headset including eye tracking, according to an embodiment of the present invention. In FIG. 7, elements of a Visible Optics Assembly (VOA) are illustrated. The VOA 700 includes a front EDOF refractive element 710 and front optics 712 that receive world light propagating from left to right toward the eye 770 of the user. The VOA 700 also includes a dimmer 702, an eyepiece 704, rear EDOF and eye tracking (ET) structure 706, and optional prescription lens insert 760. Dimmer 702 includes a world side linear polarizer 720, a first quarter waveplate 722, a liquid crystal panel 724, a second quarter waveplate 726, and an eye-side linear polarizer 728, for example, a hard coated linear polarizer (HC-LP) with a surface open to air hard coated for handling purposes. Eyepiece 704 includes three eyepiece waveguide layers: blue active layer730, green active layer 732, and red active layer 734. Although a three-layer eyepiece (i.e., an eyepiece including three eyepiece waveguide layers) is illustrated in FIG. 7, this is not required, and in other embodiments, a six-layer eyepiece structure can be utilized with, for example, two layers corresponding to two depth planes or one layer.

[0064] Rear EDOF and ET structure 706 includes rear EDOF 752, illumination layer 750 and imaging layer 754 utilized in eye tracking. Eye tracking cameras 505 illustrated in FIG. 5 can be utilized in the imaging layer 754 to capture images of the eye 770. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0065] Embodiments of the present invention provide imaging systems that are able to form an image of a close distance object that is off-center / tilted with respect to the optics. Embodiments of the present invention are applicable and useful for eye tracking in AR systems since the eye tracking cameras cannot generally be placed in the area that is used by the user to see the display and real world. Thus, to achieve an unobstructed view, the eye tracking system is placed in some implementations outside the user's field of view (FOV), which results in the imaging camera(s) viewing the eye from side using a tilted camera configuration.

[0066] In some systems, a camera / optic can be arranged in a tilted configuration with respect to the object so that the image center aligns with the image sensor center and utilize the Scheimpflug principle to produce a sharp image over the full FOV. However, these systems are characterized by potentially high levels of keystone distortion and complicated manufacturing due to the tilting of the image sensor with respect to the optics, which are arranged at an angle to the object being imaged, namely the user's eye. Because of keystone distortion, eye tracking is adversely impacted since the resolution changes across the image. To address these shortcomings in conventional systems, embodiments of the provided imaging systems that are characterized by very low distortion or no distortion and substantially constant resolution across the FOV.

[0067] FIG. 8 A is a simplified schematic diagram illustrating an object plane and an imaging system. In FIG. 8A, an imaging system 800 following the Scheimpflug principle is illustrated. Illumination source 810 (e.g., one or more infrared light emitting diodes) produces illumination light rays 812, which impinge on the object plane 805. Light reflected from the object plane is illustrated by reflected light rays 815, which are collected by the optics 820 and imaged using the image sensor 822. The object plane 805 (i.e., the plane inwhich the user eye is positioned) is tilted with respect to the optics 820 and the image sensor 822 is also tilted with respect to the optics 820 to maintain image sharpness across the FOV.

[0068] Referring to FIG. 6, the imaging system illustrated in FIG. 8A corresponds to imaging of the user's eye with the set of cameras 632, which are positioned below axis 601 and oriented at an angle that is tilted with respect to the optical elements, for example, optical element 622, and axis 601. If the portion of the eye being imaged is lying in object plane 805, the lower portions of the object plane 805 are positioned closer to the image sensor 822, e.g., the set of cameras 632, than the upper portions of the object plane 805 that are positioned farther from the image sensor 822, e.g., the set of cameras 632. As illustrated in FIGS. 6 and 8 A, the set of cameras 632 / the image sensor 822 is tilted with respect to object plane 805 such that the optical axis of the set of cameras 632 / the image sensor 822 is not perpendicular to the object plane 805.

[0069] FIG. 8B is a plot illustrating keystone distortion exhibited by the imaging system shown in FIG. 8A. Using the imaging system 800 illustrated in FIG. 8A, in which portions of the object plane, e.g., the eye being imaged, are closer to the image sensor 822, e.g., the set of cameras 632, than other portions of the object plane 805, imaging of a grid with uniformly spaced cells results in image points 830, which correspond to the centers of the uniformly spaced cells 840, being present at locations different than centers of the cells. Thus, the image produced using the Scheimpflug principle, is an image shape that is characterized by a trapezoidal shape of the image points 830 vs. the grid of uniformly spaced cells 840. Thus, an image is produced that has a resolution at the top of the image that is much higher than the resolution at the bottom of the image.

[0070] In order to address these shortcomings in the conventional approach, embodiments of the present invention reduce keystone distortion and variation in the image magnification across the image, thereby providing substantially constant resolution in the image. An unchanged magnification throughout the image results in an object of a given size being imaged at the same size on the image sensor independent of the object's lateral (i.e., x,y) position in the object plane. As described below in relation to FIG. 9A, embodiments of the present invention orient both the optics and the image sensor parallel to the object plane and use the optics in an off-axis arrangement. Additionally, for some embodiments, to produce reduced or minimum keystone distortion, the angles of the chief rays are maintained by the optics.

[0071] FIG. 9A is a simplified schematic diagram illustrating a parallel orientation object plane and imaging system according to an embodiment of the present invention. As described in relation to FIGS. 9A - 9C, some embodiments of the present invention provide an optical system 900 in which both the optics and the image sensor are oriented parallel to each other and the object plane to reduce keystone distortion. Illumination source 910 (e.g., one or more infrared light emitting diodes) produces illumination light rays 912, which impinge on the object plane 905. Light reflected from the object plane is illustrated by reflected light rays 915, which are collected by the optical system 920 and imaged using the image sensor 940. The object plane 905 (i.e., the plane in which the user eye is positioned) is perpendicular to optical axis 901, i.e., the longitudinal axis, and parallel to the optical system 920 and the image sensor 940. Thus, the orientation of the object plane and the image sensor are aligned so that they are parallel, rather than titled with respect to each other as illustrated in FIG. 8 A. Moreover, the orientation of the object plane and the optical system are also aligned so that they are also parallel, rather than titled with respect to each other as illustrated in FIG. 8A.

[0072] In order to maintain the chief-ray angles after the optical system as close as possible to the chief-ray angles in front of the optical system, the optics are realized in some embodiments approximately symmetrically around the aperture stop, with a diffractive or metaoptical element utilized before and after the aperture stop. The optical power of those two elements is selected to be in a similar range in order to maintain the symmetry of the system. As described more fully herein, the optical power in the optical system 920 and the other optical systems, i.e., imaging systems, discussed herein, can be characterized, for example, by a spherical phase function. To maintain the symmetry of the system, the curvature of the phase functions before and after the aperture stop can be kept at a similar value. For improved imaging performance, the curvature of the phase functions can be aspherical or even asymmetrical / freeform phase profiles. As a result, the phase profile can be realized, for example, either by a diffractive optical element or by a metaoptical element. In other embodiments, refractive elements are utilized. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0073] FIG. 9B is a schematic diagram illustrating the imaging system shown in FIG. 9A according to an embodiment of the present invention. In this detailed view of the optical system 920, which can also be referred to as an imaging system, the front surface of the optical system 920 and the back surface of the optical system have optical power, and theaperture stop 950 is located in the center of the optical system 920. Thus, as illustrated in FIGS. 9A and 9B, an eye tracking system that can be used to image an eye of a user is provided that includes an illumination source operable to illuminate the eye, a portion of which is lying in an object plane, and an optical system 920. As illustrated in FIG. 9A, the eye can be disposed along an optical axis 901 and the image sensor 940 can be laterally offset in the x-direction and / or the y-direction from the optical axis 901.

[0074] The optical system includes a first imaging surface, an aperture stop, and a second imaging surface. The optical system is operable to image an object plane corresponding to the eye and additionally includes an image sensor having an image plane parallel to the first imaging surface, the second imaging surface, and the object plane.

[0075] Referring once again to FIG. 9B, optical system 920 includes a first substrate 922 and a second substrate 930. These substrates can be made of glass or other suitable optical material. The first substrate 922 has a first surface 921 and a second surface 923. The second substrate 930 has a first surface 931 and a second surface 933. In the embodiment illustrated in FIG. 9B, the first surface 921 of the first substrate 922 includes a first diffractive optical element 925 with optical power that results in the front surface 921 being a first imaging surface. Thus, a portion of the first surface 921 or all of the first surface 921 can have optical power. As a result, the first surface 921 can be referred to as a first imaging surface since all or a portion of the front surface can be characterized by optical power.

[0076] The second surface 933 of the second substrate 930 includes a second diffractive optical element 934, which also has optical power, that results in the second surface 933 being a second imaging surface. Thus, a portion of the second surface 933 or all of the second surface 933 can have optical power. As a result, the second surface 933 can be referred to as a second imaging surface since all or a portion of the back surface can be characterized by optical power.

[0077] As illustrated in FIG. 9B, light incident on the first surface 921 is focused, i.e., the first imaging surface has a positive optical power and light incident on the second surface 933 is also focused, i.e., the second optical surface is also characterized by a positive optical power. In some embodiments, the first imaging surface can be characterized by a first aspheric lens profile and the second optical surface can be characterized by a second aspheric lens profile that is different from the first aspheric lens profile. In other embodiments,negative lens functionality is implemented as appropriate to the particular application. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0078] A variety of different diffractive optical elements (e.g., Liquid Crystal Polarization Grating, Holographic Optical Element, or Metasurface) can be utilized to implement first diffractive optical element 925 or second diffractive optical element 934. As an example, the first diffractive optical element can be a holographic optical element and / or the second diffractive optical element can be a holographic optical element. In other embodiments, the first diffractive optical element is a metasurface with optical power and / or the second diffractive optical element comprises a metasurface with optical power. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0079] As illustrated in FIG. 9B, an aperture stop 950 is positioned between the first diffractive optical element 925 positioned on the first surface 921 (i.e., the first imaging surface) and the second diffractive optical element 934 positioned on the second surface 933 (i.e., the second imaging surface). In this embodiment, the first imaging surface is parallel to the aperture stop, which is also parallel to the second imaging surface. Furthermore, as illustrated in FIGS. 9A and 9B, the optical system is an off-axis imaging system in which light is incident on optical system 920 at a non-zero angle of incidence since the optical system 920 and the image sensor 940 are laterally offset from the optical axis 901.

[0080] Some embodiments of the present invention implement one or more of the first imaging surface, the second imaging surface, the aperture stop, and the image plane of the image sensor in a parallel arrangement, which can be parallel to the object plane. Thus, as illustrated in FIGS. 9A and 9B, the first surface 921 (i.e., the first imaging surface), the aperture stop 950, and the second surface 933 (i.e., the second imaging surface) are parallel to each other. Moreover, in some embodiments, these surfaces are also parallel to the image plane 942 of the image sensor 940. As shown in FIG. 9A, all of these surfaces can be parallel to object plane 905. Similar orientations for the imaging surfaces, the aperture stops, and / or the image planes illustrated in other embodiments herein can also be utilized to implement a parallel orientation between one of more of these optical elements. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0081] In some embodiments, only a portion of the first surface and / or the second surface is optically transparent. Referring to FIG. 9B, the first substrate 922 has the first diffractive optical element 925 disposed on the upper portion of the first surface 921 and the aperturestop disposed on the second surface 923. The second substrate 930 has the aperture stop 950 disposed on the first surface 931 and the second diffractive optical element 934 disposed on the lower portion of the second surface 933. Thus, a portion or all of the relevant surface can have optical power. Depending on the lateral extent of the light incident on the first surface 921, a first light absorbing material 952 that is laterally offset from the first diffractive optical element 925 can be applied or otherwise joined to the first surface 921. In a similar manner, depending on the lateral extent of the light exiting the second surface 933, a second light absorbing material 954 that is laterally offset from the second diffractive optical element 934 can be applied or otherwise joined to the second surface 933.

[0082] The optical system 920 includes a center line 959 and the aperture stop 950 can be centered on the center line 959, with the first diffractive optical element 925 disposed on one side of the center line 959 and the second diffractive optical element 934 disposed on an opposite side of the center line 959 from the one side. This design is useful in off-axis imaging arrangements since, as shown in FIG. 9B, the majority of light enters the optical system 920 above the center line 959 and exits the optical system 920 below the center line 959. In these embodiments, the image plane 942 of the image sensor 940 can be laterally offset with respect to the imaging surfaces of the optical system. In this off-axis imaging setup, the center line is parallel to, but laterally offset (i.e., below) the optical axis corresponding to the eye (see optical axis 901 in FIG. 9A).

[0083] As an example, optical system 920 can include a first substrate 922, made of, for example, BK7 glass, with a thickness of 1.149 mm and a second substrate 924, also made of, for example, BK7 glass, with a thickness of 0.901 mm. In this example, the object distance to the optical system 920 can be 20 mm and the distance from the second surface 933 of the second substrate 930 facing the image sensor 940 to the image sensor 940 can be 0.241 mm.

[0084] FIG. 9C is a plot illustrating keystone distortion exhibited by the imaging system shown in FIG. 9B. Using the imaging system 920 illustrated in FIG. 9B, imaging of a grid with uniformly spaced cells results in image points 935, which correspond to the centers of the uniformly spaced cells 937, substantially overlapping with the corresponding cell, with some misalignment for the upper image points / cells as illustrated. Thus, as shown in FIG. 9C, the keystone distortion is almost entirely removed and only small differences in magnification in the x-direction and the y-direction remain, demonstrating the reductions in distortions produced by embodiments of the present invention.

[0085] Using the imaging system 800 illustrated in FIG. 8 A, in which portions of the object plane, e.g., the eye being imaged, are closer to the image sensor 822, e.g., the set of cameras 632, than other portions of the object plane 805, imaging of a grid with uniformly spaced cells results in image points 830, which correspond to the centers of the uniformly spaced cells 840, being present at locations different than centers of the cells. Thus, the image produced using the Scheimpflug principle, is an image shape that is characterized by a trapezoidal shape of the image points 830 vs. the grid of uniformly spaced cells 840. Thus, an image is produced that has a resolution at the top of the image that is much higher than the resolution at the bottom of the image.

[0086] The diffractive optical elements, e.g., first diffractive optical element 925 and second diffractive optical element 934, utilized by embodiments of the present invention can utilize a range of parameters suitable to the particular application. In an exemplary embodiment, a diffraction grating with even polynomials can be utilized as one or both of the diffractive optical elements with parameters as shown in Table 1. The phase profile for this diffraction grating can be represented by the following equation:Table 1

[0087] FIG. 9D is a plot showing the optical phase function for the first imaging surface illustrated in FIG. 9B according to an embodiment of the present invention. The plot shownin FIG. 9D illustrates the optical phase function at the entrance surface (i.e., the side of the optical system 920 that is closer to the eye, which corresponds to the first surface 921 in FIG. 9B) and the coordinate system of the plot is centered with respect to the center of the aperture stop 950 shown in FIG. 9B.

[0088] FIG. 9E is a plot showing the optical phase function for the first imaging surface for the illuminated portion of the phase profile according to an embodiment of the present invention. In this plot, only the used / illuminated portion of the phase profile is shown, which would correspond to the portion of the first surface 921 at which the first diffractive optical element 925 is present.

[0089] FIG. 9F is a plot showing the optical phase function for the second imaging surface illustrated in FIG. 9B according to an embodiment of the present invention. The plot shown in FIG. 9F illustrates the optical phase function at the exit surface (i.e., the side of the optical system 920 that is closer to the image sensor 940, which corresponds to the second surface 933 in FIG. 9B) and the coordinate system of the plot is centered with respect to the center of the aperture stop 950 shown in FIG. 9B.

[0090] FIG. 9G is a plot showing the optical phase function for the second imaging surface for the illuminated portion of the phase profile according to an embodiment of the present invention. In this plot, only the used / illuminated portion of the phase profile is shown, which would correspond to the portion of the second surface 933 at which the second diffractive optical element 934 is present.

[0091] FIG. 10 is a simplified schematic diagram illustrating a folded imaging system according to an embodiment of the present invention. In the imaging system illustrated in FIG. 10, the imaging system is folded at the aperture stop 1014 and both phase profiles for the first diffractive optical element 1012 and the second diffractive optical element 1016 are realized on the first surface 1011. The inventors have determined that the distortion correction is not adversely impacted by this folding of the imaging system. Thus, in this alternate arrangement, the elements of the optical system can be folded using a reflective aperture stop to reduce the length of the imaging system.

[0092] As shown in FIG. 10, the optical system 1000 includes substrate 1010, which can be a glass substrate, having a first surface 1011 and a second surface 1013. The first surface 1011 can be referred to as an imaging surface since it includes a first diffractive optical element 1012 and a second diffractive optical element 1016. The aperture stop 1014 isreflective and positioned on second surface 1013. Light incident from the eye impinges on the first diffractive optical element 1012, is focused a first time as is passes through the first diffractive optical element 1012, is reflected from the aperture stop 1014, and is focused a second time as is passes through the second diffractive optical element 1016. The image of the eye is formed at image plane 1042 of image sensor 1040. The description provided in relation to the optical system 920 illustrated in FIG. 9B is applicable to the optical system 1000 as appropriate, with the difference that both of the phase profiles corresponding to the diffractive optical elements are located on the first surface 1011 (i.e., the same surface) of the substrate 1010. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0093] As illustrated in FIG. 10, the optical system 1000 includes the substrate 1010, which has a center line lying along axis 1059. The aperture stop 1014 is disposed on the second surface 1013 of the substrate 1010 and centered on the center line (i.e., axis 1059). The first diffractive optical element 1012 is disposed on the first surface 1011 of the substrate 1010 on one side of (i.e., above) the center line. The second diffractive optical element 1016 is disposed on the first surface 1011 of the substrate 1010 on the opposite side of the center line from the one side, i.e., below the center line. In this off-axis imaging setup, the center line is parallel to, but laterally offset (i.e., below) the optical axis corresponding to the eye (see optical axis 901 in FIG. 9A). Because off-axis imaging is used in this embodiment, the image sensor 1040 is laterally offset from the first diffractive optical element 1012 and partially laterally overlaps with the second diffractive optical element 1016.

[0094] The inventors have determined that the large angle of incidence (i.e., the chief ray angle) at the image plane illustrated in FIG. 9A typically results in low light efficiency. Accordingly, the inventors have determined that it can be advantageous from a lightefficiency perspective to reduce the chief ray angle at the image space. Accordingly, some embodiments of the present invention, including the embodiments that are described more fully below, are characterized by smaller chief ray angles at the image plane (typically < 10°). Thus, some embodiments provide an improvement in the chief ray angle (i.e., a decrease in the chief ray angle) that offsets an increase in image distortion (e.g., both keystone distortion and lens distortion).

[0095] As described more fully below, the embodiment illustrated in FIG. 11 utilizes a single-layer diffractive optical element, which could be a holographic optical element (HOE),a liquid crystal polarization grating (LCPG) or a metalens embedded between two substrates (e.g., glass substrates). If an air-to-glass interface is utilized for the grating operation, an air space can be added between the two substrates. The aperture stop of the optical system is located near the focal plane of the optical system to produce a small chief ray angle in image space (e.g., < 10° for all fields of view for this exemplary embodiment). The diffractive lens can have a spherical phase or an aspherical phase, and the entrance pupil can be decentered from the active area of the diffractive optical element lens and the sensor to capture an off- centered object.

[0096] FIG. 11 is a simplified schematic diagram illustrating an imaging system according to an alternative embodiment of the present invention. In the imaging system illustrated in FIG. 11, the positions of the diffractive optical element 1114 and the aperture stop 1112 are interchanged with respect to the embodiment illustrated in FIG. 9B. Accordingly, the description provided in relation to optical system 920 shown in FIG. 9B is applicable to the optical system 1100 shown in FIG. 11 as appropriate. Referring to FIG. 11, the optical system 1100 includes a first substrate 1110 and a second substrate 1120. The first substrate 1110 has a first surface 1111 and a second surface 1113. The second substrate 1120 has a first surface 1121 and a second surface 1123. In this embodiment, an aperture stop 1112 is positioned on the first surface 1111 of the first substrate 1110, the diffractive optical element 1114 is formed at the bonded surface defined by the second surface 1113 of the first substrate 1110 and the first surface 1121 of the second substrate 1120. As shown in FIG. 11, in this embodiment, the first imaging surface (i.e., the second surface 1113 of the first substrate 1110 and the first surface 1121 of the second substrate 1120, which include the diffractive optical element 1114) is positioned between the aperture stop 1112 and the second surface 1123 of the second substrate 1120. The imaging plane 1142 of the image sensor 1140 is positioned adjacent the second surface 1123 of the second substrate 1120.

[0097] In some embodiments, an optional second diffractive optical element 1122 is formed on the second surface 1123 of the second substrate 1120. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

[0098] FIG. 12 is a simplified schematic diagram illustrating a multi-element imaging system according to an embodiment of the present invention. In this embodiment, two diffractive optical elements are utilized to reduce both the keystone distortion and the lens distortion. The aperture stop is located near the focal plane of the two diffractive lenses inobject space to ensure a small chief ray angle in image space. As discussed in relation to the other embodiments, spherical, aspherical, and XY polynomial type phase functions can be utilized in this embodiment. By employing high-order polynomial type phase, both types of distortion can be significantly reduced.

[0099] Referring to FIG. 12, the multi-element imaging system 1200 includes a first substrate 1210, a second substrate 1220, and a third substrate 1230. The first substrate 1210 has a first surface 1211 and a second surface 1213. The second substrate 1220 has a first surface 1221 and a second surface 1223. The third substrate 1230 has a first surface 1231 and a second surface 1233. In this embodiment, an aperture stop 1212 is positioned on the first surface 1211 of the first substrate 1210, the first diffractive optical element 1222 is formed at the bonded surface defined by the second surface 1213 of the first substrate 1210 and the first surface 1221 of the second substrate 1220, and the second diffractive optical element is formed at the bonded surface defined by the second surface 1223 of the second substrate 1220 and the first surface 1231 of the third substrate 1230.

[0100] As shown in FIG. 12, in this embodiment, both the first imaging surface (i.e., the second surface 1213 of the first substrate 1210 and the first surface 1221 of the second substrate 1220, which include the first diffractive optical element 1222) and the second imaging surface (i.e., the second surface 1223 of the second substrate 1220 and the first surface 1231 of the third substrate 1230, which include the second diffractive optical element 1224) are positioned between the aperture stop 1212 and the image plane 1242 of the image sensor 1240.

[0101] As an example, multi-element imaging system 1200 can include a first substrate 1210, made of, for example, BK7 glass, with a thickness of 0.462 mm, a second substrate 1220, also made of, for example, BK7 glass, with a thickness of 1.064 mm, and a third substrate 1230, also made of, for example, BK7 glass, with a thickness of 0.403 mm. In this example, the object distance to the multi-element imaging system 1200 can be 20 mm and the distance from the surface of the third substrate 1230 facing the image sensor 1240 to the image sensor 1240 can be 0.241 mm.

[0102] The diffractive optical elements utilized by embodiments of the present invention can utilize a range of parameters suitable to the particular application. In an exemplary embodiment, two diffraction gratings with even polynomials can be utilized as the diffractiveoptical elements with parameters as shown in Table 2. The grating phase parameters for this set of diffraction gratings can be represented by the following equation:Table 2

[0103] FIG. 13 is a simplified schematic diagram illustrating a multi-element imaging system according to an alternative embodiment of the present invention. The multi-element imaging system 1300 illustrated in FIG. 13 shares common elements with the multi-element imaging system 1200 illustrated in FIG. 12 and the description provided in relation to the multi-element imaging system 1200 illustrated in FIG. 12 is applicable to the multi-element imaging system 1300 illustrated in FIG. 13 as appropriate.

[0104] Referring to FIG. 13, a symmetric design is illustrated that utilizes two diffractive lenses with the aperture stop positioned between the two diffractive lenses. As discussed above, spherical, aspherical and XY polynomial type phase functions can be utilized in this embodiment. Similar to the multi-element imaging system 1200 discussed in relation to FIG. 12, the multi-element imaging system 1300 illustrated in FIG. 13 can employ high-order polynomial type phase, in order to reduce both types of distortion while maintaining a small chief ray angle in image space.

[0105] As shown in FIG. 13, the multi -element imaging system 1300 includes a first substrate 1310, a second substrate 1320, and a third substrate 1330. The first substrate 1310 has a first surface 1311 and a second surface 1313. The second substrate 1320 has a first surface 1321 and a second surface 1323. The third substrate 1330 has a first surface 1331 and a second surface 1333. In this embodiment, the first diffractive optical element 1312 is positioned on the first surface 1311 of the first substrate 1310, the aperture stop 1322 is formed at the bonded surface defined by the second surface 1313 of the first substrate 1310 and the first surface 1321 of the second substrate 1320, and the second diffractive optical element is formed at the bonded surface defined by the second surface 1323 of the second substrate 1320 and the first surface 1331 of the third substrate 1330.

[0106] As shown in FIG. 13, in this embodiment, the aperture stop 1322 is positioned between the first imaging surface (i.e., the first surface 1311 of the first substrate 1310, which includes the first diffractive optical element 1312) and the second imaging surface (i.e., the second surface 1323 of the second substrate 1320 and the first surface 1331 of the third substrate 1330, which include the second diffractive optical element 1324). The image plane 1342 of the image sensor 1340 is positioned adjacent the second surface 1333 of the third substrate 1330. Thus, in this embodiment, the first substrate includes the first imaging surface and the aperture stop and the third substrate includes the second imaging surface.

[0107] FIG. 14 is a simplified schematic diagram illustrating a multi-element imaging system according to yet another alternative embodiment of the present invention. The multielement imaging system 1400 illustrated in FIG. 14 shares common elements with the multielement imaging system 1200 illustrated in FIG. 12 and the description provided in relation to the multi-element imaging system 1200 illustrated in FIG. 12 is applicable to the multielement imaging system 1400 illustrated in FIG. 14 as appropriate.

[0108] Referring to FIG. 14, two diffractive optical elements are provided on the same substrate (i.e., second substrate 1420. The first substrate 1410 can be implemented using a cover glass with aperture stop on the back surface (i.e., second surface 1413). As discussed above, spherical, aspherical and XY polynomial type phase functions can be utilized in this embodiment. Similar to the multi-element imaging system 1200 discussed in relation to FIG. 12, the multi-element imaging system 1400 illustrated in FIG. 14 can employ high-order polynomial type phase, in order to reduce both types of distortion while maintaining a small chief ray angle in image space.

[0109] As shown in FIG. 14, the multi -element imaging system 1400 includes a first substrate 1410 and a second substrate 1420. The first substrate 1410 has a first surface 1411 and a second surface 1413. The second substrate 1420 has a first surface 1421 and a second surface 1423. In this embodiment, the first diffractive optical element 1412 is positioned on the first surface 1422 of the second substrate 1420. The aperture stop 1422 is formed adjacent the air gap 1415 and is formed on the second surface 1413 of the first substrate 1410. The second diffractive optical element 1424 is formed on the second surface 1423 of the second substrate 1420. The image plane 1442 of the image sensor 1440 is positioned adjacent the second surface 1423 of the second substrate 1420. Thus, in this embodiment, , the aperture stop is formed on the first substrate, and the second substrate includes both the first imaging surface and the second imaging surface.

[0110] As an example, multi-element imaging system 1400 can include a cover window represented by first substrate 1410, made of, for example, BK7 glass, with a thickness of 0.3 mm, and a second substrate 1420, made of, for example, BK7 glass, with a thickness of 1.146 mm. The first substrate 1410 and the second substrate 1420 are separated by an air gap 1415 in this example, with a gap thickness of 0.02 mm. In this example, the object distance to the multi-element imaging system 1400 can be 20 mm and the distance from the surface of the second substrate 1420 facing the image sensor 1440 to the image sensor 1440 can be 0.670 mm.[OHl] In the embodiment illustrated in FIG. 14, a higher order XY-polynomial is utilized for the diffractive optical elements. For this example, the coordinate origin is located at the center of the aperture stop 1422 and the second diffractive optical element 1424 is decentered in Y by -0.461 mm. Using higher order phase function helps to correct both keystone and radial distortion. Furthermore, the use of an elliptical shape aperture stop with the major axis oriented in Y axis helps to improve image quality.

[0112] The diffractive optical elements utilized by embodiments of the present invention can utilize a range of parameters suitable to the particular application. In an exemplary embodiment, two diffraction gratings with XY-polynomial s can be utilized as the diffractive optical elements with parameters as shown in Table 3. The grating phase parameters for this set of diffraction gratings can be represented by the following equation:Table 3

[0113] FIG. 15 is a simplified block diagram illustrating components of an AR system according to an embodiment of the present invention. AR system 1500 as illustrated in FIG. 15 may be incorporated into the AR devices as described herein. FIG. 15 provides aschematic illustration of one embodiment of AR system 1500 that can perform some or all of the steps of the methods provided by various embodiments. It should be noted that FIG. 15 is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. FIG. 15, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

[0114] AR system 1500 is shown comprising hardware elements that can be electrically coupled via a bus 1505, or may otherwise be in communication, as appropriate. The hardware elements may include one or more processors 1510, including without limitation one or more general-purpose processors and / or one or more special-purpose processors such as digital signal processing chips, graphics acceleration processors, and / or the like; one or more input devices 1530, which can include without limitation a mouse, a keyboard, a camera, and / or the like; and one or more output devices 1540, which can include without limitation a display device, a printer, and / or the like. Additionally, AR system 1500 includes an eye tracking system 1570 that can provide the user's eye gaze location to the AR system. Utilizing one or more processors 1510, the presentation of virtual content using the AR system discussed herein can be implemented.

[0115] AR system 1500 may further include and / or be in communication with storage device(s) 1520 (e.g., one or more non-transitory storage devices), which can comprise without limitation local and / or network accessible storage, and / or can include without limitation a disk drive, a drive array, an optical storage device, a solid-state storage device such as a random access memory (RAM) and / or a read-only memory (ROM) which can be programmable, flash-updateable, and / or the like. Such storage devices may be configured to implement any appropriate data stores including without limitation various file systems, database structures, and / or the like.

[0116] AR system 1500 might also include a communications subsystem 1550, which can include without limitation a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc., and / or the like. Communications subsystem 1550 may include one or more input and / or output communication interfaces to permit data to be exchanged with a network such as the network described below to name one example, other computer systems, television, and / or any other devices described herein. Depending on the desiredfunctionality and / or other implementation concerns, a portable electronic device or similar device may communicate image and / or other information via communications subsystem 1550. In other embodiments, a portable electronic device, e.g., the first electronic device, may be incorporated into AR system 1500, e.g., an electronic device as an input device 1530. In some embodiments, AR system 1500 will further comprise a working memory 1560, which can include a RAM or ROM device, as described above.

[0117] AR system 1500 also can include software elements, shown as being currently located within working memory 1560, including an operating system 1562, device drivers, executable libraries, and / or other code, such as one or more application programs 1564, which may comprise computer programs provided by various embodiments, and / or may be designed to implement methods, and / or configure systems, provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above might be implemented as code and / or instructions executable by a computer and / or a processor within a computer; in an aspect, then, such code and / or instructions can be used to configure and / or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.

[0118] A set of these instructions and / or code may be stored on a non-transitory computer- readable storage medium, such as storage device(s) 1520 described above. In some cases, the storage medium might be incorporated within a computer system, such as AR system 1500. In other embodiments, the storage medium might be separate from a computer system e.g., a removable medium, such as a compact disc, and / or provided in an installation package, such that the storage medium can be used to program, configure, and / or adapt a general purpose computer with the instructions / code stored thereon. These instructions might take the form of executable code, which is executable by AR system 1500 and / or might take the form of source and / or installable code, which, upon compilation and / or installation on AR system 1500, e.g., using any of a variety of generally available compilers, installation programs, compression / decompression utilities, etc., then takes the form of executable code.

[0119] It will be apparent to those skilled in the art that substantial variations may be made in accordance with specific requirements. For example, customized hardware might also be used, and / or particular elements might be implemented in hardware, software including portable software, such as applets, etc., or both. Further, connection to other computing devices such as network input / output devices may be employed.

[0120] As mentioned above, in one aspect, some embodiments may employ a computer system such as AR system 1500 to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all procedures of such methods are performed by AR system 1500 in response to one or more processors 1510 executing one or more sequences of one or more instructions, which might be incorporated into operating system 1562 and / or other code, such as an application program 1564, contained in working memory 1560. Such instructions may be read into working memory 1560 from another computer-readable medium, such as one or more of storage device(s) 1520. Merely by way of example, execution of the sequences of instructions contained in working memory 1560 might cause one or more processors 1510 to perform one or more procedures of the methods described herein. Additionally, or alternatively, portions of the methods described herein may be executed through specialized hardware.

[0121] The terms machine-readable medium and computer-readable medium, as used herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using AR system 1500, various computer-readable media might be involved in providing instructions / code to one or more processors 1510 for execution and / or might be used to store and / or carry such instructions / code. In many implementations, a computer-readable medium is a physical and / or tangible storage medium. Such a medium may take the form of a non-volatile media or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device(s) 1520. Volatile media include, without limitation, dynamic memory, such as working memory 1560.

[0122] Common forms of physical and / or tangible computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0123] Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to one or more processors 1510 for execution. Merely by way of example, the instructions may initially be carried on a magnetic disk and / or optical disc of a remote computer. A remote computer might load the instructions into itsdynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by AR system 1500.

[0124] Communications subsystem 1550 and / or components thereof generally will receive signals, and bus 1505 then might carry the signals and / or the data, instructions, etc. carried by the signals to working memory 1560, from which one or more processors 1510 retrieves and executes the instructions. The instructions received by working memory 1560 may optionally be stored on storage device(s) 1520, e.g., a non-transitory storage device, either before or after execution by one or more processors 1510.

[0125] Various examples of the present disclosure are provided below. As used below, any reference to a series of examples is to be understood as a reference to each of those examples disjunctively (e.g., "Examples 1-4" is to be understood as "Examples 1, 2, 3, or 4").

[0126] Example 1 is an eye tracking system operable to image an eye of a user, the eye tracking system comprising: an illumination source operable to illuminate the eye; an optical system including a first imaging surface, an aperture stop, and a second imaging surface, wherein the optical system is operable to image an object plane corresponding to the eye; and an image sensor having an image plane parallel to the first imaging surface, the second imaging surface, and the object plane.

[0127] Example 2 is the eye tracking system of example 1 wherein the eye is disposed along an optical axis and the image sensor is laterally offset from the optical axis.

[0128] Example 3 is the eye tracking system of example(s) 1-2 wherein: the first imaging surface comprises a first diffractive optical element; and the second imaging surface comprises a second diffractive optical element.

[0129] Example 4 is the eye tracking system of example 3 wherein the first diffractive optical element or the second diffractive optical element comprises a holographic optical element.

[0130] Example 5 is the eye tracking system of example 3 wherein the first diffractive optical element or the second diffractive optical element comprises a metasurface.

[0131] Example 6 is the eye tracking system of example 3 further comprising: a first light absorbing material laterally offset from the first diffractive optical element; and a second light absorbing material laterally offset from the second diffractive optical element.

[0132] Example 7 is the eye tracking system of example(s) 1-6 wherein the first imaging surface is parallel to the aperture stop and the second imaging surface.

[0133] Example 8 is the eye tracking system of example(s) 1-7 wherein the first imaging surface and the second imaging surface are characterized by a positive optical power.

[0134] Example 9 is the eye tracking system of example(s) 1-8 wherein the first imaging surface is characterized by a first aspheric lens profile and the second imaging surface is characterized by a second aspherical lens profile different from the first aspheric lens profile.

[0135] Example 10 is the eye tracking system of example(s) 1-9 wherein the optical system comprises an off-axis imaging system.

[0136] Example 11 is the eye tracking system of example(s) 1-10 wherein the aperture stop is positioned between the first imaging surface and the second imaging surface.

[0137] Example 12 is the eye tracking system of example(s) 1-11 wherein the first imaging surface is positioned between the aperture stop and the second imaging surface.

[0138] Example 13 is the eye tracking system of example(s) 1-12 wherein the optical system comprises a first substrate and a second substrate.

[0139] Example 14 is the eye tracking system of example 13 wherein: the first substrate comprises a first glass substrate; and the second substrate comprises a first second substrate.

[0140] Example 15 is the eye tracking system of example 13 wherein: the first substrate comprises the first imaging surface and the aperture stop; and the second substrate comprises the aperture stop and the second imaging surface.

[0141] Example 16 is the eye tracking system of example 13 wherein: the first substrate includes the aperture stop and the first imaging surface; and the second substrate includes the first imaging surface and the second imaging surface.

[0142] Example 17 is the eye tracking system of example 13 wherein: the optical system comprises a center line; the aperture stop is centered on the center line; the first imaging surface is disposed on one side of the center line; and the second imaging surface is disposed on an opposite side of the center line from the one side.

[0143] Example 18 is the eye tracking system of example(s) 1-17 wherein the image plane of the image sensor is laterally offset with respect to the first imaging surface of the optical system.

[0144] Example 19 is the eye tracking system of example(s) 1-18 wherein: the optical system comprises a substrate having a center line; the aperture stop is disposed on a first surface of the substrate and centered on the center line; the first imaging surface is disposed on a second surface of the substrate on one side of the center line; and the second imaging surface is disposed the second surface of the substrate on an opposite side of the center line from the one side.

[0145] Example 20 is the eye tracking system of example 19 wherein the eye is disposed along an optical axis and the center line is parallel to and laterally offset from the optical axis.

[0146] Example 21 is the eye tracking system of example 19 wherein the image sensor is laterally offset from the first imaging surface and laterally overlaps with the second imaging surface.

[0147] Example 22 is the eye tracking system of example(s) 1-21 wherein the optical system comprises a first substrate, a second substrate, and a third substrate.

[0148] Example 23 is the eye tracking system of example 22 wherein: the first substrate includes the aperture stop and the first imaging surface; and the third substrate includes the second imaging surface.

[0149] Example 24 is the eye tracking system of example 22 wherein: the first substrate includes the first imaging surface and the aperture stop; and the third substrate includes the second imaging surface.

[0150] Example 25 is the eye tracking system of example(s) 1-24 wherein the aperture stop has an elliptical shape.

[0151] Example 26 is an augmented reality (AR) system comprising: a wearable device including a frame; a communication device coupled to the frame; a projector coupled to the frame; a waveguide display optically coupled to the projector and disposed along an optical axis; a lens element disposed along the optical axis; an illumination source operable to illuminate an eye of a user of the wearable device; one or more image sensors, each of the one or more image sensors having an image plane oriented perpendicular to the optical axis,wherein the one or more image sensors are positioned at predetermined lateral distances from the optical axis; and an optical element having an imaging surface parallel to the image plane, wherein the optical element is disposed between the eye of the user and the image sensor.

[0152] Example 27 is the AR system of example 26 further comprising a remote computing device including: a processor; a memory; and a communication system coupled to the processor.

[0153] Example 28 is the AR system of example(s) 26-27 wherein the one or more image sensors consist of two image sensors.

[0154] Example 29 is the AR system of example(s) 26-28 wherein the eye is disposed along an optical axis and the one or more image sensors is laterally offset from the optical axis.

[0155] Example 30 is the AR system of example(s) 26-29 wherein the imaging surface comprises a first diffractive optical element, wherein the optical element further comprises a second diffractive optical element.

[0156] Example 31 is the AR system of example 30 wherein the first diffractive optical element or the second diffractive optical element comprises a holographic optical element.

[0157] Example 32 is the AR system of example 30 wherein the first diffractive optical element or the second diffractive optical element comprises a metasurface.

[0158] Example 33 is the AR system of example 30 further comprising: a first light absorbing material laterally offset from the first diffractive optical element; and a second light absorbing material laterally offset from the second diffractive optical element.

[0159] Example 34 is the AR system of example(s) 26-33 wherein the optical element further comprises an aperture stop and the imaging surface is parallel to the aperture stop.

[0160] Example 35 is the AR system of example 34 wherein the optical element further comprises a second imaging surface, wherein the aperture stop is positioned between the imaging surface and the second imaging surface.

[0161] Example 36 is the AR system of example 35 wherein the imaging surface is positioned between the aperture stop and the second imaging surface.

[0162] Example 37 is the AR system of example(s) 26-36 wherein the imaging surface is characterized by a positive optical power.

[0163] Example 38 is the AR system of example(s) 26-37 wherein the optical element comprises a second imaging surface, wherein the imaging surface is characterized by a first aspheric lens profile and the second imaging surface is characterized by a second aspherical lens profile different from the first aspheric lens profile.

[0164] Example 39 is the AR system of example(s) 26-38 wherein the optical element comprises an off-axis imaging system.

[0165] Example 40 is the AR system of example(s) 26-39 wherein the optical element comprises a first substrate, a second substrate, and an aperture stop.

[0166] Example 41 is the AR system of example 40 wherein: the first substrate comprises a first glass substrate; and the second substrate comprises a first second substrate.

[0167] Example 42 is the AR system of example 40 wherein: the first substrate comprises the imaging surface and the aperture stop; and the second substrate comprises the aperture stop and a second imaging surface.

[0168] Example 43 is the AR system of example 40 wherein: the first substrate includes the aperture stop and the imaging surface; and the second substrate includes the imaging surface and a second imaging surface.

[0169] Example 44 is the AR system of example 40 wherein: the optical element comprises a center line; the aperture stop is centered on the center line; the imaging surface is disposed on one side of the center line; and a second imaging surface is disposed on an opposite side of the center line from the one side.

[0170] Example 45 is the AR system of example(s) 26-44 wherein the image plane of the one or more image sensors is laterally offset with respect to the imaging surface of the optical element.

[0171] Example 46 is the AR system of example(s) 26-43 wherein: the optical element comprises a substrate having a center line and an aperture stop is disposed on a first surface of the substrate and centered on the center line; the imaging surface is disposed on a second surface of the substrate on one side of the center line; and a second imaging surface isdisposed the second surface of the substrate on an opposite side of the center line from the one side.

[0172] Example 47 is the AR system of example 46 wherein the eye is disposed along an optical axis and the center line is parallel to and laterally offset from the optical axis.

[0173] Example 48 is the AR system of example 46 wherein the optical element comprises a second imaging surface, wherein the one or more image sensors is laterally offset from the imaging surface and laterally overlaps with the second imaging surface.

[0174] Example 49 is the AR system of example(s) 26-48 wherein the optical element comprises a first substrate, a second substrate, and a third substrate.

[0175] Example 50 is the AR system of example 49 wherein: the first substrate includes an aperture stop and the imaging surface; and the third substrate includes a second imaging surface.

[0176] Example 51 is the AR system of example 49 wherein: the first substrate includes the imaging surface and an aperture stop; and the third substrate includes the second imaging surface.

[0177] Example 52 is the AR system of example(s) 26-51 wherein the optical element further comprises an aperture stop having an elliptical shape.

[0178] In the foregoing specification, the disclosure has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.

[0179] Indeed, it will be appreciated that the systems and methods of the disclosure each have several innovative aspects, no single one of which is solely responsible or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another, or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure.

[0180] Certain features that are described in this specification in the context of separate embodiments also may be implemented in combination in a single embodiment. Conversely,various features that are described in the context of a single embodiment also may be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. No single feature or group of features is necessary or indispensable to each and every embodiment.

[0181] It will be appreciated that conditional language used herein, such as, among others, "can," "could," "might," "may," "e.g.," and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps. Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and / or steps are included or are to be performed in any particular embodiment. The terms "comprising," "including," "having," and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the" as used in this application and the appended claims are to be construed to mean "one or more" or "at least one" unless specified otherwise. Similarly, while operations may be depicted in the drawings in a particular order, it is to be recognized that such operations need not be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Furthermore, the drawings may schematically depict one or more example processes in the form of a flowchart. However, other operations that are not depicted may be incorporated in the example methods and processes that are schematically illustrated. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. Additionally, the operations may be rearranged or reordered in other embodiments. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments,and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0182] Accordingly, the claims are not intended to be limited to the embodiments shown herein but are to be accorded the widest scope consistent with this disclosure, the principles, and the novel features disclosed herein. Thus, it is also understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and scope of the appended claims.

Claims

WHAT IS CLAIMED IS:

1. An eye tracking system operable to image an eye of a user, the eye tracking system comprising: an illumination source operable to illuminate the eye; an optical system including a first imaging surface, an aperture stop, and a second imaging surface, wherein the optical system is operable to image an object plane corresponding to the eye; and an image sensor having an image plane parallel to the first imaging surface, the second imaging surface, and the object plane.

2. The eye tracking system of claim 1 wherein the eye is disposed along an optical axis and the image sensor is laterally offset from the optical axis.

3. The eye tracking system of claim 1 wherein: the first imaging surface comprises a first diffractive optical element; and the second imaging surface comprises a second diffractive optical element.

4. The eye tracking system of claim 3 wherein the first diffractive optical element or the second diffractive optical element comprises a holographic optical element.

5. The eye tracking system of claim 3 wherein the first diffractive optical element or the second diffractive optical element comprises a metasurface.

6. The eye tracking system of claim 3 further comprising: a first light absorbing material laterally offset from the first diffractive optical element; and a second light absorbing material laterally offset from the second diffractive optical element.

7. The eye tracking system of claim 1 wherein the first imaging surface is parallel to the aperture stop and the second imaging surface.

8. The eye tracking system of claim 1 wherein the first imaging surface and the second imaging surface are characterized by a positive optical power.

9. The eye tracking system of claim 1 wherein the first imaging surface is characterized by a first aspheric lens profile and the second imaging surface is characterized by a second aspherical lens profile different from the first aspheric lens profile.

10. The eye tracking system of claim 1 wherein the optical system comprises an off-axis imaging system.

11. The eye tracking system of claim 1 wherein the aperture stop is positioned between the first imaging surface and the second imaging surface.

12. The eye tracking system of claim 1 wherein the first imaging surface is positioned between the aperture stop and the second imaging surface.

13. The eye tracking system of claim 1 wherein the optical system comprises a first substrate and a second substrate.

14. The eye tracking system of claim 13 wherein: the first substrate comprises a first glass substrate; and the second substrate comprises a first second substrate.

15. The eye tracking system of claim 13 wherein: the first substrate comprises the first imaging surface and the aperture stop; and the second substrate comprises the aperture stop and the second imaging surface.

16. The eye tracking system of claim 13 wherein: the first substrate includes the aperture stop and the first imaging surface; and the second substrate includes the first imaging surface and the second imaging surface.

17. The eye tracking system of claim 13 wherein: the optical system comprises a center line; the aperture stop is centered on the center line; the first imaging surface is disposed on one side of the center line; andthe second imaging surface is disposed on an opposite side of the center line from the one side.

18. The eye tracking system of claim 1 wherein the image plane of the image sensor is laterally offset with respect to the first imaging surface of the optical system.

19. The eye tracking system of claim 1 wherein: the optical system comprises a substrate having a center line; the aperture stop is disposed on a first surface of the substrate and centered on the center line; the first imaging surface is disposed on a second surface of the substrate on one side of the center line; and the second imaging surface is disposed the second surface of the substrate on an opposite side of the center line from the one side.

20. The eye tracking system of claim 19 wherein the eye is disposed along an optical axis and the center line is parallel to and laterally offset from the optical axis.

21. The eye tracking system of claim 19 wherein the image sensor is laterally offset from the first imaging surface and laterally overlaps with the second imaging surface.

22. The eye tracking system of claim 1 wherein the optical system comprises a first substrate, a second substrate, and a third substrate.

23. The eye tracking system of claim 22 wherein: the first substrate includes the aperture stop and the first imaging surface; and the third substrate includes the second imaging surface.

24. The eye tracking system of claim 22 wherein: the first substrate includes the first imaging surface and the aperture stop; and the third substrate includes the second imaging surface.

25. The eye tracking system of claim 1 wherein the aperture stop has an elliptical shape.

26. An augmented reality (AR) system comprising:a wearable device including a frame; a communication device coupled to the frame; a projector coupled to the frame; a waveguide display optically coupled to the projector and disposed along an optical axis; a lens element disposed along the optical axis; an illumination source operable to illuminate an eye of a user of the wearable device; one or more image sensors, each of the one or more image sensors having an image plane oriented perpendicular to the optical axis, wherein the one or more image sensors are positioned at predetermined lateral distances from the optical axis; and an optical element having an imaging surface parallel to the image plane, wherein the optical element is disposed between the eye of the user and the image sensor.

27. The AR system of claim 26 further comprising a remote computing device including: a processor; a memory; and a communication system coupled to the processor.

28. The AR system of claim 26 wherein the one or more image sensors consist of two image sensors.

29. The AR system of claim 26 wherein the eye is disposed along an optical axis and the one or more image sensors is laterally offset from the optical axis.

30. The AR system of claim 26 wherein the imaging surface comprises a first diffractive optical element, wherein the optical element further comprises a second diffractive optical element.

31. The AR system of claim 30 wherein the first diffractive optical element or the second diffractive optical element comprises a holographic optical element.

32. The AR system of claim 30 wherein the first diffractive optical element or the second diffractive optical element comprises a metasurface.

33. The AR system of claim 30 further comprising: a first light absorbing material laterally offset from the first diffractive optical element; and a second light absorbing material laterally offset from the second diffractive optical element.

34. The AR system of claim 26 wherein the optical element further comprises an aperture stop and the imaging surface is parallel to the aperture stop.

35. The AR system of claim 34 wherein the optical element further comprises a second imaging surface, wherein the aperture stop is positioned between the imaging surface and the second imaging surface.

36. The AR system of claim 35 wherein the imaging surface is positioned between the aperture stop and the second imaging surface.

37. The AR system of claim 26 wherein the imaging surface is characterized by a positive optical power.

38. The AR system of claim 26 wherein the optical element comprises a second imaging surface, wherein the imaging surface is characterized by a first aspheric lens profile and the second imaging surface is characterized by a second aspherical lens profile different from the first aspheric lens profile.

39. The AR system of claim 26 wherein the optical element comprises an off-axis imaging system.

40. The AR system of claim 26 wherein the optical element comprises a first substrate, a second substrate, and an aperture stop.

41. The AR system of claim 40 wherein: the first substrate comprises a first glass substrate; and the second substrate comprises a first second substrate.

42. The AR system of claim 40 wherein: the first substrate comprises the imaging surface and the aperture stop; and the second substrate comprises the aperture stop and a second imaging surface.

43. The AR system of claim 40 wherein: the first substrate includes the aperture stop and the imaging surface; and the second substrate includes the imaging surface and a second imaging surface.

44. The AR system of claim 40 wherein: the optical element comprises a center line; the aperture stop is centered on the center line; the imaging surface is disposed on one side of the center line; and a second imaging surface is disposed on an opposite side of the center line from the one side.

45. The AR system of claim 26 wherein the image plane of the one or more image sensors is laterally offset with respect to the imaging surface of the optical element.

46. The AR system of claim 26 wherein: the optical element comprises a substrate having a center line and an aperture stop is disposed on a first surface of the substrate and centered on the center line; the imaging surface is disposed on a second surface of the substrate on one side of the center line; and a second imaging surface is disposed the second surface of the substrate on an opposite side of the center line from the one side.

47. The AR system of claim 46 wherein the eye is disposed along an optical axis and the center line is parallel to and laterally offset from the optical axis.

48. The AR system of claim 46 wherein the optical element comprises a second imaging surface, wherein the one or more image sensors is laterally offset from the imaging surface and laterally overlaps with the second imaging surface.

49. The AR system of claim 26 wherein the optical element comprises a first substrate, a second substrate, and a third substrate.

50. The AR system of claim 49 wherein: the first substrate includes an aperture stop and the imaging surface; andthe third substrate includes a second imaging surface.

51. The AR system of claim 49 wherein: the first substrate includes the imaging surface and an aperture stop; and the third substrate includes the second imaging surface.

52. The AR system of claim 26 wherein the optical element further comprises an aperture stop having an elliptical shape.

Images