Apparatus and method for eye tracking based on eye imaging via light guide optical element

Abstract

To provide an eye tracker and a corresponding method for tracking the gaze direction of a human eye on the basis of imaging the eye via a light guide optical element.SOLUTION: A light-transmitting substrate has parallel surfaces deployed with a first of the parallel surfaces facing an eye. An optical element is associated with the first surface, deflects incident light of a first type so as to collimate the incident light, and does not substantially deflect incident light of a second type. An optical coupling configuration is associated with a substrate and is configured for coupling-in a portion of collimated light of the first type incident on the first surface so as to propagate within the substrate, and for coupling-out a portion of light of the second type propagating within the substrate. Optics associated with the substrate convert collimated light of the first type into converging beams of light, which are sensed by an optical sensor. A processor derives the current gaze direction of the eye by processing signals from the optical sensor.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority from U.S. Provisional Patent Application No. 62 / 867,249, filed June 27, 2019, and U.S. Provisional Patent Application No. 62 / 869,582, filed July 2, 2019, the disclosures of which are incorporated by reference in their entireties herein.

[0002] The present invention relates to eye tracking. [Background technology]

[0003] The optical arrangements of near-eye displays (NEDs), head-mounted displays (HMDs), and head-up displays (HUDs) require a large aperture to cover the area where the observer's eyes are located (commonly called the eye movement box or EMB). To implement compact devices, the image projected to the observer's eyes is produced by a small optical image generator (projector) with a small aperture that is multiplied to produce the large aperture.

[0004] An approach to aperture multiplication in one dimension has been developed based on a parallel-sided slab of transparent material within which the image propagates by internal reflection. Part of the image wavefront is coupled out of the slab by using an obliquely angled partial reflector or by using a diffractive optical element on one side of the slab. Such a slab is referred to herein as a light guide optical element (LOE), light-transmitting substrate, or waveguide. The principle of such aperture multiplication is shown diagrammatically in FIG. 1, which shows a light guide optical element 20 having a pair of parallel faces 26, 26A for guiding light by internal reflection. A projected image 18, represented diagrammatically here by an illumination beam 18 including sample rays 18A and 18B spanning the beam, is coupled in to the light guide optical element 20, as diagrammatically shown here by a first reflecting surface 16, to generate a reflected ray 28 that is trapped by internal reflection in the substrate, and also generates a ray 30. The image propagates along the substrate by repeated internal reflections and is reflected at an oblique angle (α sur ) impinges on a sequence of partially reflective surfaces 22 where a portion of the image intensity is reflected and coupled out of the substrate as rays 32A, 32B towards the observer's eye 24. To minimize undesired reflections that may result in ghost images, the partially reflective surfaces 22 are preferably coated to have a low reflectivity for a first range of angles of incidence, while having a desired partial reflectivity for a second range of angles of incidence. Here, rays that have a small inclination with respect to the normal to the partially reflective surfaces 22 (here angle β ref ) are split to generate reflected rays for coupling out, while rays at high inclinations (with respect to the normal) are transmitted with negligible reflection.

[0005] The projected image 18 is a collimated image, i.e., each pixel is represented by a beam of parallel rays at a corresponding angle, corresponding to light from the scene far away from the observer (a collimated image is said to be "infinitely collimated"). Here, the image is simply represented by rays corresponding to a single point in the image, typically the image's center of gravity, but in reality a range of angles to each side of this central beam are included, with the rays coupled into the substrate at a corresponding range of angles, and likewise coupled out at corresponding angles, thereby creating a field of view corresponding to portions of the image arriving in different directions at the observer's eye 24.

[0006] An optical feature that may be useful in the design of a NED, HMD, or HUD is eye tracking, or sensing the direction an observer's eyes are looking relative to the head direction (commonly referred to as gaze direction). Past eye tracking approaches have relied on imaging the EMB via one or more off-axis cameras looking toward the EMB from the side. To reduce user discomfort, the cameras need to be relatively small in size, which can limit the performance of EMB imaging. The small size of the cameras and the general difficulty of deriving gaze direction from EMB images sampled at high off-axis angles result in relatively poor performance for such eye tracking approaches. Summary of the Invention

[0007] Aspects of the present invention provide an eye tracker and corresponding method for tracking the gaze direction of a human eye based on imaging the eye through light guide optics, and are particularly suitable for integration as part of a NED, HMD or HUD.

[0008] According to the teachings of one embodiment of the present invention, there is provided an optical coupling arrangement comprising an optically transparent substrate having at least two parallel major surfaces for directing light by internal reflection, a first of the major surfaces being positioned facing an eye, an optical element associated with the first major surface, the optical element configured to impart refraction to incident light of a first type to collimate the incident light of the first type and to impart substantially no refraction to incident light of a second type in accordance with at least one characteristic of the incident light, and an optical coupling arrangement associated with the substrate and configured to couple the incident light to the optical element. An apparatus is provided, comprising: an optical coupling arrangement configured to couple in a portion of a first type of collimated light incident on a first major surface to propagate within the substrate, and to couple out a portion of a second type of light propagating within the substrate; an optical system associated with the substrate and configured to convert the first type of collimated light into a converging beam of captured light; an optical sensor positioned to sense the captured light; and at least one processor electrically associated with the optical sensor and configured to process a signal from the optical sensor to derive a current gaze direction of the eye.

[0009] Optionally, the at least one characteristic of the incident light includes a polarization direction of the incident light.

[0010] Optionally, the at least one characteristic of the incident light includes a region of the electromagnetic spectrum occupied by the incident light.

[0011] Optionally, the at least one characteristic of the incident light includes a polarization direction of the incident light and a region of the electromagnetic spectrum occupied by the incident light.

[0012] Optionally, the first type of light comprises a component of light polarized in a first polarization direction, and the second type of light is polarized in a second polarization direction.

[0013] Optionally, the first type of light is in a first light spectrum and the second type of light is in a second light spectrum.

[0014] Optionally, the first type of light includes a component of light polarized in a first polarization direction and is within a first light spectrum, and the second type of light is polarized in a second polarization direction and is within a second light spectrum.

[0015] Optionally, the apparatus further comprises a polarizer associated with a second of the major surfaces of the substrate.

[0016] Optionally, the substrate is positioned with the first major surface at an eye relief distance from the eye, and the optical element has a focal length approximately equal to the eye relief distance.

[0017] Optionally, the apparatus further comprises a second optical coupling arrangement associated with the optical system and configured to couple out a portion of the first type of light propagating within the substrate such that the coupled-out light is accepted by the optical system, and to couple in a portion of the second type of light from the display source to propagate within the substrate by internal reflection.

[0018] Optionally, the apparatus further comprises an illumination arrangement arranged to illuminate the eye with the first type of light.

[0019] Optionally, the apparatus further comprises an image projector coupled to the substrate to introduce a second type of collimated light corresponding to an image into the substrate, such that the second type of coupled-in collimated light propagates by internal reflection within the substrate and is coupled out from the substrate towards the eye by the light coupling arrangement.

[0020] Optionally, the image projector includes a reflective display device that generates polarized light in response to illumination from the polarized light source, and the polarized light generated by the reflective display device is collimated by an optical system.

[0021] Optionally, the optical coupling arrangement comprises a plurality of partially reflective surfaces disposed within the substrate at an angle relative to a major surface of the substrate.

[0022] Optionally, a first type of light propagates in a first propagation direction within the substrate, and a second type of light propagates in a second propagation direction within the substrate, opposite the first propagation direction.

[0023] According to an embodiment of the present teachings, there is provided an optically transparent substrate having a pair of parallel major surfaces for directing light by internal reflection, a first of the major surfaces being positioned facing an eye of a viewer; a lens associated with the first major surface configured to impart refraction to a first type of incident light to collimate the first type of incident light, the first type of incident light being within a first light spectrum and including a component of light polarized in a first polarization direction, and to impart substantially no refraction to a second type of incident light, the second type of incident light being within a second light spectrum and having a polarization in a second polarization direction; an illumination arrangement arranged to illuminate the eye with the first type of light such that a portion of the first type of light is reflected back by the eye towards the lens to be collimated by the lens; an optical module, a reflective display device responsive to illumination from a light source to generate a second type of light corresponding to an image; and a reflector configured to reflect light from the reflective display device to generate the collimated second type of light. an optical coupling arrangement configured to couple in the collimated light of a second type to the substrate to propagate within the substrate by internal reflection in a first propagation direction; a plurality of partially reflective surfaces disposed within the substrate at an oblique angle to a major surface of the substrate, the plurality of partially reflective surfaces configured to couple out a portion of the light of the second type propagating within the substrate in the first propagation direction and to couple in a portion of the collimated light of the first type incident on the first major surface to propagate within the substrate in the second propagation direction, the optical coupling arrangement further configured to couple out the propagating light of the first type; and at least one processor electrically coupled to the optical sensor, wherein the optical coupling arrangement of the optical module is further configured to accept light coupled out by the optical coupling arrangement and to convert the coupled out light into a convergent beam of captured light, and the optical sensor is further configured to:Also provided is an apparatus configured to sense the captured light, and wherein at least one processor is configured to process signals from the optical sensor to derive a current gaze direction of the eye.

[0024] According to an embodiment of the present teachings, a light guiding device is provided comprising: a first light-transmitting substrate having at least two substantially parallel major surfaces for directing light by internal reflection, a first of the major surfaces being positioned facing an eye of a viewer; an at least partially reflective surface positioned within the first substrate at an angle to the major surfaces, the at least partially reflective surface being configured to couple in incident light rays incident on the first major surface in a coupling-in region to propagate within the first substrate by internal reflection, the incident light rays being in a first light spectrum and emanating from the eye in response to illumination of the eye, the incident light rays comprising at least a first set of light rays and a second set of light rays, the first set of light rays having an angular distribution that spans at least a portion of the coupling-in region in a first dimension, and the second set of light rays spanning at least a portion of the coupling-in region in a second dimension; and Also provided is an apparatus comprising: a coupling-out arrangement configured to couple out light rays propagating within the plate; an optical module including at least one lens having a first focal length in a first dimension and a second focal length in a second dimension and configured to convert the coupled-out light rays corresponding to the first set of light rays into a non-converging beam of captured light having an angular distribution indicative of the angular distribution of the first set of light rays, and convert the coupled-out light rays corresponding to the second set of light rays into a converging beam of captured light; and an optical sensor positioned at a distance substantially equal to the first focal length from the lens and configured to sense the captured light; and at least one processor electrically coupled to the optical sensor configured to process a signal from the optical sensor to derive a current gaze direction of the eye.

[0025] Optionally, the apparatus further comprises an illumination arrangement arranged to illuminate the eye with light of the first light spectrum.

[0026] Optionally, the device further comprises a second optically transparent substrate having a plurality of surfaces including first and second major surfaces parallel to each other for guiding light by internal reflection, wherein the first major surface of the second substrate is positioned facing the eye and the second major surface of the second substrate faces the first major surface of the first substrate, and a coupling-out arrangement associated with the second substrate, configured to couple out a portion of light of a second optical spectrum, different from the first optical spectrum and propagating within the second substrate, from the second substrate towards the eye.

[0027] Optionally, the apparatus further comprises an image projector coupled to the second substrate and configured to generate collimated light in a second light spectrum corresponding to an image, wherein the collimated light propagates by internal reflection within the second substrate and is coupled out from the second substrate towards the eye by a coupling-out arrangement.

[0028] Optionally, the apparatus further comprises a coupling-in arrangement associated with the image projector and with the second substrate configured to couple in collimated light generated by the image projector into the second substrate.

[0029] Optionally, the coupling-out arrangement comprises a plurality of partially reflective surfaces arranged in the second substrate at an angle to a major surface of the second substrate.

[0030] Optionally, the coupling-out arrangement comprises a diffractive optical element associated with one of the major surfaces of the second substrate.

[0031] Unless otherwise defined herein, all technical and / or scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. Although methods and materials similar or equivalent to those described herein can be used to carry out or test embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. Moreover, the materials, methods, and examples are illustrative only and are not necessarily intended to be limiting. [Brief description of the drawings]

[0032] Some embodiments of the present invention are described herein, by way of example only, with reference to the accompanying drawings. With particular reference to the drawings in detail, it is emphasized that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the present invention. In this regard, the description taken in conjunction with the drawings will make apparent to those skilled in the art how embodiments of the present invention may be practiced.

[0033] Attention is now directed to the drawings, in which like reference numbers or letters indicate corresponding or similar elements, in which:

[0034] [Figure 1] 1 is a schematic side view of a prior art light guide optic employing a partially reflective surface for use in a near-eye display; FIG. [Diagram 2] FIG. 1 is a schematic side view of an apparatus constructed and operable in accordance with an embodiment of the present invention for displaying images and tracking the gaze direction of a human eye, showing the propagation of light from the eye through a light-transmitting substrate to an image projector. [Diagram 3] FIG. 3 is a schematic side view of an alternative configuration of the apparatus of FIG. 2, in which the light coupling arrangement for coupling light between the light-transmitting substrate and the image projector is implemented as a reflective surface. [Figure 4]FIG. 3 is a schematic side view of the apparatus of FIG. 2, showing the propagation of image light from the image projector and the propagation of external scene light to the eye. [Diagram 5] FIG. 5 is a schematic exploded plan view of the image projector of FIGS. 2-4, illustrating the propagation of image light to the output of the image projector. [Figure 6] FIG. 6 is a schematic exploded plan view of the image projector of FIG. 5, illustrating the propagation of eye tracking light to the optical sensor. [Figure 7] FIG. 7 is a schematic exploded plan view of the image projector of FIGS. 5 and 6 modified by the addition of a retardation plate, showing the propagation of eye tracking light to an optical sensor. [Figure 8] FIG. 2 is a partial schematic isometric view of the device of FIG. 1 implemented in an eyeglass form factor. [Figure 9] FIG. 1 is a schematic side view of an apparatus constructed and operative in accordance with another embodiment of the present invention for displaying images and tracking the gaze direction of a human eye, having a first optically transparent substrate for eye tracking and a second optically transparent substrate for image projection, illustrating the propagation of light from the eye through the second optically transparent substrate to an image projector. [Figure 10] 10 is a partial schematic side view of a first optically transmissive substrate of the device of FIG. 9, illustrating the coupling of a light beam from the eye to the first optically transmissive substrate. [Figure 11] FIG. 11 is a partially schematic side view similar to FIG. 10 showing multiple light rays spanning a light beam from an eye in a first dimension. [Figure 12] FIG. 12 is a schematic side view of the optical module of the apparatus of FIG. 9, showing light beams corresponding to those of FIG. 11 being directed by lenses to an optical sensor. [Figure 13] FIG. 11 is an isometric view of a first optically transmissive substrate of the device of FIGS. 9 and 10, showing multiple light rays spanning the light beam from the eye in the second dimension. [Figure 14] FIG. 14 is a schematic side view of the optical module of the apparatus of FIG. 9, showing light rays corresponding to those of FIG. 13 being focused by a lens onto an optical sensor. [Figure 15] FIG. 10 is a schematic exploded plan view of the image projector of FIG. 9, illustrating the propagation of image light and eye tracking light to the output of the image projector. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Embodiments of the present invention provide various apparatus and corresponding methods for tracking the gaze direction of a human eye based on imaging the eye and / or identifying the angular distribution of light reflected by the eye through light guide optics.

[0036] The principles and operation of various eye tracking devices according to the present invention may be better understood with reference to the drawings accompanying the description.

[0037] Before describing at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangements of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples, as the invention is capable of other embodiments or of being practiced or carried out in various ways.

[0038] By way of introduction, in many applications, particularly in the context of head-up or near-eye displays, it is useful to provide an eye tracking arrangement for determining a user's gaze direction. One common approach to performing eye tracking is to sample an image of the eye, typically with the purpose of determining the position of the pupil in the image and thereby deriving the eye orientation. It would be particularly advantageous to employ light guide optics operating on a principle similar to that of FIG. 1 to sample images for eye tracking.

[0039] Eye tracking solutions using light guide optics operating on such principles or similar such principles are described herein. In one set of solutions according to certain aspects of the invention, the eye is imaged by coupling light reflected from the eye (referred to as a first type of light) into the light guide optics, whereby the light propagates along a reverse path through the light guide optics in a counter-propagating direction of the image light from the image projector (referred to as a second type of light) and is focused onto an optical sensor disposed on the image projector, where a signal generated by the optical sensor in response to sensing the light is processed by a processing system to derive the gaze direction. Since the eye is not located at infinity from the light guide optics (rather, at an eye relief distance typically on the order of about 20 millimeters), the light reflected from the eye is collimated by an optical element, preferably a polarizing and / or spectrally selective lens that distinguishes between the first type and the second type of light, before being coupled into the light guide optics to accurately derive the gaze direction from the light focused onto the optical sensor.

[0040] In another set of solutions according to aspects of the present invention, the gaze direction is determined by a special partially reflective surface, preferably in a dedicated light guide optics separate from the LOE through which the projected image propagates, which couples the uncollimated light reflected from the eye into the light guide optics, whereby the coupled-in light propagates along a reverse path through the light guide optics and is coupled out to an optical module that includes a lens with two focal lengths in orthogonal dimensions, which directs the coupled-out light to an optical sensor.

[0041] 2-8 illustrate various aspects of the structure and operation of an apparatus, generally designated 100, constructed and operable in accordance with various embodiments of the present invention for displaying images and guiding the gaze direction of a human eye 110 by a collimating optical element 112 (hereinafter referred to as lens 112) disposed between the eye 110 and a light guide optical element (LOE) 102. The LOE 102 is formed from a transparent material and has a pair of parallel faces (planar major surfaces) 104, 106 for guiding light by internal reflection (preferably total internal reflection). The LOE 102 is positioned with one of the parallel faces 104 facing the eye 110, where the eye 110 is located within the EMB 109 at an eye relief (ER) distance 111 from the faces 104. An optical coupling arrangement, implemented as a set of partially reflective surfaces 108, is associated with the LOE 102 and configured to couple in a portion of light incident on face 104 within a coupling-in region for propagation within the LOE 102 by (total) internal reflection. In particular, the partially reflective surfaces 108 are disposed within the LOE 102 (i.e., between the faces 104, 106) at an angle to the parallel faces 104, 106. The coupling-in region of the LOE 102, also referred to as the "active region" or "active area", is generally defined as the area spanned by the projection of the partially reflective surfaces 108 in the plane of the faces 104.

[0042] Lens 112 is associated with surface 104 (by optical attachment to LOE 102) such that lens 112 is positioned between LOE 102 and eye 110. Lens 112 preferably has a focal length approximately equal to ER 111. Light reflected from eye 110 (in response to illumination of eye 110 by illumination arrangement 138) is collimated by lens 112, and the collimated light is incident on surface 104 and coupled into LOE 102 by partially reflective surface 108 by internal reflection to propagate within LOE 102. Optical element 140 (hereinafter lens 140) is associated with LOE 102 to receive the captured light propagating within LOE 102 and convert the collimated light (a set of parallel light rays) propagating within LOE 102 into a converging beam of captured light. Preferably, lens 140 is integrated into optical module 126 along with optical sensor 128 configured to sense the captured light, and lens 140 is associated with LOE 102 via an optical coupling arrangement 124 that couples captured light propagating within LOE 102 out from LOE 102 to optical module 126. Processing system 130 includes at least one computerized processor 132 coupled to storage medium 134 (such as a computer memory, etc.), is electrically associated with optical sensor 128, and is configured to process signals from optical sensor 128 to derive a current gaze direction of eye 110.

[0043] Optical coupling arrangement 124 may be any coupling arrangement that redirects incident light from LOE 102 to optical module 126. Suitable optical coupling arrangements include, but are not limited to, reflective surfaces (shown diagrammatically in FIG. 2) and prisms (shown diagrammatically in FIG. 3).

[0044] Generally speaking, the eye 110 is illuminated with light by the illumination arrangement 138. As discussed, the illumination arrangement 138 is configured to illuminate the eye 110 with light having wavelengths outside the photopic region of the electromagnetic spectrum. In other words, the illumination arrangement 138 is configured to illuminate the eye 110 with light that is invisible to the human eye. Reflection from the human eye, and in particular from the retina of the eye, is significantly higher in the near infrared than in visible wavelengths. Therefore, the illumination arrangement 138 is preferably configured to illuminate the eye 110 with light having wavelengths in the near infrared (NIR) region of the electromagnetic spectrum. Furthermore, as will be discussed in detail in subsequent sections of this disclosure, the illumination arrangement 138 is also preferably configured to illuminate the eye 110 such that the light reflected by the eye 110 in response to illumination from the illumination arrangement 138 includes at least a component of light having a specific polarization direction (literally p-polarized) with respect to the surface of the lens 112.

[0045] 2 and 3, which illustrate the traversal of a ray of light from eye 110 through LOE 102 to optical sensor 128. In general, light propagating within LOE 102 from eye 110 to optical sensor 128 is referred to as propagating within LOE 102 in a counterpropagation direction (interchangeably referred to as first / second propagation direction, first / second direction, or reverse direction), while image light propagating within LOE 102 from image projector to eye 110 is referred to as propagating within LOE 102 in a forward propagation direction (interchangeably referred to as second / first propagation direction, second / first direction, or forward direction). A portion of the intensity of light from illumination arrangement 138 that is incident on eye 110 is reflected by eye 110. The reflected light emanating from eye 110 is generally represented in FIGS. 2 and 3 as sample rays 114A-114F. Light emanating from eye 110 is collimated by lens 112, and the collimated light is represented diagrammatically as rays 116A-116F (each of rays 114A-114F has a corresponding collimated ray 116A-116F). Collimated rays 116A-116F enter face 104 of LOE 102 generally perpendicular to face 104 and are coupled into LOE 102 by partially reflective surface 108 to produce reflected ray 118 (a downward ray) that is trapped by internal reflection within LOE 102, and also produces (upward) ray 120. Light reflected from eye 110 propagates along the substrate until it reaches optical coupling arrangement 124 (schematically shown as a reflective surface in FIG. 2 and as a prism in FIG. 3 ), which couples the light (rays 118 and 120) out of LOE 102 to optical module 126 as rays 122A, 122B, and 122C. Lens 140 converts the collimated coupled-out light (rays 122A, 122B, and 122C) into a converging beam of captured light so as to focus the coupled-out light (rays 122A, 122B, and 122C) onto optical sensor 128.

[0046] In addition to integrating lens 140 and optical sensor 128 therein, optical module 126 preferably also includes components for generating and projecting an image onto LOE 102 for viewing by eye 110 (similar to projected image 18 of FIG. 1), such that optical module 126 performs the dual functions of image projection and light focusing and sensing. As discussed, lens 140 also functions to collimate light rays generated by the display device of optical module 126.

[0047] Reference is now made to Figure 4, which illustrates the forward light propagation within LOE 102. Similar to Figure 1, projected image 142, here represented diagrammatically by illumination beam 142 including beam-spanning sample rays 142A, 142B, and 142C, is produced by optical module 126 and coupled into LOE 102 via optical coupling arrangement 124 (shown diagrammatically here by a reflecting surface) to produce reflected ray 144 (upward ray) that is trapped by internal reflection within LOE 102, and also produces ray 146 (downward ray). Image 142 propagates along LOE 102 by repeated internal reflections between faces 104, 106, and impinges on partially reflecting surface 108, where a portion of the image intensity is reflected and coupled out of LOE 102 towards eye 110 as rays 148A, 148B, and 148C. However, before reaching the eye 110 , the light rays 148 A- 148 C necessarily pass through the lens 112 .

[0048] While it is important that lens 112 imparts refraction to light emanating from eye 110 so as to collimate light rays 114A-114F to enable accurate sensing of the captured light (by optical sensor 128) and processing of signals from optical sensor 128 to derive the current gaze direction of eye 110 (by processing system 130), it is equally important that lens 112 does not impart refraction to image light propagating from optical module 126 to eye 110 via LOE 102, since refraction of light rays 148A, 148B, and 148C would distort the projected image 142 when viewed by eye 110. Thus, a particular feature of this embodiment is that lens 112 is designed to distinguish between two types of light (light reflected from the eye propagating through LOE 102 to the focusing and sensing components of optical module 126 (referred to as a first type of light), represented by light rays 114A-114F, and image light from the image projection components of optical module 126 (referred to as a second type of light), represented by light rays 142A-142C), and imparts refraction to only one of these types of light (i.e., the first type of light, i.e., the reflected eye light). In the context of this document, the terms "first type of light", "first type of light wave", "first type of light", "first type of light wave", and variations thereof are used interchangeably. In the context of this document, the terms "second type of light", "second type of light wave", "second type of light", "second type of light wave", and variations thereof are used interchangeably.

[0049] According to certain preferred embodiments, the distinction is performed based on at least one characteristic of the light incident on the lens 112. In other words, the lens 112 is designed such that the lens 112 selectively imparts refraction to the incident light according to at least one characteristic of the incident light. In certain embodiments, one characteristic, e.g., the wavelength (i.e., the optical spectrum) of the incident light is used as the basis for distinguishing between the first type of light and the second type of light, in other embodiments, another characteristic, e.g., the polarization direction or polarization orientation of the components of the incident light, is used as the basis for distinguishing between the first type of light and the second type of light, while in still other preferred embodiments, both the optical spectrum (wavelength) and the polarization direction of the incident light are used as the basis for distinguishing between the first type of light and the second type of light.

[0050] It should be noted that, in general, in contrast to the light illuminating the eye 110, the image light 142 (second type of light) has a wavelength in the photopic region of the electromagnetic spectrum (i.e., between 380 nanometers (nm) and about 700 nm). Thus, the lens 112 can be designed to impart refraction only to light having a wavelength outside the photopic region of the electromagnetic spectrum. Furthermore, in many applications, it is preferred that the image light projected by the optical module 126 is linearly polarized in a particular polarization direction (preferably s-polarized). Thus, the lens 112 can be designed to impart refraction to polarized light having a polarization direction rotated relative to the polarization direction of the coupled-out image light projected by the optical module 126. Thus, the lens 112 is preferably designed to be polarizing and spectrally selective such that refraction is imparted to the first type of incident light waves to collimate the first type of incident light waves, and the lens 112 does not impart refraction to the second type of incident light waves, in which the first type of incident light waves have components of a first polarization direction (e.g., p-polarized) and have wavelengths in a first light spectrum (e.g., the NIR region of the electromagnetic spectrum), and the second type of incident light waves have a second polarization direction rotated relative to the first polarization direction (e.g., s-polarized) and have wavelengths in a second light spectrum (e.g., the photopic (or visible) region of the electromagnetic spectrum). For this purpose, in the case of the first type of incident light waves, the lens 112 has a focal length approximately equal to ER111.

[0051] In the above-described exemplary configuration of lens 112, light rays 114A-114F (first type of light) represent the p-polarized (with respect to the surface of lens 112) component of light emanating from eye 110 and have wavelengths in the NIR region of the electromagnetic spectrum, while light rays 148A-148C (second type of light) coupled out of LOE 102 are s-polarized (with respect to the surface of lens 112) and have wavelengths in the visible region of the electromagnetic spectrum. As a result of the polarization- and wavelength-dependent refractive discrimination performed by lens 112, lens 112 imparts refraction to p-polarized NIR light waves so as to collimate rays 116A-116F (first type of light), and does not impart refraction to s-polarized visible image light waves coupled out of LOE 102 so that rays 148A-148C (second type of light) coupled out of LOE 102 (by partially reflective surface 108) pass through lens 112 undistorted by lens 112. Furthermore, lens 112 does not impart refraction to any s-polarized component of the NIR light reflected from eye 110.

[0052] One particular class of materials that exhibits birefringence (polarization) and / or spectral properties are liquid crystals, which have different effects on different polarizations of light, and sometimes different wavelengths. For example, nematic phase liquid crystal molecules respond differently to incident light of two different linear polarizations (s-polarized and p-polarized). In an exemplary but non-limiting embodiment, lens 112 is implemented as a nematic phase liquid crystal lens comprised of a layer of liquid crystal material. The layer of liquid crystal material assumes a state that provides an adjustable focal length, such that lens 112 has a defined focal length for light polarized in one polarization direction (e.g., p-polarized) so as to act as a collimator for that light, and lens 112 imparts no refraction to light of the orthogonal polarization (e.g., s-polarized). Each liquid crystal molecule in a nematic phase liquid crystal has a different sensitivity to each linear polarization, and thus can induce different refractive indices in the liquid crystal molecules. As such, incident light of one polarization direction does not "see" a change in refractive index, while incident light of the other polarization "sees" a change in refractive index, thereby inducing a lens effect for light of that polarization.

[0053] In twisted nematic liquid crystals, each liquid crystal molecule has a different sensitivity to each circular polarization (e.g., right-handed circularly polarized light (or RHP), and left-handed circularly polarized light (or LHP)). Typically, the sensitivity of twisted nematic liquid crystals is such that a positive power lens effect is induced for RHP, and a negative power lens effect is induced for LHP. Introducing another isotropic lens with the same focal length induced by the liquid crystal lens 112 can double the refraction for one polarization and cause no refraction for the other polarization. Since the refraction is imparted differently to RHP and LHP light, a quarter-wave plate 150 is preferably placed between the eye 110 and the lens 112 to appropriately rotate the circular polarization direction of the reflected light from the eye 110.

[0054] Lenses constructed of liquid crystal material are generally constructed of thin grating-type structures (similar to Fresnel lenses) that produce a diffractive dispersion of incident light. Each grating can be designed to have a greater intensity for a particular diffraction order of that grating. The high intensity of that particular diffraction order is chromatic (i.e., wavelength dependent). Thus, the grating can be designed such that for wavelengths in the NIR region, the paired intensity of the first and higher order diffraction nodes is greater than the intensity of the zeroth order diffraction node. In the photopic region, the intensity of the higher order nodes should be small or nonexistent. The orientation of the grating is spatially varied such that the above conditions are met for light in the NIR and photopic regions, thereby producing a lensing effect such that the lens 112 effectively collimates light, while incident light having wavelengths in a second light spectrum (e.g., the photopic (or visible) region of the electromagnetic spectrum) is essentially unaffected by the lens 112. It should be noted that the orientation of the grating of liquid crystal molecules is altered to spatially vary the refractive index of the liquid crystal molecules without utilizing the birefringent properties of the liquid crystal material.

[0055] In general, the lens 112 may be designed to discriminate based on a combination of wavelength and polarization. However, if the spectral separation between the first and second light spectra is large enough without adversely affecting the light from the image projector, discrimination between the first and second types of light based only on wavelength may be sufficient. Generally speaking, the effect of the lens 112 on the light from the image projector may be evaluated based on one or more image quality metrics including, for example, MTF, haze, checkerboard contrast, etc.

[0056] It should be noted that the device 100 of the present disclosure is particularly applicable when used in an augmented reality (AR) system in which an image projected by the optical module 126 is overlaid on a real-world scene that is visible to an observer through the faces 104, 106 and the partially reflective surface 108. It is therefore also preferable that light waves from the real-world scene passing through the faces 104, 106 of the LOE 102 are not distorted by the lens 112 before reaching the eye 110. To prevent light waves from the real-world scene from being distorted by the lens 112, a polarizer 136 is associated with the face 106 that transmits only the component of the incident light in a second polarization direction (e.g., s-polarized). The polarizer 136 and the LOE 102 preferably have a common elongation direction (arbitrarily shown herein as corresponding to the x-axis). Preferably, polarizer 136 is positioned to extend across the entirety (or nearly the entirety) of surface 106, such that light from the entire real-world field of view (corresponding to a wide angular distribution of incident light rays) is appropriately polarized by polarizer 136 before impinging on surface 106.

[0057] The effect of polarizer 136 on a real-world scene is shown diagrammatically in FIG. 4. As shown, a real-world scene image 152, as diagrammatically represented by illumination beam 152, here including sample rays 152A and 152B spanning the beam, impinges on polarizer 136, which transmits only the s-polarized components of rays 152A and 152B. Because rays 152A and 152B are s-polarized and have wavelengths in the photopic region, s-polarized rays 152A and 152B, like rays 148A-148C, pass through lens 112 and reach eye 110 without being distorted by lens 112 (i.e., lens 112 does not impart any refraction to rays 152A and 152B).

[0058] As discussed in the background section, to minimize unwanted reflections that can result in ghost images, the partially reflective surfaces are preferably coated to have low reflectivity for a first range of angles of incidence, while having a desired partial reflectivity for a second range of angles of incidence. In the prior art configuration of FIG. 1, these coatings are typically specific to the wavelength range and polarization of the projected image. For example, if the projected image is comprised of s-polarized light with wavelengths in the photopic region of the electromagnetic spectrum, the partially reflective surfaces are coated to have low reflectivity for s-polarized light in the photopic region at a first range of angles of incidence, while having a desired partial reflectivity for s-polarized light in the photopic region at a second range of angles of incidence. This coating scheme is ideal for the configuration of FIG. 1 because light only propagates in the forward direction, and the partially reflective surface 22 is only used to couple light out of the LOE 20. However, in the configuration of Figures 2-4, where the first type of light (p-polarized component of NIR light reflected from eye 110) propagates in the reverse direction and the second type of light (s-polarized photopic light from the image projector) propagates in the forward direction, and partially reflective surface 108 is configured to couple the first type of light into LOE 102 and couple the second type of light out of LOE 102, a modified coating scheme should be followed to ensure the desired reflectance appropriate for the first type of light. Specifically, partially reflective surface 108 is preferably coated as in the configuration of Figure 1 and further coated to have the desired reflectance for p-polarized light in the NIR region at a prescribed range of angles of incidence.

[0059] As previously discussed, the optical module 126 performs the dual role of image projection and light focusing and sensing. The following paragraphs describe the structure and operation of the optical module 126 in its role as both an image projector for projecting an image 142, and a focusing and sensing arrangement for focusing light reflected from the eye 110 onto the optical sensor 128.

[0060] 5, the optical module 126 (also referred to as image projector 126) includes an illumination prism 160 and a collimation focusing prism 180, each formed from a light wave transmissive material. The illumination prism 160 has several exterior surfaces, including a light wave entrance face 168, an image display face 170, a light wave entrance face 172, and a light wave exit face 174. A polarization selective beam splitter arrangement 166 is disposed within the prism 160 on a plane oblique to the light wave entrance face 168. The prism 160 is based on two constituent prisms, namely a first constituent prism 162 and a second constituent prism 164, at least one of which is provided on its hypotenuse side with a polarizing beam splitter (e.g. a wire grid beam splitter) forming at least part of a polarization-selective beam splitter arrangement 166, which reflects s-polarized light and transmits p-polarized light (incident on the surface of the beam splitter). The two hypotenuse sides of the prisms 162, 164 are bonded together to form a bonded single illumination prism assembly. This single bonded prism is used to illuminate a reflective display device (for image projection) and to direct the incident light reflected from the eye 110 to an optical sensor 128 associated to a light wave exit surface 174. The polarizing beam splitter can be provided via a polarization selective coating directly on one of the sides of the hypotenuse, or via a thin piece of material, e.g. a sheet, foil or glass plate, on which a polarization selective coating has been deposited, whereby the thin piece of material is attached to one of the sides of the hypotenuse.

[0061] In certain preferred embodiments, surfaces 170 and 172 are mutually parallel, and surfaces 168 and 174 are mutually parallel. In certain particularly preferred embodiments, prism 160 is a rectangular prism, i.e., has mutually orthogonal rectangular faces, which in the particular particularly preferred example illustrated here is a square rectangular prism, with each constituent prism 162 and 164 having a 45 degree right angle cross-sectional shape.

[0062] Collimation focusing prism 180 also has several exterior surfaces including a first light wave entrance face 190 (aligned and parallel to light wave entrance face 172), a second light wave entrance face 194, a collimation focusing surface 192, and a fourth surface 188. A polarizing and spectrally selective beam splitter arrangement 186 is disposed within prism 180 on a plane oblique to surface 188. As can be seen in Figure 5, beam splitter arrangements 166 and 186 are in parallel planes. The prism 180 is based on two constituent prisms, namely a first constituent prism 182 and a second constituent prism 184, at least one of which is provided on its hypotenuse side with a polarizing and spectrally selective beam splitter forming at least part of a polarizing and spectrally selective beam splitter arrangement 186, which reflects p-polarized light, transmits s-polarized light with wavelengths in a first optical spectrum (e.g. in the NIR region), reflects s-polarized light and transmits p-polarized light with wavelengths in a second optical spectrum (e.g. in the photopic (or visible) region). The two hypotenuse sides of the prisms 182, 184 are joined together to form a single joined collimation focusing prism assembly. This single cemented prism is used to direct light from the reflective display device to an optical element (collimating focusing component lens 140) to collimate the display light, and also to direct incident light reflected from the eye 110 to an optical element to focus the light via illumination prism 160 to optical sensor 128. A polarizing and spectrally selective beam splitter can be provided in the form of a dielectric coating directly on one of the hypotenuse sides, via a polarizing and spectrally selective coating.

[0063] In certain preferred embodiments, surfaces 190 and 192 are mutually parallel, and surfaces 188 and 194 are mutually parallel. In certain particularly preferred embodiments, prism 180 is a rectangular prism, i.e., has mutually orthogonal rectangular faces, which in the particular particularly preferred example illustrated here is a square rectangular prism, with each constituent prism 182 and 184 having a 45 degree right angle cross-sectional shape.

[0064] A polarized light source 176 (which may be a combination of a light source (e.g., an LED) and a polarizer) is associated with the light wave entrance surface 168. The polarized light source 176 is configured to emit polarized light in a second light spectrum (i.e., the visible region), represented diagrammatically as an incident beam 158. A reflective display device 178 (preferably implemented as a liquid crystal on silicon (LCoS) microdisplay), which generates a spatial modulation of the reflected light corresponding to an image, is associated with the image display surface 170. The reflective display device 178 is illuminated by the incident beam 158 from the polarized light source 176 reflected from the beam splitter arrangement 166. The reflective display device 178 is configured such that the reflected light corresponding to bright areas of the desired image has a polarization rotated with respect to the polarized light source. Thus, as shown in FIG. 5, the polarized illumination 158 enters the prism 160 through the light wave entrance surface 168 with a first polarization, typically s-polarized light with respect to the surface of the beam splitter arrangement 166, and is reflected towards the image display surface 170 where it impinges on the reflective display device 178. Pixels corresponding to bright areas of the image are reflected with a modulated circular polarization (typically p-polarized), so that radiation from the bright pixels is transmitted through the beam splitter arrangement 166 and exits the prism 160 via transmission through the light wave entrance face 172. The light then enters the prism 180 through the light wave entrance face 190 with a second polarization (typically p-polarized with respect to the surface of the polarizing and spectrally selective beam splitter arrangement 186) and reaches the collimation focusing face 192, where it passes through at least one retardation plate 196, preferably a quarter wave plate, associated with at least a portion of the collimation focusing face 192, enters at least one light wave collimating focusing component, i.e. lens 140, covering at least a portion of the retardation plate 196, and is reflected back through the retardation plate 196 by the reflective surface 141 of the lens 140.A double pass through retardation plate 196, aligned with its fast axis at 45 degrees to the polarization axis, rotates the polarization (e.g., converting p-polarized light to s-polarized light) so that the collimated image illumination is reflected off polarizing and spectrally selective beamsplitter arrangement 186 towards light wave entrance face 194 and exits prism 180 as illumination beam 142. Illumination beam 142 is then coupled into LOE 102 by optical coupling arrangement 124.

[0065] 6 shows a schematic optical path followed by collimated light reflected from eye 110 through optical module 126 after being coupled out of LOE 102 by optical coupling arrangement 124. Looking back to Figures 2 and 3, collimated light 116A-116F reflected from eye 110 propagating backwards through LOE 102 is coupled out of LOE 102 by optical coupling arrangement 124 as rays 122A-122C, which are representative of illumination beam 122. The illumination 122 (typically in the NIR region) may include two orthogonally polarized components that are collimated (i.e., a first polarization component (e.g., p-polarized relative to the surface of the beam splitter arrangement 186) and a second polarization component (e.g., s-polarized, not collimated, relative to the surface of the beam splitter arrangement 186). The illumination 122 enters the prism 180 through a light wave entrance face 194. As discussed, the polarizing and spectrally selective beam splitter arrangement 186 reflects p-polarized light and transmits s-polarized light having wavelengths in the first optical spectrum (e.g., the NIR region). Thus, the second polarization component of the illumination 122 is transmitted by the beam splitter arrangement 186 and exits the prism 180 via surface 188. The first polarization component of the illumination 122 (typically collimated) is orthogonally polarized relative to the surface of the beam splitter arrangement 186. 6. The p-polarized light) is reflected by the polarizing and spectrally selective beam splitter configuration 186 and reaches the collimation focusing surface 192 where it passes through a retardation plate 196, enters the collimating focusing component 140, and is reflected back through the retardation plate 196 by the reflective surface 141 of the lens 140. While the collimating focusing component 140 acts to collimate the uncollimated illumination 158 of FIG. 5, the collimating focusing component 140 performs the opposite function for the collimated illumination 122 of FIG. 6, namely, imparting refraction to the incoming collimated light rays to convert a set of parallel light rays (collimated light rays 122A, 122B, and 122C) into a converging beam of captured light, i.e., focusing the illumination 122 onto the optical sensor 128.Further, similar to that described with reference to FIG. 5, a double pass through a retardation plate 196 aligned with its fast axis at 45 degrees to the polarization axis rotates the polarization of the illumination 122 (e.g., converting p-polarized light to s-polarized light) so that the focused illumination is transmitted through the polarizing and spectrally selective beam splitter arrangement 186 and exits the prism 180 via transmission through the light wave entrance face 190. The light then enters the prism 160 through the light wave entrance face 172 with a first polarization (typically s-polarized light relative to the surface of the beam splitter arrangement 166). As previously described, the beam splitter arrangement 166 reflects s-polarized light and transmits p-polarized light. Both NIR and visible light are treated in the same manner by the beam splitter arrangement 166, since these reflection and transmission properties are based solely on the polarization of the incident light. Thus, the s-polarized NIR light is reflected off the beam splitter arrangement 166 towards the light wave exit face 174 and exits the prism 160 as a focused illumination beam 198 that impinges on the optical sensor 128 .

[0066] It should be noted that for each case where a particular polarized wave path is followed in the embodiments described herein, the polarizations are interchangeable, such that, for example, when changing the polarization selection properties of the beam splitter arrangements 166, 186 and the lens 112, each reference to p-polarized light can be replaced with s-polarized light, and vice versa. For example, the lens 112 can be configured to collimate the s-polarized component of (NIR) light. In such a configuration, the polarized light source 176 is configured to emit a p-polarized input beam 158, the beam splitter arrangement 166 reflects p-polarized light and transmits s-polarized light (in both the photopic and NIR regions), and the beam splitter arrangement 186 reflects s-polarized light and transmits p-polarized light with wavelengths in the NIR region, and reflects p-polarized light and transmits s-polarized light with wavelengths in the photopic (visible) region.

[0067] The polarization and spectrally selective beam splitter configuration 186 shown in Figures 5 and 6 may have certain drawbacks, particularly the complexity of designing coatings that provide the appropriate spectral and polarization selective transmission and reflection of the incident light. One alternative to the beam splitter design shown in Figures 5 and 6 is shown in Figure 7. Here, the beam splitter configuration 186 is implemented as a polarization selective beam splitter configuration (similar to the beam splitter configuration 166), i.e., it reflects s-polarized light and transmits p-polarized light of both the first and second optical spectrums (i.e., visible and NIR light are treated the same by the beam splitter configuration 186). Because the beam splitter configuration 186 shown in Figure 7 does not distinguish between light of the first or second optical spectrums, two additional retardation plates are placed to handle the polarization rotation of the NIR light. Specifically, a retardation plate 195 is associated with at least a portion of light wave entrance face 194, and another retardation plate 197 is associated with light wave entrance face 172 and light wave entrance face 190 such that it is disposed between prisms 160 and 180. The retardation plates 195, 197 act as half wave plates for incident light of a first light spectrum (i.e., NIR light), thereby rotating the polarization of the incident NIR light, and act as full wave plates for incident light of a second light spectrum (i.e., photopic (visible) light), thereby leaving the polarization state of the incident photopic light unaffected.

[0068] Thus, a first polarization (typically p-polarized) component of the illumination 122 that impinges on the retardation plate 195 has its polarization rotated by the retardation plate 195 to a second orthogonal polarization (e.g., converting p-polarized light to s-polarized light), and a second polarization (typically s-polarized) component of the illumination 122 that impinges on the retardation plate 195 has its polarization rotated by the retardation plate 195 to the first orthogonal polarization (e.g., converting s-polarized light to p-polarized light). The illumination 122 (after passing through the retardation plate 195) enters the prism 180 through the light wave entrance face 194. The component of the illumination 122 that enters the prism as p-polarized light (with respect to the surface of the polarization-selective beam splitter arrangement 186) is transmitted by the beam splitter arrangement 186 and exits the prism 180 via surface 188. The component of the illumination 122 that enters the prism as s-polarized light (relative to the surface of the polarization-selective beamsplitter arrangement 186) is reflected by the beamsplitter arrangement 186 and reaches the collimation focusing surface 192, where it passes through the retardation plate 196, enters the collimation focusing component (i.e., lens) 140, and is reflected back through the retardation plate 196 by the reflective surface 141 of the lens 140 so as to rotate the polarization (e.g., convert s-polarized light to p-polarized light), resulting in the focused illumination being transmitted through the polarization-selective beamsplitter arrangement 186 and exiting the prism 180 via transmission through the light wave entrance surface 190. The light then strikes retardation plate 197 with a first polarization (typically p-polarized) and has its polarization rotated by retardation plate 197 to a second, orthogonal polarization (e.g., converting p-polarized light to s-polarized light) such that illumination 122 enters prism 160 through light wave entrance face 172 with s-polarized light relative to the surface of beam splitter arrangement 166. The s-polarized light is now reflected by beam splitter arrangement 166 towards light wave exit face 174 and exits prism 160 as a focused illumination beam 198 that strikes optical sensor 128.

[0069] It should be noted that since the retardation plates 195, 197 act as full wave plates for photopic light, the path of light passing through the prisms 160, 180 from the polarized light source 176 to the output of the prism 180 (light wave entrance face 194), as well as the polarization direction of the light passing through, is unaffected by the retardation plates 195, 197.

[0070] It should be noted that the configuration of the optical module 126 shown in FIG. 7 is applicable to situations where the lens 112 distinguishes between a first type of light, i.e., eye tracking light (light from the eye 110), and a second type of light, i.e., image light (light from the reflective display device 178), based at least in part on polarization separation. In configurations where the lens 112 distinguishes between these two types of light based only on spectral separation, the retardation plate 195 is not required. This is due to the fact that the eye may be illuminated such that the eye tracking light generally includes s- and p-polarized components that are both collimated by the lens 112 (because the lens 112 collimates light in the light spectrum occupied by the eye tracking light, e.g., light in the NIR region that is not related to polarization). Thus, the illumination 122 coupled out from the LOE 102 to the optical module 126 is collimated (with respect to the surface of the beam splitter arrangement 186) for s-polarized and p-polarized components. Here, the p-polarized light component enters prism 180 through surface 194, is transmitted by beam splitter arrangement 186, and exits prism 180 through surface 188. The s-polarized light component enters prism 180 through surface 194, is reflected by beam splitter arrangement 186, exits prism 180 through surface 192, and reaches collimation focusing surface 192, where it passes through retardation plate 196, enters collimation focusing component (i.e., lens) 140, and is reflected back through retardation plate 196 by reflective surface 141 of lens 140 so as to rotate the polarization (e.g., convert s-polarized light to p-polarized light), resulting in focused illumination transmitted through beam splitter arrangement 186 and exiting prism 180 via transmission through light wave entrance face 190.

[0071] Other implementations of the beam splitter configurations 166, 186 are contemplated herein including, for example, implementation of one or both of the beam splitter configurations 166, 186 of the optical module 126 as a simple 50-50 beam splitter that reflects approximately half the intensity of the incident light and transmits approximately half the intensity of the incident light. Alternatively, both beam splitter configurations can be implemented as a polarization selective beam splitter configuration for incident light of the second light spectrum (visible light) and as a simple 50-50 beam splitter for incident light of the first light spectrum (NIR light). For example, a beam splitter configuration can reflect s-polarized visible light, transmit p-polarized visible light, reflect approximately half the intensity of the incident NIR light, and transmit approximately half the intensity of the incident NIR light. Note, however, that in such a 50-50 beam splitter configuration, only approximately 25% of the intensity of the light that is initially incident reaches the output.

[0072] Various configurations of the illumination arrangement 138 are contemplated herein. In all illumination arrangement configurations, the illumination arrangement 138 includes one or more light sources configured to illuminate the eye 110 with a first type of light (i.e., light of a first light spectrum (e.g., NIR light) that includes a component of light polarized in a first polarization direction (e.g., p-polarized). Ideally, the light sources of the illumination arrangement 138 are positioned to illuminate the eye 110 in an illumination direction as close to normal to the EMB 109 as possible. In an alternative configuration, the light sources are positioned at the periphery of the field of view of the eye 110 to illuminate the eye 110 from the side. In yet another configuration, the illumination arrangement 138 is positioned as part of the optical module 126, and in addition to generating an image 142 and projecting it onto the LOE 102 for viewing by the eye 110, the illumination arrangement 138 can also be configured to inject light from the illumination arrangement 138 into the LOE 102 and propagate in a forward direction to be coupled out of the LOE 102 by a partially reflective surface 108 in a coupling-out direction normal to the EMB 109.

[0073] The following paragraphs describe some of the placement options for the lighting arrangement 138, with particular reference to FIG. 8. The non-limiting implementation of the device 100 shown in FIG. 8 is intended to provide context for general placement options for the lighting arrangement 138. In the particular non-limiting implementation shown here, the device 100 is implemented in an eyeglasses form factor with a head-mounted mechanism implemented as an eyeglass frame 154 with side arms 156 for hanging over the ears of a viewer. It should be noted that other form factors, such as helmet-mounted form factors, vehicle windshield form factors, and other head-up and near-eye display form factors, are also clearly within the scope of the present invention. The lighting arrangement 138 may include at least one light source 138A (represented as two light sources in FIG. 8) positioned near the active area of ​​the LOE 102 (e.g., via direct or indirect attachment to the surface 204) such that light rays emitted by the light source 138A reach the EMB 190 close to perpendicular to the EMB 109. Alternatively, or in addition to the light source 138A, the illumination arrangement 138 may include at least one other light source 138B positioned near the side of the observer's head (in FIG. 8, this is shown as being attached to the light coupling arrangement 124 attached to the side arm 156). In such a configuration, the light rays emitted by the polarized light source 138B reach the EMB 190 at an off-axis angle. As discussed, the light reflected from the eye 110 may include two orthogonal polarization components of light (i.e., an s-polarized component and a p-polarized component), and the lens 112 is configured to collimate only one of these two polarization directions. In the example described herein, the lens 112 is preferably configured to impart refraction to the p-polarized component of the light reflected from the eye 110 (so as to collimate the p-polarized light) and not to impart refraction to the s-polarized component of the light reflected from the eye 110.

[0074] The illumination arrangement 138 may be configured to illuminate a particular region of the eye 110 or the entire eye 110 with NIR light. As discussed in detail, the illumination reflected by the eye 110 (i.e., the first type of light represented by light rays 114A-114F) is collimated (by lens 112), coupled into the LOE 102 by the partially reflective surface 108, and then coupled out of the LOE 102 (by optical coupling arrangement 124) where it is focused (by lens 140) onto the optical sensor 128. The optical sensor 128 generates signals in response to sensing the focused light, and those signals are forwarded to a processing system 130 configured to process the signals and derive a current gaze direction of the eye 110. In a specific, non-limiting embodiment, the device 100 obtains the gaze direction (angular orientation of the eye 110, or line of sight of the eye 110) by imaging a pattern present in a particular region of the eye 110. The location of such a pattern and its movement indicate the current gaze direction and eye movement. The human eye contains various trackable features, including, for example, patterns produced by the blood vessels of the retina. These trackable features can be tracked using suitable tracking algorithms implemented by suitable image processing instructions executed by the processing system 130.

[0075] In a non-limiting process for deriving and tracking gaze direction, the retinal pattern is mapped and trackable features are determined during an initialization process, after which a continuous tracking process is performed. For example, an image marker may be displayed to the observer so that the observer can see it during initialization. While the observer is looking towards the marker, the illumination arrangement 138 fully illuminates the fundus (the visible part of the retina) with a short pulse and a complete image of the fundus obtained (via the optical sensor 128). This image is then processed by the processing system 130 to identify trackable features (e.g., the optic disc and the fovea). During the continuous tracking process, a selected region of interest (ROI) 110 of the eye is selectively illuminated by the illumination arrangement 138, and an image of the ROI (acquired by the optical sensor 128) is sampled and processed (by the processing system 130) during a corresponding illumination pulse to determine the current gaze direction (line of sight), and this derived gaze direction is used to update the position of the ROI for subsequent illumination cycles, and the continuous tracking process is repeated by illuminating the updated ROI. Assuming that the frequency of tracking measurements is high compared to the speed of eye movement, this update process is typically effective in maintaining continuous tracking, optionally combined with tracking information from the other eye. As gaze direction changes, the illuminated area also changes. The ROI update may be performed according to the "current" gaze direction determined from the last sampled image, or in some cases may use a predictive extrapolation based on eye movements between the previous two or more measurements. In case of tracking failure, the size of the illuminated area may be temporarily increased until trackable function is restored.

[0076] Returning to FIG. 8, the processing system 130 may be implemented using any suitable type of processing hardware and / or software known in the art, including, but not limited to, any combination of various dedicated graphic processors, display drivers, and computerized processors (collectively referred to as processor 132) operating under any suitable operating system and implementing suitable software or firmware modules. The storage medium 134 may be one or more computerized memory devices, such as volatile data storage. The processing system 130 may further include various communication components to enable wired or wireless communication with LAN and / or WAN devices for bidirectional transfer of information and graphical content. The device 100 is powered from a suitable power source, which may be any combination of a battery and / or an external power source provided, shown here diagrammatically as a power source 131 connected via a cable 133. When using a battery power source, the battery may be integrated as part of the structure attached to the glasses or helmet.

[0077] The optical components associated with the faces 104, 106 of the LOE 102, such as the lens 112 and the polarizer 136, are optically attached to the LOE 102 using any suitable attachment technique, including, for example, mechanical attachment to the LOE 102, while maintaining an air gap or material (e.g., gel) gap between the optical components and the faces of the LOE 102. The material occupying such an air gap or material gap has a sufficiently low refractive index to maintain a state of total internal reflection within the LOE 102. Other suitable alternatives for optically attaching the optical components to the LOE 102 include the placement of a microstructured air gap film or a transparent layer of low refractive index material (such as a thin plate of low refractive index material) between the faces of the LOE 102 and the optical components. Details of such optical attachment methods are described in commonly owned U.S. Pat. No. 10,520,731 and U.S. Patent Application Publication No. 2018 / 0067315. The quarter wave plate 150 may be attached to the lens 112 using similar optical attachment techniques.

[0078] Although the embodiments of the apparatus 100 described thus far relate to an optical coupling configuration implemented as a set of partially reflective surfaces 108 for coupling eye tracking light into the LOE 102 and image light (from the optical module 126) out of the LOE 102, the partially reflective surfaces 108 are merely illustrative of one non-limiting optical coupling configuration, and other optical coupling configurations can be used to couple eye tracking light into the LOE 102 and image light out of the LOE 102. The optical coupling configuration can be any optical coupling arrangement that deflects a portion of the eye tracking incident radiation from the lens 112 to an angle at which it propagates through internal reflection within the LOE 102, and similarly deflects a portion of the image incident radiation (from the optical module 126) already propagating within the LOE 102 by internal reflection to an angle at which the deflected portion of the image incident radiation exits the LOE 102. Other examples of such suitable optical coupling arrangements include, but are not limited to, one or more diffractive optical elements disposed on either of the faces 104, 106.

[0079] The device embodiments described with respect to Figures 2-8 involve the use of polarization and / or spectrum (wavelength) sensitive collimating elements (lenses 112) to collimate only certain components of the eye tracking light (light reflected from the observer's eye) so that the coupled-in collimated light can be focused onto an optical sensor integrated into the image projector (optics module 126). Other eye tracking solutions are contemplated herein in which uncollimated light from the eye is coupled into the light guide optics and directed to the optical sensor by a biconic lens with different radii of curvature for two orthogonal axes to image the eye and determine the angle at which the light leaves the eye. Such solutions employ specialized at least partially reflective surfaces, preferably located on a dedicated light guide optics separate from the LOE through which the projected image propagates.

[0080] 9-15, various aspects of the structure and operation of an apparatus, generally designated 200, constructed and operative in accordance with various embodiments of the present invention for displaying an image and deriving a gaze direction of a human eye 210 by a coupling-in arrangement associated with a light guide optical element (LOE) are shown. In a preferred but non-limiting implementation shown here, the coupling-in arrangement is implemented as a surface 208 that is at least partially reflective to light emanating from the eye 210. The surface 208 is hereinafter interchangeably referred to as an at least partially reflective surface 208. The surface 208 is associated with a first LOE 202 that is configured to propagate light reflected from the eye (in response to illumination by the illumination arrangement 242) and is separate from a second LOE 212 that is configured to propagate a projected image to be coupled out for viewing by the eye 210. The LOE 202 is formed from a transparent material and has a pair of parallel faces (planar major surfaces) 204, 206 for directing light by internal reflection (preferably total internal reflection). Surface 208 is configured to couple in a portion of light incident on face 204 into coupling-in region 243 for propagation within LOE 202 by (total) internal reflection. In particular, surface 208 is disposed at an angle with respect to parallel faces 204, 206 within LOE 202 (i.e., between faces 204, 206), such that the coupled-in light is trapped within substrate 202 by internal reflection from faces 204, 206. Coupling-in region 243 of LOE 202, also referred to as the "active region" or "active area," is a two-dimensional region of face 204.

[0081] Light coupled in by surface 208 propagates in the reverse direction through LOE 202 until it reaches coupling-out optical configuration 207 (schematically shown as a prism in FIG. 9 but which may also be implemented as a reflective surface, for example). A mixer 205, preferably implemented as a partially reflective surface, is disposed within LOE 202 on a plane upstream from coupling-out optical configuration 207 and parallel to adjacent faces 204, 206 (preferably the mid-plane between faces 204 and 206) to reduce non-uniformity of the light propagating through LOE 202. Light is coupled out from LOE 202 by coupling-out optical configuration 207 to optical module 236. Optical module 236 includes lens 240 (biconic lens) and optical sensor 238 (although sensor 238 may be external to optical module 236). The coupled-out light passes through a lens 240, which directs the light to an optical sensor 238 configured to sense light reflected from the eye 210. The processing system 230 includes at least one computerized processor 232 coupled to a storage medium 234 (such as a computer memory, etc.), is in electrical association with the optical sensor 238, and is configured to process signals from the optical sensor 238 to derive a current gaze direction of the eye 210.

[0082] A second LOE 212 configured to propagate a projected image to be coupled out for viewing by eye 210 is formed from a transparent material and has a pair of parallel faces (planar major surfaces) 214, 216 for directing light by internal reflection (preferably total internal reflection). The LOE 212 is deployed with one of the parallel faces 214 facing toward the eye 210, where the eye 210 is positioned within the EMB 209 at an eye relief (ER) distance 213 from the face 214. An image projector 228 is configured to project an image 222 (collimated to infinity), here as represented diagrammatically by an illumination beam 222 including sample rays 222A, 222B, and 222C spanning the beam. The projected image 222 is coupled into the LOE 212 by a coupling-in optical configuration 224, as shown here diagrammatically by a reflective surface (although other configurations, such as, for example, a prism, are contemplated herein), generating reflected light ray 223 that is trapped by internal reflection within the substrate, and also generating light ray 225. The image propagates along the substrate by repeated internal reflections and impinges on an optical coupling-out configuration associated with the second LOE 212, implemented as a sequence of partially reflective surfaces 218 at oblique angles to the parallel faces 214, 216, where a portion of the image intensity is reflected and coupled out of the substrate towards the eye 210 as light rays 226A, 226B, and 226C. It should be noted that the partially reflective surfaces 218 are merely illustrative of one non-limiting optical coupling-out configuration suitable for use with the LOE 212, and that other optical coupling configurations can be used to couple out the image light from the LOE 212. The optical coupling-out configuration may be any optical coupling configuration that deflects a portion of an image propagating within LOE 212 by internal reflection to an angle such that the deflected portion of the image exits LOE 212. Other examples of such suitable optical coupling arrangements include, but are not limited to, one or more diffractive optical elements disposed on either of faces 214, 216.

[0083] LOE 202 has one of its faces 204 parallel to face eye 210, while LOE 212 is interposed between eye 210 and LOE 202, and is positioned such that faces 204 and 216 are parallel (or nearly parallel), aligned, and adjacent to one another. Eye 210 is located an eye relief (ER) distance 211 from face 204. In the non-limiting configuration shown in the drawings, LOEs 202 and 212 are positioned such that LOEs 202 and 212 have a common elongation direction (arbitrarily shown herein as corresponding to the x-axis) and faces 204, 206, 214, 216 are mutually parallel. LOEs 202 and 212 are preferably optically attached to one another at faces 204, 216 to define an interface plane. Any suitable mechanism may be used to optically attach LOEs 202 and 212 to one another, including, but not limited to, mechanical arrangements and optical bonding. For example, the faces 204, 206 may be bonded together by providing a layer of optical bonding on at least a portion of at least one of the faces 204, 206 to form a bonded single optical structure formed from two light guides performing separate functions.

[0084] As in the embodiment described with reference to Figures 2-8, in this embodiment, eye 210 is preferably illuminated with light of a first light spectrum (preferably in the NIR region) such that light of the first light spectrum is coupled into LOE 202 by surface 208 and illumination 222 (i.e., the projected image) is in a second light spectrum (photopic, i.e., visible region). As discussed in the previous embodiment, partially reflective surface 218 is preferably coated to have a low reflectance for a first range of angles of incidence while having a desired partial reflectance for a second range of angles of incidence. Additionally, faces 214, 216 and partially reflective surface 218 are preferably coated to have a high transmittance for light of the first light spectrum such that light reflected by eye 210 passes through LOE 212 with minimal loss of intensity before being coupled into LOE 202.

[0085] In contrast to the incident light from the eye that is coupled into LOE 102 in the previous embodiments (FIGS. 2-8), in this embodiment the incident light from eye 210 that is coupled into LOE 202 is not collimated, resulting in an angular distribution of incident rays that impinge on surface 208 at different respective angles of incidence. With particular reference to FIG. 10, incident light from two different points within EMB 209 along a first dimension of EMB 209 (the first dimension is along the x-axis of an XYZ coordinate system, arbitrarily labeled in the drawings) impinges on surface 208 to be coupled into LOE 202. The incident light, here represented diagrammatically by first illumination beam 244 and second illumination beam 250, is light (preferably in the NIR range) that is reflected by eye 210 in response to illumination from illumination arrangement 242. It should be noted that the two beams 244, 250 merely represent a sample of the beams from EMB 209 that are coupled into LOE 202 by surface 208; additional beams from additional respective points within EMB 209 are also coupled into LOE 202 by surface 208. As can be seen, each of beams 244, 250 generally arrives at surface 208 at a different angle of incidence, such that beams 244, 250 generate respective reflected beams 245, 251 that are trapped within LOE 202 by internal reflection, but which propagate within LOE 202 at different angles relative to faces 204, 206.

[0086] 11, each of beams 244, 250 includes rays that span the beam. In the illustrated example, beam 244 includes sample rays 246A, 246B, and 246C that span beam 244 along at least a portion of coupling-in region 243 in the first dimension (along the x-axis), rays 246A and 246C being marginal rays of beam 244. Similarly, beam 250 includes sample rays 252A, 252B, and 252C along at least a portion of coupling-in region 243 in the first dimension (along the x-axis), rays 252A and 252C being marginal rays of beam 250. Each of light rays 246A, 246B, 246C, 252A, 252B, and 252C is incident on surface 204 at a different respective point on surface 204 along the first dimension of surface 204 (the x-axis in the drawing), and thus is incident on surface 208 at a different respective angle of incidence. Thus, each of light rays 246A, 246B, 246C, 252A, 252B, and 252C arrives at surface 208 at a different angle of incidence, such that reflected beams 245, 251 each include spaced apart reflected rays (spanning respective beams) that propagate within LOE 202.

[0087] The angular distribution of light (spanning beams 244, 250 and spanning rays that span beams 244, 250) coupled into LOE 202 by surface 208 is a function of the aperture width (width projected onto a plane parallel to EMB 209) of surface 208. The aperture width is inversely proportional to the steepness of the alignment angle β of surface 208 (measured relative to face 204), so that for steep alignment angles, the aperture width is effectively small, thereby providing high resolution in the angle span dimension (x-axis in the drawings). In this embodiment, surface 208 is aligned at a steeper angle than partially reflective surface 218, and at a steep enough angle such that the aperture width of surface 208 is sufficiently narrow relative to the distance between LOE 202 and EMB 209 such that light covering only a narrow angular distribution of angles is coupled into LOE 202 by surface 208.

[0088] Resolution can be roughly estimated by the width of surface 208 projected onto a plane parallel to face 204. In Figure 9, the width is shown as w and can be calculated as h / tan(β), where h is the thickness of LOE 202 (i.e., the minimum distance between flat faces 204, 206). For example, if h=1 mm and β=65°, then

number

[0089] 12, coupled out rays 247A, 247B, and 247C, as well as coupled out rays 253A, 253B, and 253C, are shown being received at optical module 236. Rays 247A, 247B, and 247C, as well as rays 253A, 253B, and 253C correspond to incident rays spanning an angular distribution in the first dimension (width) of surface 208. In particular, coupled out rays 247A, 247B, and 247C correspond to incident rays 246A, 246B, and 246C, and coupled out rays 253A, 253B, and 253C correspond to incident rays 252A, 252B, and 252C. The coupled out light beams 247A, 247B, and 247C, as well as the coupled out light beams 253A, 253B, and 253C, pass through lens 240, which imparts refraction to the light beams and directs them towards optical sensor 238.

[0090] As mentioned above, lens 240 is biconic, which in this context refers to having different radii of curvature for different axes. The different radii of curvature lead to lens 240 having two focal lengths in two respective dimensions (orthogonal dimensions), i.e., a first focal length of f1 in the first dimension and a second focal length of f2 in the second dimension (orthogonal to the first dimension). Optical sensor 238 is positioned at a distance f1 from lens 240. The first focal length f1 and the positioning of the optical sensor 238 at the first focal length f1 are such that the lens 240 converts the light rays 247A, 247B, and 247C (and the light rays 253A, 253B, and 253C) into non-converging beams of captured light that reach different respective regions of the optical sensor 238 such that the angular distribution of the light rays 247A, 247B, and 247C (and the light rays 253A, 253B, and 253C) indicates the angular distribution (in the dimension of the width of the surface 208) of the corresponding beams 244 and (beam 250). Additionally, rays 247A, 247B, and 247C, and rays 253A, 253B, and 253C arrive at different respective regions of optical sensor 238 such that rays 253A, 253B, and 253C at optical sensor 238 indicate the overall angular separation between the sets of rays 247A, 247B, and 247C, as well as the angular separation (in the dimension of the width of surface 208) between beams 244 and 250. Thus, optical sensor 238 can measure with reasonably high angular resolution the relative angles of light (beams 244, 250) emanating from EMB 209 that are coupled into LOE 202 by surface 208. Angular resolution is generally a function of the effective aperture width and eye relief (ER 211) of surface 208 (discussed above), and is given by sin -1 For an effective aperture width (w) of 0.47 mm and an eye relief (ER) of 27 mm, the angular resolution provided by optical sensor 238 is approximately 1 degree. Additionally, as a result of the small angular resolution, the parallelism requirements between the principal planes of LOE 202 are much more lenient than those of LOE 212 used to project an image onto eye 210, which may require approximately 1 arc minute of parallelism.

[0091] The lens 240 has a second focal length f2 in a dimension orthogonal to the first focal length dimension. The biconical aspect of the lens 240 allows imaging of the eye 210 via incident light (reflected from the eye 210) that spans the coupling-in region 243 along two orthogonal dimensions. Imaging (by the lens 240) by directing coupled-out light corresponding to incident light rays that span one dimension (along the x-axis) was discussed with reference to Figures 10-12. The following paragraphs describe imaging of the eye 210 by focusing by the lens 240 the coupled-out light rays corresponding to incident light rays that span a second dimension (along the z-axis) onto the optical sensor 238.

[0092] 13, beams 244, 250 also include rays that span the respective beams along at least a portion of the coupling-in region 243 in the second dimension (along the z-axis). In the illustrated example, beam 244 includes sample rays 248A, 248B, and 248C that originate from a common point on the EMB 209 and span beam 244 along at least a portion of the coupling-in region 243 in the second dimension (along the x-axis), rays 248A and 248C being marginal rays of beam 244. Similarly, beam 250 includes sample rays 254A, 254B, and 254C that originate from a common point on the EMB 209 and span beam 250 along at least a portion of the coupling-in region 243 in the second dimension (along the x-axis), rays 254A and 254C being marginal rays of beam 250. The projections of rays 246A, 246B, and 246C in the XY plane are incident on surface 208 at a common angle of incidence. Thus, when rays 246A, 246B, and 246C are coupled into LOE 202 by surface 208, they generate a set of spaced apart parallel rays that propagate through LOE 202. Similarly, the projections of rays 254A, 254B, and 254C in the XY plane are incident on surface 208 at a common angle of incidence. Thus, when rays 254A, 254B, and 254C are coupled into LOE 202 by surface 208, they generate a set of spaced apart parallel rays that propagate through LOE 202.

[0093] 14, coupled out rays 249A, 249B, and 249C, and coupled out rays 255A, 255B, and 255C are shown being received at optical module 236. Rays 249A, 249B, and 249C, and rays 255A, 255B, and 255C correspond to incident rays that span the first dimension (height along the z-axis) of surface 208. In particular, coupled out rays 249A, 249B, and 249C correspond to incident rays 248A, 248B, and 248C, and coupled out rays 255A, 255B, and 255C correspond to incident rays 254A, 254B, and 254C. The coupled-out rays 249A, 249B, and 249C, as well as the coupled-out rays 255A, 255B, and 255C, pass through lens 240, which imparts refraction to the rays to focus rays 249A, 249B, and 249C to different common areas (or spots) on the image plane of optical sensor 238, and to focus rays 255A, 255B, and 255C to different common spots on the image plane of optical sensor 238. In other words, lens 240 transforms the set of rays 249A, 249B, and 249C into a converging beam of captured light, and similarly transforms the set of rays 255A, 255B, and 255C into a converging beam of captured light. The ability to focus these sets of rays at the image plane is made possible by positioning lens 240 at the output aperture of LOE 202 and by designing lens 240 with an appropriate second focal length f2.

[0094] In general, lens 240 is preferably designed such that the second focal length f2 is given by f2=uf1 / (u-f1), where u is the in-plane distance that a ray of light travels along the second dimension from surface 208 to lens 240, and lens 240 can be given by u=ER+L2 / cos(θ), where L2 is the in-plane distance from surface 208 to coupling-out optical configuration 207, and θ is the angle (measured relative to surface 204) that the light propagates.

[0095] Similar to the embodiment described with reference to Figures 2-8, the optical sensor 238 in this embodiment generates signals in response to sensing light rays reaching the sensor, and these signals are forwarded to a processing system 130 configured to process the signals to derive a current gaze direction of the eye 110. The derivation of the gaze direction may be performed using steps similar to those described above with reference to Figures 2-8. Additionally, the ability of the optical sensor 238 to measure the relative angle of the incident light (beams 244, 250) may be used to enhance the derivation of the gaze direction in this embodiment.

[0096] Possible configurations of the lighting arrangement 242 are generally similar to the configurations of the lighting arrangement 138 described with reference to FIG. 8. For example, the device 200 may be implemented in an eyeglass form factor with a head-mounted mechanism implemented as an eyeglass frame with side arms for hanging over the ears of a viewer. Other form factors, such as helmet-mounted, vehicle windshield, and other head-up and near-eye display form factors, are also contemplated herein. The lighting arrangement 242 can include one or more NIR light sources, which can be positioned, for example, near the active area of ​​the LOE 202, for example, via direct or indirect attachment to the surface 214, such that light rays emitted by the light sources reach the EMB 209 near normal to the EMB 209. Alternatively, or in addition to the aforementioned configurations, the illumination arrangement 242 can include at least one other NIR light source positioned near the side of the observer's head, for example attached to the image projector 228 or the coupling-in optical arrangement 224 (which is preferably attached to one of the side arms of the eyeglass frame). In such a configuration, the light rays emitted by the NIR light source reach the EMB 209 at an off-axis angle.

[0097] Additionally, the LOE 212 may be used to illuminate the eye 210 in a direction perpendicular to the EMB 209. In such a configuration, the illumination arrangement 242 is integrated as part of the image projector 228, as shown in FIG. 15. The image projector 228 is generally similar to the image projector 126 shown in FIG. 5, except that the image projector 228 does not include an optical sensor (as the optical sensor 238 is located in the optical module 236, which is optically separate from the image projector 228). Additionally, since the optical sensor 238 is not part of the image projector 228, there is no need to design the beam splitter arrangements 166 and 186 with appropriate coatings for spectral selectivity. 15, both beamsplitter configurations 166 and 186 are polarization-selective beamsplitters that reflect incident light in a first polarization direction (e.g., s-polarized with respect to the surfaces of beamsplitter configurations 166, 186) and transmit incident light in a second polarization direction (e.g., p-polarized with respect to the surfaces of beamsplitter configurations 166, 186) for incident light of a first light spectrum and incident light of a second light spectrum (i.e., visible and NIR light are treated the same by beamsplitter configurations 166 and 186). Furthermore, since light only propagates in the forward direction through LOE 212 and thus light does not enter image projector 228 from LOE 212, surface 172 is light wave exit surface 172, surface 190 is light wave entrance surface 190, and surface 194 is light wave exit surface 194.

[0098] 5, the polarized light source 176 emits light of a polarization of a second light spectrum (i.e., in the visible range), represented diagrammatically as an incident beam 158. The polarized illumination 158 enters the prism 160 through the light wave entrance face 168 with a first polarization, typically s-polarized light relative to the surface of the polarization-selective beam splitter arrangement 166, and is reflected by the polarization-selective beam splitter arrangement 166 towards the image display surface 170, where it impinges on the reflective display device 178. Pixels corresponding to bright areas of the image are reflected with a modulated circular polarization (typically p-polarized), so that radiation from the bright pixels is transmitted through the beam splitter arrangement 166 and leaves the prism 160 via transmission through the light wave exit face 172. The light then enters the prism 180 through the light wave entrance face 190 with a second polarization (typically p-polarized light relative to the surface of the polarization-selective beam splitter arrangement 186) and reaches the collimation face 192, where it passes through the retardation plate 196, enters the lens 140, and is reflected back through the retardation plate 196 by the reflective surface 141 of the lens 140. The double pass through the retardation plate 196, aligned with its fast axis at 45 degrees to the polarization axis, rotates the polarization (e.g., converting p-polarized light to s-polarized light), so that the collimated image illumination is reflected at the polarization-selective beam splitter arrangement 186 towards the light wave exit face 194 and exits the prism 180 as illumination beam 222. The illumination beam 222 is then coupled into the LOE 212 by the coupling-in optical arrangement 224 (as shown in FIG. 9 ).

[0099] For example, illumination arrangement 242 implemented as a polarized NIR light source (which may be a combination of an NIR light source and a polarizer) is associated with surface 188, which in this configuration is light wave entrance surface 188. The polarized NIR light source is configured to emit light of a first light spectrum polarization (i.e., polarized NIR light), represented diagrammatically as incident beam 268. Polarized illumination 268 enters prism 180 through light wave entrance surface 188 with a first polarization, typically p-polarized light relative to the surface of polarization-selective beam splitter arrangement 188, transmits through polarization-selective beam splitter arrangement 186, and exits prism 180 as illumination beam 270 via transmission through light wave exit surface 194. Illumination beam 270 with p-polarization is then coupled into LOE 212 by coupling-in optical arrangement 224 (similar to illumination beam 222). P-polarized illumination 270 propagates through LOE 212 (similar to illumination 222) and is coupled out of LOE 212 by partially reflective surface 218. In this configuration, care must be taken to ensure that NIR illumination propagating within LOE 212 is coupled out by partially reflective surface 218, and that NIR light emanating from eye 210 is not coupled back into the LOE (in response to illumination by the coupled-out NIR light). To this end, partially reflective surface 218 is preferably coated to have a desired reflectivity for s-polarized light in the NIR region at a prescribed range of angles of incidence such that s-polarized NIR illumination propagating within LOE 212 is coupled out by partially reflective surface 218, but s-polarized NIR illumination emanating from eye 210 is incident on partially reflective surface 218 at angles of incidence outside of the prescribed range of angles of incidence and thus passes through the partially reflective surface without being reflected.

[0100] Although embodiments of device 200 have been described thus far in the context of LOEs 202 and 212 having a common (parallel) elongation direction, other embodiments are possible in which the LOEs have elongation directions that are orthogonal to one another. For example, LOE 212 may be arranged to have an elongation direction in the direction of the x-axis (as shown in FIG. 9 ), while LOE 202 may be arranged to have an elongation direction in the direction of the z-axis. Furthermore, it should be noted that embodiments of devices 100 and 200 have been described in the context of LOEs 102 and 212 being “one-dimensional waveguides” or “1D waveguides,” meaning that LOEs 102 and 212 each have a pair of parallel major surfaces (faces 104, 106 and faces 214, 216) that define a “slab-shaped waveguide” that directs image light (from image projectors 126, 228) to perform aperture expansion in one dimension. However, the eye tracking device according to the present embodiment is equally applicable to other waveguide structures, including structures in which an additional slab-type waveguide is coupled to each of the LOEs 102, 212 that directs image light in an orthogonal dimension to perform aperture expansion in the orthogonal dimension, producing an overall two-dimensional aperture expansion effect. Alternatively, one or both of the LOEs 102, 212 are "two-dimensional waveguides" or "2D waveguides," meaning that they have two mutually orthogonal pairs of major surfaces that serve to direct image light (from the image projector 126, 228) in two dimensions as it propagates along the LOE to perform aperture expansion using a single waveguide.

[0101] Although the embodiments of the present disclosure have been described in the context of a lighting arrangement arranged to illuminate the eye with light in the near infrared region of the electromagnetic spectrum, the embodiments of the present disclosure should not be limited to lighting arrangements that emit eye tracking light in a particular region of the electromagnetic spectrum. The description of the use of NIR light for eye tracking purposes is intended, for example, to provide a clearer description of the structure and operation of the various devices of the present disclosure. Other types of light may also be used for eye tracking purposes, including, but not limited to, light in the infrared region, and ultraviolet light emitted at low intensity and short pulse duration.

[0102] According to certain non-limiting embodiments, various eye tracking devices of the present disclosure may be replicated to simultaneously track both eyes of a subject, as well as to project images to both eyes. For example, device 100 and / or device 200 may be replicated for both eyes. Combining data from two eye trackers may allow for improved stability and continuity of tracking. For example, while the eye is moving, a trackable portion of the eye may be visible to the tracker in one eye and not in the other. When using tracking algorithms that employ tracking of trackable features, simultaneous tracking of both eyes allows for continuous maintenance of tracking throughout periods when only one eye tracker may be able to track blind spots.

[0103] If the device is binocular, each eye has its own image projection and eye tracking device, and various processing and power components may optionally be shared between the two eye tracking systems. Eye tracking information collected by the binocular eye tracking devices may be fused to enhance tracking stability and continuity, as described above.

[0104] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations that do not depart from the scope and spirit of the described embodiments will be apparent to those skilled in the art. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to allow persons other than those skilled in the art to understand the embodiments disclosed herein.

[0105] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0106] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0107] It will be understood that certain features of the invention that are described, for clarity, in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, for brevity, various features of the invention that are described in the context of a single embodiment may also be provided separately or in any suitable subcombination, or as appropriate with any other described embodiment of the invention. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

[0108] To the extent that the appended claims have been drafted without multiple dependencies, this has been done solely to accommodate the formal requirements of jurisdictions that do not permit such multiple dependencies. It should be noted that all possible combinations of features that would be implied by making the claims multiple dependent are expressly contemplated and should be considered part of the present invention.

[0109] While the present invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

Claims

1. An apparatus comprising: an optically transparent substrate having at least two parallel major surfaces for directing light by internal reflection, a first of said major surfaces positioned facing the eye; an optical element associated with the first of the major surfaces, the optical element configured to refract incident light of a first type to collimate the incident light of the first type and to refract incident light of a second type in accordance with at least one characteristic of the incident light, the optical element configured to substantially not refract incident light of a second type; an optical coupling arrangement associated with the substrate; coupling in a portion of the first type of light collimated by the optical element and incident on the first of the major surfaces so as to propagate within the substrate; and coupling out a portion of the second type of light propagating in the substrate. an optical system associated with the substrate and configured to convert the first type of collimated light into a converging beam of captured light; an optical sensor positioned to sense the captured light; at least one processor electrically associated with the optical sensor and configured to process signals from the optical sensor to derive a current gaze direction of the eye.

2. The apparatus of claim 1 , wherein the at least one characteristic of the incident light includes a polarization direction of the incident light.

3. The apparatus of claim 1 , wherein the at least one characteristic of the incident light comprises a region of the electromagnetic spectrum occupied by the incident light.

4. The apparatus of claim 1 , wherein the at least one characteristic of the incident light includes a polarization direction of the incident light and a region of the electromagnetic spectrum occupied by the incident light.

5. 2. The apparatus of claim 1, wherein the light of the first type includes a component of light polarized in a first polarization direction and the light of the second type is polarized in a second polarization direction.

6. 2. The apparatus of claim 1, wherein the first type of light is in a first light spectrum and the second type of light is in a second light spectrum.

7. 2. The apparatus of claim 1, wherein the first type of light includes components of light polarized in a first polarization direction and is within a first optical spectrum, and the second type of light is polarized in a second polarization direction and is within a second optical spectrum.

8. The apparatus of claim 1 , further comprising a polarizer associated with a second of the major surfaces of the substrate.

9. 2. The apparatus of claim 1, wherein the substrate is positioned with the first of the major surfaces at an eye relief distance from the eye, and the optical element has a focal length approximately equal to the eye relief distance.

10. a second optical coupling arrangement associated with the optical system; coupling out a portion of the first type of light propagating within the substrate such that the coupled out light is accepted by the optical system; 13. The apparatus of claim 1, further comprising: a second light coupling arrangement configured to: couple in a portion of the second type of light from a display source to propagate within the substrate by internal reflection.

11. The apparatus of claim 1 , further comprising an illumination arrangement arranged to illuminate the eye with the first type of light.

12. 2. The apparatus of claim 1, further comprising an image projector coupled to the substrate to introduce the second type of collimated light corresponding to an image into the substrate such that the second type of coupled-in collimated light propagates by internal reflection within the substrate and is coupled out of the substrate towards the eye by the light coupling arrangement.

13. 13. The apparatus of claim 12, wherein the image projector includes a reflective display device that generates polarized light in response to illumination from a polarized light source, the polarized light generated by the reflective display device being collimated by the optical system.

14. The apparatus of claim 1 , wherein the optical coupling arrangement comprises a plurality of partially reflective surfaces disposed within the substrate at an angle relative to the major surface of the substrate.

15. 2. The apparatus of claim 1, wherein the first type of light propagates in a first propagation direction within the substrate and the second type of light propagates in a second propagation direction within the substrate that is opposite to the first propagation direction.

16. An apparatus comprising: an optically transparent substrate having a pair of parallel major surfaces for directing light by internal reflection, a first of said major surfaces positioned opposite a viewer's eye; a lens associated with the first of the major surfaces, refracting the incident light of a first type to collimate the incident light, the incident light of the first type including a component of light within a first optical spectrum and having a polarization in a first polarization direction; a lens configured to substantially not refract incident light of a second type, the incident light of the second type being within a second optical spectrum and having a polarization in a second polarization direction; an illumination arrangement arranged to illuminate the eye with light of the first type such that a portion of the light of the first type is reflected by the eye back towards the lens to be collimated by the lens; An optical module, comprising: a reflective display device that generates the second type of light corresponding to an image in response to illumination from a light source; an optical system configured to collimate the light generated by the reflective display device to produce the second type of collimated light; an optical module including an optical sensor; an optical coupling arrangement configured to couple the collimated light of the second type into the substrate for propagation within the substrate by internal reflection in a first propagation direction; a plurality of partially reflective surfaces disposed within the substrate at an angle relative to the major surface of the substrate, coupling out a portion of the second type of light propagating in the substrate in the first propagation direction; a plurality of partially reflective surfaces configured to couple in a portion of the collimated light of the first type incident on the first of the major surfaces to propagate within the substrate in a second propagation direction, the optical coupling arrangement being further configured to couple out the propagating light of the first type; at least one processor electrically coupled to the optical sensor; The device, wherein the optical system of the optical module is further configured to accept the light coupled out by the optical coupling arrangement and to convert the coupled out light into a convergent beam of captured light, the optical sensor is configured to sense the captured light, and the at least one processor is configured to process signals from the optical sensor to derive a current gaze direction of the eye.

17. An apparatus comprising: a first light-transmitting substrate having at least two substantially parallel major surfaces for directing light by internal reflection, a first of said major surfaces positioned opposite a viewer's eye; an at least partially reflective surface disposed within the first substrate at an angle to the major surfaces, the at least partially reflective surface being configured to couple in incident light rays incident on the first of the major surfaces within a coupling-in region to propagate within the first substrate by internal reflection, the incident light rays being in a first light spectrum and emanating from the eye in response to illumination of the eye, the incident light rays comprising at least a first set of light rays and a second set of light rays, the first set of light rays having an angular distribution that spans at least a portion of the coupling-in region in a first dimension and the second set of light rays spanning at least a portion of the coupling-in region in a second dimension; a coupling-out arrangement configured to couple out the light beam propagating in the first substrate; An optical module, comprising: at least one lens having a first focal length in a first dimension of the lens and a second focal length in a second dimension of the lens; converting the coupled-out light beams corresponding to the first set of light beams into a non-converging beam of captured light having an angular distribution indicative of the angular distribution of the first set of light beams; and converting the coupled-out light beams corresponding to the second set of light beams into a converging beam of captured light. an optical sensor disposed at a distance from the lens substantially equal to the first focal length and configured to sense the captured light; at least one processor electrically coupled to the optical sensor configured to process signals from the optical sensor to derive a current gaze direction of the eye.

18. 20. The apparatus of claim 17, further comprising an illumination arrangement arranged to illuminate the eye with light of the first light spectrum.

19. a second optically transparent substrate having a plurality of surfaces including first and second mutually parallel major surfaces for directing light by internal reflection, the first major surface of the second substrate being positioned opposite the eye and the second major surface of the second substrate being opposite the first of the major surfaces of the first substrate; 18. The apparatus of claim 17, further comprising a coupling-out arrangement associated with the second substrate, the coupling-out arrangement being configured to couple out a portion of light of a second spectrum of light, different from the first spectrum of light and propagating within the second substrate, from the second substrate towards the eye.

20. 20. The apparatus of claim 19, further comprising an image projector coupled to the second substrate and configured to generate collimated light in the second light spectrum corresponding to an image, the collimated light propagating by internal reflection within the second substrate and coupled out from the second substrate towards the eye by the coupling-out arrangement.

21. 21. The apparatus of claim 20, further comprising a coupling-in arrangement associated with the image projector and with the second substrate configured to couple the collimated light generated by the image projector into the second substrate.

22. 20. The apparatus of claim 19, wherein the coupling-out arrangement comprises a plurality of partially reflective surfaces disposed in the second substrate at an angle to the major surface of the second substrate.

23. The apparatus of claim 19 , wherein the coupling-out arrangement comprises a diffractive optical element associated with one of the major surfaces of the second substrate.

Images