Optical system and method for eye tracking based on imaging the eye via a collimating element and a light guiding optical element

Abstract

A light-transmissive substrate is disclosed that is arranged with a first major surface of two major surfaces in a facing relationship with an eye of a viewer and directs light by internal reflection between the two major surfaces. An optical out-coupling configuration out-couples image light corresponding to a collimated image and directed by internal reflection between the two major surfaces from the light-transmissive substrate. A first optical in-coupling configuration collimates light from the eye to produce collimated light and in-couples the collimated light into the light-transmissive substrate for direction by internal reflection. A second optical in-coupling configuration in-couples collimated light from the light-transmissive substrate toward an optical sensor that senses the in-coupled light. A processing system derives a current gaze direction of the eye by processing signals from the optical sensor.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 126,551, filed December 17, 2020, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This invention relates to eye tracking. Background Technology

[0004] Optical arrangements for near-eye displays (NEDs), head-mounted displays (HMDs), and head-up displays (HUDs) require large apertures to cover the two-dimensional area where the viewer's eyes reside (often referred to as the eye movement box or EMB). To achieve compact devices, the image to be projected onto the viewer's eyes is generated by a small optical image generator (projector) with a small aperture, which is multiplied to create a large aperture.

[0005] A method for aperture multiplication in one dimension has been developed based on parallel-faced plates made of transparent material, in which images propagate via internal reflection. A portion of the image wavefront is coupled out of the plate by using angled partial reflectors or by using diffractive optics on one surface of the plate. Such a plate is referred to herein as a light-guide optical element (LOE), a transparent substrate, or an optical waveguide. Figure 1 The diagram illustrates the principle of this aperture multiplication. Figure 1 A light-guiding optical element 20 for guiding light by internal reflection is shown, having a pair of parallel surfaces 26, 26A. A projected image 18, schematically represented here by an illumination beam 18, is coupled into the light-guiding optical element 20 (schematically represented here by a first reflective surface 16) to generate reflected light ray 28, which is trapped within the substrate by internal reflection, thereby also generating light ray 30. The illumination beam 18 includes sample light ray 18A and sample light ray 18B spanning the beam. The image propagates along the substrate by repeated internal reflections, incident at an angle (α) relative to the parallel surfaces 26, 26A. surOn a series of partially reflective surfaces 22, a portion of the image intensity is reflected so that it is coupled out of the substrate as light rays 32A and 32B toward the viewer's eye 24. To minimize unwanted reflections that may cause ghosting, the partially reflective surfaces 22 are preferably coated to have low reflectivity for a first angle range and desired partial reflectivity for a second angle range, wherein the reflective surface 22 has a small angle of inclination (denoted here as β) with respect to the normal to the partially reflective surface 22. ref The light rays are separated to generate reflected rays for coupling out, while rays with a high tilt angle (relative to the normal) are transmitted with negligible reflection.

[0006] The projected image 18 is a collimated image, meaning that each pixel is represented by a parallel beam of light at a corresponding angle, equivalent to light from a scene far from the viewer (the collimated image is referred to as "collimated to infinity"). Here, the image is simply represented by rays of light corresponding to a single point in the image, typically the centroid of the image, but actually including angular ranges to each side of that central beam, which couples into the substrate at corresponding angular ranges and similarly couples out at corresponding angular ranges, thereby creating a field of view corresponding to the portions of the image reaching the viewer's eye 24 in different directions.

[0007] An optical function that can be used in NED, HMD, or HUD designs is eye tracking, or sensing the direction in which a viewer's eyes are looking relative to their head (often referred to as gaze direction). Past eye-tracking methods relied on imaging the EMB via one or more off-axis cameras viewing from the side. To reduce user discomfort, the cameras should be relatively small, which can limit EMB imaging performance. The small camera size, and the general difficulty in obtaining the gaze direction from EMB images sampled at high off-axis angles, resulted in the relatively low performance of such eye-tracking methods. Summary of the Invention

[0008] Aspects of the present invention provide an eye tracker and a corresponding method for tracking the gaze direction of a human eye based on imaging the eye via a light-guiding optical element, and are particularly suitable for integration as part of a NED, HMD, or HUD, especially when used as part of an augmented reality (AR) or virtual reality (VR) system.

[0009] According to the teachings of embodiments of the present invention, an optical system is provided. The optical system includes: a light-transmitting substrate having at least two main surfaces, the at least two main surfaces being arranged such that a first main surface of the main surfaces faces the viewer's eye for guiding light via internal reflection between the two main surfaces of the light-transmitting substrate; an optical coupling configuration associated with the light-transmitting substrate for coupling image light, corresponding to a collimated image, guided by internal reflection between the two main surfaces, out of the light-transmitting substrate; a first optical coupling configuration associated with the light-transmitting substrate, the first optical coupling configuration being configured to: collimate light from the eye to generate collimated light, and couple the collimated light into the light-transmitting substrate for propagation within the light-transmitting substrate via internal reflection; a second optical coupling configuration associated with the light-transmitting substrate, the second optical coupling configuration being configured to couple the collimated light out of the light-transmitting substrate as coupled-out light; an optical sensor arranged for sensing the coupled-out light; and at least one processor communicating with the optical sensor and configured to process signals from the optical sensor to obtain the current gaze direction of the eye.

[0010] Optionally, the optical coupling configuration includes a plurality of partially reflective surfaces, which are arranged obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate.

[0011] Optionally, the optical coupling configuration includes a diffraction element associated with one of the two main surfaces of the light-transmitting substrate.

[0012] Optionally, the light from the eye is in the first spectrum, and the image light is in the second spectrum.

[0013] Optionally, the optical system further includes an image projector for generating the collimated image.

[0014] Optionally, the second optical coupling configuration is further configured to couple image light corresponding to the collimated image into the light-transmitting substrate so that it can propagate within the light-transmitting substrate via internal reflection.

[0015] Optionally, the optical system further includes a selectively reflective surface that transmits or reflects image light corresponding to the collimated image toward the second optical coupling configuration, and reflects or transmits coupled light from the second optical coupling configuration toward the optical sensor.

[0016] Optionally, the optical system further includes an optical coupling configuration associated with the light-transmitting substrate for coupling image light corresponding to the collimated image into the light-transmitting substrate so as to propagate within the light-transmitting substrate by internal reflection.

[0017] Optionally, the optical coupling configuration reflects the image light corresponding to the collimated image and transmits the collimated light propagating within the light-transmitting substrate toward the second optical coupling configuration.

[0018] Optionally, the optical coupling configuration is coupled into the light-transmitting substrate so that the image light corresponding to the collimated image propagating within the light-transmitting substrate reaches the second optical coupling configuration via internal reflection, and the second optical coupling configuration transmits the image light corresponding to the collimated image propagating within the light-transmitting substrate.

[0019] Optionally, the optical system further includes: an optical element arranged in an optical path from the second optical coupling configuration to the optical sensor for forming at least one image of at least a portion of the eye on the optical sensor.

[0020] Optionally, the optical system further includes an image projector, which includes a spatial light modulator for generating image light, and the optical element forms part of the image projector and collimates the image light generated by the spatial light modulator to generate a collimated image.

[0021] Optionally, the first optical coupling configuration reflects light from the eye and transmits image light corresponding to the collimated image.

[0022] Optionally, the first optical coupling configuration includes a curved surface having a curvature sufficient to align light from the eye up to infinity.

[0023] Optionally, the curvature is a function of the distance between the eye and the first main surface of the main surface of the light-transmitting substrate.

[0024] Optionally, the first optical coupling configuration is arranged obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate.

[0025] Optionally, the optical system further includes: a second light-transmitting substrate having at least two main surfaces, one of the two main surfaces of the second light-transmitting substrate being associated with another of the two main surfaces of the light-transmitting substrate, and at least one optical element of the first optical coupling configuration being disposed within the second light-transmitting substrate.

[0026] Optionally, at least one optical element of the first optical coupling configuration is arranged in the region of the second light-transmitting substrate located in front of the eye, such that the normal of the at least one optical element of the first optical coupling configuration approximately reaches the center of the pupil of the eye.

[0027] Optionally, the two main surfaces of the light-transmitting substrate are parallel to each other, and the two main surfaces of the second light-transmitting substrate are parallel to each other and parallel to the two main surfaces of the light-transmitting substrate.

[0028] Optionally, at least one of the main surfaces of the second light-transmitting substrate is a curved surface.

[0029] Optionally, the second light-transmitting substrate is formed as a lens for applying optical power to light from a real-world scene.

[0030] Optionally, the optical system further includes: a second light-transmitting substrate having at least two main surfaces including a first main surface and a second main surface, the first main surface of the second light-transmitting substrate being associated with the second main surface of the light-transmitting substrate, and the first optical coupling configuration including: at least one collimating element disposed within the second light-transmitting substrate, and a partial reflector disposed obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate, the partial reflector: transmitting light from the eye toward the at least one collimating element such that the at least one collimating element generates collimated light according to the light from the eye, and reflecting the collimated light generated by the at least one collimating element to couple the collimated light into the light-transmitting substrate so as to propagate within the light-transmitting substrate by internal reflection.

[0031] Optionally, the optical system further includes: a second light-transmitting substrate having at least two main surfaces including a first main surface and a second main surface, the second main surface of the second light-transmitting substrate being associated with the first main surface of the light-transmitting substrate, and the first optical coupling configuration including: at least one collimating element disposed within the second light-transmitting substrate for collimating light from the eye to generate collimated light; a first reflector disposed obliquely within the second light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate, the first reflector deflecting the collimated light out of the second light-transmitting substrate and into the light-transmitting substrate; and a second reflector disposed obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate, the second reflector deflecting light from the first reflector to couple the collimated light into the light-transmitting substrate for propagation within the light-transmitting substrate by internal reflection.

[0032] Optionally, the optical system further includes an illumination arrangement configured to illuminate the eye with illumination light such that the eye reflects the illumination light as reflected light corresponding to light from the eye collimated by the first optical coupling configuration.

[0033] Optionally, the second optical coupling configuration includes a reflective surface that deflects collimated light out of the light-transmitting substrate.

[0034] Optionally, the second coupling configuration includes a planar opening end of the light-transmitting substrate, which is formed by cutting the light-transmitting substrate along a plane orthogonal to the two main surfaces of the light-transmitting substrate.

[0035] Optionally, the at least one processor is configured to receive signals from the optical sensor via one or more communication networks.

[0036] According to an embodiment of the teachings of the present invention, an optical system is also provided. The optical system includes: a light-guide optical element (LOE) having at least two main surfaces, the at least two main surfaces being arranged such that a first main surface of the main surfaces faces the viewer's eye for guiding light through internal reflection between the two main surfaces of the LOE; a plurality of partially reflective surfaces, the plurality of partially reflective surfaces being obliquely arranged within the LOE relative to the two main surfaces of the LOE for coupling image light, guided by internal reflection between the two main surfaces and corresponding to a collimated image, out of the LOE; and an optical element arranged within the LOE, the optical element being selectively reflective and selectively... An optical power is applied to the incident light such that the optical element: collimates the light reflected from the eye to produce collimated light, and reflects the collimated light to couple it into the LOE for guidance via internal reflection between the two main surfaces, and transmits image light guided by internal reflection between the two main surfaces, corresponding to the collimated image, and substantially does not apply optical power to the image light; an imaging system; and an optical coupling configuration associated with the LOE, the optical coupling configuration being configured to couple the collimated light from the LOE to the imaging system, the imaging system forming an image of the eye based on the collimated light coupled from the LOE via the optical coupling configuration.

[0037] Optionally, the imaging system includes an optical sensor for sensing collimated light coupled out of the LOE via the optical coupling configuration, and the optical system further includes a processing system that communicates with the optical sensor and is configured to process signals from the optical sensor to obtain the current gaze direction of the eye.

[0038] According to an embodiment of the teachings of the present invention, an optical system is also provided. The optical system includes: a light-transmitting substrate having at least two main surfaces, the at least two main surfaces being arranged such that a first main surface of the main surfaces faces the eye of a viewer for guiding light via internal reflection between the two main surfaces of the light-transmitting substrate; a first optical coupling configuration associated with the light-transmitting substrate, the first optical coupling configuration being operable to: collimate reflected light from the eye to generate collimated light, and couple the collimated light into the light-transmitting substrate to propagate within the light-transmitting substrate via internal reflection; a second optical coupling configuration associated with the light-transmitting substrate, the second optical coupling configuration being configured to couple the collimated light from the light-transmitting substrate as coupled-out light; a focusing optics device associated with the second optical coupling configuration and operable to convert the coupled-out light into a converging beam of capture light; an optical sensor arranged to sense the capture light; and at least one processor communicating with the optical sensor and configured to process signals from the optical sensor to obtain the current gaze direction of the eye.

[0039] In the context of this document, the term "guiding" generally refers to light trapped within a light-transmitting material (e.g., a substrate) by internal reflection at the main outer surface of the light-transmitting material, such that the trapped light propagates through the material in its propagation direction. When propagating light is incident on the main outer surface of the light-transmitting material at an angle of incidence within a given angular range, the light propagating within the light-transmitting substrate is trapped by internal reflection. The internal reflection of the trapped light can be in the form of total internal reflection, whereby propagating light incident on the main outer surface of the light-transmitting material at an angle greater than a critical angle (defined in part by the refractive index of the light-transmitting material and the refractive index of the medium in which the light-transmitting material is disposed, such as air) undergoes internal reflection at the main outer surface. Alternatively, the internal reflection of the trapped light can be achieved by a coating, such as an angle-selective reflective coating, applied to the main outer surface of the light-transmitting material to achieve reflection of light incident on the main outer surface within a given angular range.

[0040] Eye trackers according to various embodiments of the invention rely on the collimation and deflection of light reflected from the eye toward an optical sensor via an optical coupling configuration. The light reflected from the eye is also referred to herein as eye-tracking light. This eye-tracking light is within a specific spectrum, also referred to herein as being within the “eye-tracking spectrum.” Eye trackers according to various embodiments of the invention are particularly effective when the eye-tracking spectrum is in the near-infrared (NIR) region of the electromagnetic spectrum (i.e., when the eye-tracking spectrum is in the NIR region). In the context of this document, light in the NIR region of the electromagnetic spectrum generally refers to light having wavelengths in the range of 700 nanometers (nm) to 1400 nm, and in some cases 680 nm to 1400 nm. Wavelengths near 700 nm, i.e., in the range of 680 nm to 750 nm, can erode darker red visible light, but can be particularly advantageous when irradiating the eye for eye-tracking purposes. In the context of this document, unless otherwise explicitly stated, light described as having wavelengths in the NIR region generally refers to light having wavelengths in the range of 700 nm to 1400 nm or 680 nm to 1400 nm. In the context of this article, unless otherwise explicitly stated, light described as having wavelengths outside the NIR region generally refers to light having wavelengths less than 700 nm (or less than 680 nm) or greater than 1400 nm.

[0041] Although eye trackers according to various embodiments of the invention are particularly effective when the eye-tracking spectrum is in the NIR region, eye trackers are also effective when the eye-tracking light is outside the NIR region—including, for example, the infrared (IR) region, the ultraviolet (UV) region, and in some cases outside the visible region of the electromagnetic spectrum (i.e., when the eye-tracking spectrum is in the visible light region) (as will be discussed).

[0042] In the context of this paper, light in the visible region of the electromagnetic spectrum generally refers to light with wavelengths in the range of 380 nm to 750 nm. Therefore, there may be some overlap between the NIR region and the visible region. In the context of this paper, unless otherwise explicitly stated, light described as having wavelengths primarily in the visible region generally refers to light with wavelengths in the range of 380 nm to 700 nm or 380 nm to 680 nm. In the context of this paper, unless otherwise explicitly stated, light described as having wavelengths primarily outside the visible region generally refers to light with wavelengths less than 380 nm or greater than 700 nm (or greater than 680 nm). The visible region is interchangeably referred to herein as the "visible region," "photopic region," and "photopic spectrum."

[0043] Unless otherwise defined herein, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. While methods and materials similar to or equivalent to those described herein may be used in the practice or testing of embodiments of the invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including its definitions, shall prevail. Furthermore, materials, methods, and examples are illustrative only and are not intended to be necessary limiting. Attached Figure Description

[0044] This document describes some embodiments of the invention by way of example only, with reference to the accompanying drawings. It is emphasized that, by referring specifically to the drawings, the details shown are illustrative and for the purpose of discussing embodiments of the invention in an illustrative manner. In this regard, how embodiments of the invention can be practiced will become apparent to those skilled in the art from the description taken in conjunction with the drawings.

[0045] Now turn your attention to the accompanying drawings, where the same reference numerals or characters indicate corresponding or identical parts. In the drawings:

[0046] Figure 1 This is a schematic side view of the prior art light-guiding optical element with a partially reflective surface used in near-eye displays.

[0047] Figure 2A This is a schematic side view of an optical system constructed and operated according to an embodiment of the present invention, which is used to display images and to track the gaze direction of a human eye. The optical system has a first optical coupling configuration and a second optical coupling configuration. The first optical coupling configuration is used to collimate and couple light from the eye into a light-transmitting substrate, and the second optical coupling configuration is used to couple light from the eye from the light-transmitting substrate to an optical sensor via optical devices. The propagation of light from the eye through the light-transmitting substrate is also shown.

[0048] Figure 2B It corresponds to Figure 2A A schematic isometric view;

[0049] Figure 3 yes Figure 2A and Figure 2B A schematic side view of the optical system, showing the propagation of image light generated by an image projector through a light-transmitting substrate;

[0050] Figure 4 It is achieved using the shape factor of eyeglasses. Figures 2A to 3 A partial schematic isometric view of the optical system;

[0051] Figure 5This is a schematic side view of an optical system constructed and operated according to an embodiment of the present invention, which is similar to... Figures 2A to 3 The optical system, but wherein the collimating element is arranged in the second light-transmitting substrate, and wherein the optical coupling configuration includes a portion of the reflector arranged in the light-transmitting substrate;

[0052] Figure 6 This is a schematic side view of an optical system constructed and operated according to an embodiment of the present invention, which is similar to... Figure 5 The optical system, but wherein the optical coupling configuration includes a collimating element and two reflectors or partial reflectors, and wherein the collimating element and one of the two reflectors are arranged in a second light-transmitting substrate, and wherein the other of the two reflectors is arranged in the light-transmitting substrate;

[0053] Figure 7 This is a schematic side view of an optical system constructed and operated according to an embodiment of the present invention, which is similar to... Figure 2A The optical system, but in which the second optical coupling configuration is implemented as a planar opening end of the light-transmitting substrate;

[0054] Figure 8 This is a schematic side view of an optical system constructed and operated according to an embodiment of the present invention, which is similar to... Figures 2A to 3 The optical system disclosed includes an image projector apparatus for generating collimated image light coupled into a light-transmitting substrate, an optical coupling configuration for coupling the collimated image light out of the light-transmitting substrate, and an imaging system for imaging the light coupled out of the light-transmitting substrate from the eye; and

[0055] Figure 9 This is a schematic side view of an optical system constructed and operated according to an embodiment of the present invention, which is similar to... Figure 8 The optical system, but the imaging system and image projector components are integrated into a single module. Detailed Implementation

[0056] Embodiments of the present invention provide an optical system and a corresponding method for tracking the gaze direction of a human eye based on imaging the eye via a light-guiding optical element.

[0057] The principles and operation of the optical system and method according to the present invention can be better understood by referring to the accompanying drawings.

[0058] Before explaining at least one embodiment of the present invention in detail, it should be understood that the invention is not necessarily limited to the details of the construction and arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples. The invention can have other embodiments or can be practiced or performed in various ways.

[0059] As introduced, providing an eye-tracking device for determining a user's gaze direction is useful in many applications, especially in the context of head-up displays or near-eye displays. A common method for performing eye tracking is to sample an image of the eye, typically for the purpose of determining the position of the pupil within the image, and thereby deriving the eye's orientation. This is based on... Figure 1 Using optical elements that operate on a similar principle to sample images for eye tracking would be particularly advantageous.

[0060] This document describes an eye-tracking solution employing a light-guided optics element operating according to such a principle or a similar principle. According to certain aspects of the invention, the eye is imaged by coupling light reflected from the eye into an optical coupling configuration (“first optical coupling configuration”). Since the eye is not located at infinity from the light-guided optics element (but rather at an exit pupil distance typically on the order of about 20 millimeters), the first optical coupling configuration also collimates the light reflected from the eye, such that the eye-tracking light coupled into the light-guided optics element through the first optical coupling configuration is also collimated. This collimated light propagates through the light-guided optics element along an opposite path in a propagation direction substantially opposite to the propagation direction of any image light (from an image projector) propagating through the light-guided optics element. The collimated light from the eye is then coupled out of the light-guided optics element through another optical coupling configuration (“second optical coupling configuration”) and focused onto an optical sensor by a focusing optics device. In response to sensing the coupled light, a signal generated by the optical sensor is processed by a processing system to determine the gaze direction.

[0061] Now for joint reference Figures 2A to 8 This illustration shows various aspects of the structure and operation of an optical system (generally referred to as optical system 100) for obtaining the gaze direction of a human eye 110, constructed and operated according to various embodiments of the present invention. Generally, optical system 100 includes a light-transmitting substrate (interchangeably referred to as a light-guiding optical element or LOE) 102, and means associated with LOE 102 for obtaining the gaze direction of the eye 110. The means includes: an optical coupling configuration 120 (interchangeably referred to as a "first optical coupling configuration" and a "collimator-coupler") associated with LOE 102 for collimating and coupling light from the eye 110 into LOE 102 for guidance by LOE 102; a second optical coupling configuration 136 for coupling the LOE-guided light from the eye toward an optical sensor 152 out of LOE 102; and a processing system 154 electrically associated with optical sensor 152 and configured to process signals from optical sensor 152 to obtain the current gaze direction of the eye 110.

[0062] LOE 102 is roughly similar to Figure 1 The LOE 102 is shown. Specifically, the LOE 102 is formed of a transparent material (e.g., glass) and has at least a pair of parallel surfaces (planar main surfaces) 104 and 106 for guiding light via internal reflection. In some embodiments, the propagation (guiding) via internal reflection is in the form of total internal reflection (i.e., internal reflection is controlled by a critical angle as described above), while in other embodiments, the propagation via internal reflection is achieved by a coating (e.g., an angle-selective reflective coating) applied to surfaces 104 and 106. The LOE 102 is arranged such that one parallel surface 104 faces the eye 110, which is located in the EMB 114 at a distance 116 from the eye relief (ER) of surface 104.

[0063] Collimator-coupler 120 operates to collimate incident illumination (light) from eye 110 (or EMB 114) to generate collimated light, which is typically generated in response to illumination of eye 110 (EMB 114) by illumination arrangement 214. In addition to collimating the incident light to generate collimated light, collimator-coupler 120 operates to deflect the incident light, thereby coupling the resulting collimated light into LOE 102, such that the collimated light is trapped within LOE 102 by internal reflection. The trapped collimated light propagates within / through LOE 102 (i.e., guided by LOE 102) by internal reflection between surfaces 104 and 106 (i.e., at that location) until reaching the second optical coupling configuration 136, which couples the collimated light out of LOE 102 as the output light.

[0064] A focusing optics element 144 (interchangeably referred to as a “lens”) is schematically shown as a single lens, but may comprise a group of lenses. The focusing optics element 144 is associated with LOE 102 (at a portion of surface 104) and is arranged in the optical path between collimator-coupler 120 and optical sensor 152. The focusing optics element 144 receives the coupled light and converts the coupled light (a group of parallel rays) into one or more converging beams of captured light, which are incident on optical sensor 152. In some preferred embodiments, the focusing optics element 144 is an imaging optics element that forms an image of at least a portion of the eye 110 onto optical sensor 152.

[0065] Preferably, the focusing optics 144 are integrated together with the optical sensor 152 into the optical imaging module (or imaging system / camera system) 145, which is configured to sense and capture light and form an image of the portion of the eye that emits illumination. The imaging module 145 focuses to infinity, and the converging beam of captured light (through the lens 144) is focused onto different points / areas of the optical sensor 152.

[0066] For details, please refer to the following: Figures 2A to 3 The illustration shows a LOE 102 with an optical coupling configuration 120 according to a non-limiting embodiment of the invention. In the illustrated embodiment, the optical coupling configuration 120 is arranged obliquely within the LOE 102 relative to surfaces 104 and 106, and is located in a region of the LOE 102 at or near its distal end. The distal end is opposite to the proximal end of the LOE 102, and a second optical coupling configuration 136 is located at or near the proximal end.

[0067] Figure 2A The diagram illustrates the passage of light from eye 110 through LOE 102 to optical sensor 152. Typically, the propagation of light from eye 110 to optical sensor 152 within LOE 102 is referred to as propagation in the reverse direction within LOE 102 (interchangeably referred to as first propagation direction / second propagation direction, first direction / second direction, or reverse direction), while image light (in...) Figure 3 The beam 160 generated by the image projector 202 propagates toward the eye 110 within the LOE 102, which is referred to as propagation in the forward propagation direction (which may be interchangeably referred to as the second propagation direction / first propagation direction, second direction / first direction, or forward direction) within the LOE 102, and the forward propagation direction is approximately opposite to the reverse propagation direction.

[0068] Reflected light emanating from eye 110 (in response to illumination arrangement 214) is schematically represented in the figure by illumination beams 124, 126, and 128, each originating from a different corresponding region / part of eye 110. Beam 124 includes sample rays 124A and 124B spanning beam 124. Similarly, beam 126 includes sample rays 126A and 126B spanning beam 126, and beam 128 includes sample rays 128A and 128B spanning beam 128. Note that although each beam is illustrated as including two sample rays for simplicity, each beam typically includes a large number or continuous rays spanning the beam.

[0069] Each of beams 124, 126, and 128 has an angular distribution spanning two dimensions; in other words, each beam spans the lateral dimension (along...). Figure 2A The angular range within the horizontal axis (and across the vertical dimension (along the horizontal axis) and the vertical dimension (along the horizontal axis) is within the range of angles. Figure 2A The angular range within the plane of the paper (axis). Additionally, different regions / parts of the eye 110 from which light 124, light 126, and light 128 originate can include different regions in both the horizontal and vertical dimensions. Figure 2A You can see different regions along the horizontal dimension, while Figure 2B The isometric view of the optical system shown reveals different regions along the vertical dimension. Regarding... Figure 2B Note that, for the sake of simplicity, the passage of light through LOE 102 is not shown.

[0070] Continue to refer to Figure 2A Light 124, 126, and 128 from eye 110 enter LOE 102 via surface 104 and reach collimator-coupler 120. Depending on the angle of incidence of incident light 124, 126, and 128 relative to surface 104, some rays of one or more of beams 124, 126, and 128 may undergo refraction at surface 104 before reaching collimator-coupler 120. Light 124, 126, and 128 are collimated by collimator-coupler 120 to produce one or more sets of collimated beams. In particular, collimator-coupler 120 collimates light 124 to produce a collimated beam, which is schematically represented in the figures by rays 130A and 130B spanning the beam. Collimator-coupler 120 also collimates light 126 to produce a collimated beam, schematically represented in the figures by rays 132A and 132B crossing the beam, and collimator-coupler 120 collimates light 128 to produce a collimated beam, schematically represented in the figures by rays 134A and 134B crossing the beam. In addition to collimating the incident light, collimator-coupler 120 also deflects the light to couple collimated beams 130A, 130B, 132A, 132B, 134A, and 134B into LOE 102, such that the collimated light is captured within LOE 102 by internal reflection (generating reflected rays (upward rays) and also generating downward rays).

[0071] Collimated beams 130A, 130B, 132A, 132B, 134A, and 134B propagate along the substrate (LOE 102) via internal reflection (i.e., guided / passed through the substrate) until they reach the second optical coupling configuration 136 (schematically shown as a reflective surface in the figures, but it can also be implemented as a coupling prism or other coupling optical arrangement). The second optical coupling configuration 136 deflects (couples) the beams (lights 130A, 130B, 132A, 132B, 134A, and 134B) out of LOE 102 as coupled-out beams, which are schematically shown in the figures as coupled-out beams 138A, 138B, 140A, 140B, 142A, and 142B. Here, rays 138A and 138B cross the output beam corresponding to the input beam 124, rays 140A and 140B cross the output beam corresponding to the input beam 126, and rays 142A and 142B cross the output beam corresponding to the input beam 128. Note that, depending on the angle of incidence of the output rays 138A, 138B, 140A, 140B, 142A, and 142B relative to the surface 104, some rays deflected by the second optical coupling configuration 136 may undergo refraction at the surface 104 when leaving LOE 102.

[0072] Lens 144 converts each collimated output beam (i.e., the beam traversed by rays 138A and 138B, the beam traversed by rays 140A and 140B, and the beam traversed by rays 142A and 142B) into a set of convergent beams of captured light reaching optical sensor 152, such that each collimated output beam is focused by lens 144 onto a different corresponding portion of optical sensor 152. Three example convergent beams of captured light are shown in the figures, where each convergent beam of captured light corresponds to a different one of the three input beams 124, 126, and 128. The first convergent beam of captured light is schematically represented in the figures by sample rays 146A and 146B, which are focused by lens 144 onto a first region of optical sensor 152. Sample rays 146A and 146B correspond, respectively, to rays 138A and 138B of the output beams corresponding to input beam 124. Another converging beam of the captured light is schematically represented in the accompanying drawings by sample rays 148A and 148B, which are focused by lens 144 onto a second region of optical sensor 152. Sample rays 148A and 148B correspond to rays 140A and 140B of the output beam corresponding to input beam 126, respectively. A third converging beam of the captured light is schematically represented in the accompanying drawings by sample rays 150A and 150B, which are focused by lens 144 onto a third region of optical sensor 152. Sample rays 150A and 150B correspond to rays 142A and 142B of the output beam corresponding to input beam 128, respectively.

[0073] Optical sensor 152 generates signals in response to sensing focused light (e.g., an image corresponding to the eye), and these signals are transmitted to processing system 154, which is electrically associated with optical sensor 152 and configured to process the signals from optical sensor 152 to obtain the current gaze direction of eye 110.

[0074] According to some embodiments, the optical system is also configured to display an image to the eye 110 via an image projector and an optical coupling configuration (similar to the reference). Figure 1 (As described). Now refer to Figure 3This illustrates the propagation of light in the positive direction within LOE 102. The projected image 160, schematically represented here by the illumination beam 160 (including sample rays 160A, 160B, and 160C spanning the beam), is generated by an image projector 202 associated with one of the surfaces 104 and coupled into LOE 102 via an optical coupling configuration such that the collimated image 160 is captured within LOE 102 by internal reflection (generating reflected rays (upward rays) and also generating downward rays). In the illustrated embodiment, this optical coupling configuration is a second optical coupling configuration 136.

[0075] Image 160 propagates along LOE 102 via repeated internal reflections between surfaces 104 and 106 (i.e., guided / passed through LOE), and is incident on optical coupling configuration 108 (shown here as mutually parallel partially reflective surfaces arranged obliquely relative to surfaces 104 and 106 within LOE 102), wherein a portion of the image intensity is reflected by the partially reflective surfaces 108 so as to be coupled out of LOE 102 towards eye 110 as rays 162A, 162B, and 162C.

[0076] Image light 160 is (i.e., corresponds to) a collimated image, where each pixel is represented by a beam of parallel light rays at a corresponding angle, equivalent to light from a scene far from the viewer (the collimated image is referred to as "collimated to infinity"). Figure 3 In this image, the image is simply represented by rays 160A, 160B, and 160C, which correspond to a single point in the image, typically the centroid of the image, but in practice include angular ranges to each side of the central ray. These rays are coupled into the LOE 102 at corresponding angular ranges and similarly coupled out at corresponding angles, thereby producing a field of view corresponding to the portion of the image that reaches the viewer's eye 110 in different directions.

[0077] although Figure 3 Not shown, but the image projector 202 includes a microdisplay, typically a spatial light modulator such as a liquid crystal on silicon (LCoS) or liquid crystal display (LCD), but could also be another type of microdisplay used to generate image light, such as an organic light-emitting diode (OLED). The image projector 202 also includes corresponding collimating optics (…). Figure 3(Not shown in the image) is used to collimate the image up to infinity. When the microdisplay is implemented as a spatial light modulator, the illumination element (e.g., one or more LEDs) can be suitably arranged on the surface of one or more polarization beamsplitter (PBS) cubes or other prism arrangements together with the microdisplay and collimating optics to guide light from the illumination element to the microdisplay and image light to the collimating optics. It should also be noted that, although in Figures 2A to 2B and Figure 3 The optical imaging module 145 and image projector 202 are shown separately, but various configurations in which the imaging module operates in conjunction with and is integrated with the image projector 202 in some cases are conceivable, and references will be made to... Figure 7 and Figure 8 An example of such a configuration will be described later in this disclosure.

[0078] In some embodiments, the processing system 154 is also electrically associated with the image projector 202 to provide image generation control functions. Additionally, the processing system 154 may also be electrically associated with the lighting arrangement 214 to control the illumination timing of the EMB by the lighting arrangement 214. The following paragraphs describe the structure and operation of the lighting arrangement 214, as well as the structure and operation of the processing system 154 for obtaining the gaze direction based on the light sensed by the optical sensor 152.

[0079] The illumination arrangement 214 includes at least one light source, and preferably multiple light sources, each configured to illuminate one or more areas of the EMB 114 with light in the eye-tracking spectrum, such that a portion of the intensity of the light incident on the EMB 114 / eye 110 from the illumination arrangement 214 is reflected back as reflected light by the eye 110 and specifically by the collimator-coupler 120 toward the LOE 102. The light source (or multiple sources) of the illumination arrangement 214 may be implemented as an LED, or any other source configured to emit (generate) light in the eye-tracking spectrum. In some non-limiting implementations, the light source of the illumination arrangement 214 is an isotropic (or nearly isotropic) source emitting light in all directions. Preferably, the illumination arrangement 214 is configured to illuminate the eye 110 with light whose wavelength is outside the visual region of the electromagnetic spectrum. In other words, the illumination arrangement 214 is preferably configured to illuminate the eye 110 with light invisible to the human eye. Reflections from the human eye, especially from the retina, are significantly higher in the near-infrared region than at visible wavelengths. Therefore, it is preferable that the illumination arrangement 214 is configured to illuminate the eye 110 with light in the NIR region of the electromagnetic spectrum.

[0080] Various deployment configurations of the lighting arrangement 214 can be employed to illuminate the eye 110 for eye tracking purposes. In one non-limiting example configuration of the lighting arrangement 214, the lighting arrangement 214 includes one or more light sources arranged near the optical sensor 152 and / or around the mechanical body of the optical system 100 on which the LOE 102 is mounted. Figure 4 A non-limiting example configuration is shown, in which the optical system 100 is implemented as a spectacle shape factor having a head-mounted mechanical body, which is implemented as a spectacle frame 216 having side arms 218 for engaging the viewer's (the viewer's) ears. The optical system 100 is powered by a suitable power source, which can be any combination of a battery and / or a supplied external power source, schematically shown here as a power source 220 connected via cable 222. In the case of battery power, the battery can be integrated as part of the spectacle. It should be noted that other shape factors, such as helmet-mounted shape factors, vehicle windshield shape factors, and other head-up display and near-eye display shape factors, also clearly fall within the scope of this invention. Figure 4 In the non-limiting configuration shown, the lighting arrangement 214 includes three separate light sources 215A, 215B, and 215C (e.g., implemented as three LEDs). Two light sources, namely light sources 215A and 215B, are arranged on the top portion facing LOE102 and on the peripheral portion of the eyeglass frame 216 at or near the partially reflective surface 108. The third light source 215C is arranged near one side of the viewer's head near the optical coupling configuration 136.

[0081] Incidentally, it should be noted that other shape factors, such as helmet-mounted shape factors, vehicle windshield shape factors, and other head-up display and near-eye display shape factors, also clearly fall within the scope of this invention. Certain embodiments of this disclosure can be particularly valuable when arranged as part of a head-up display (HUD) in a vehicle or aircraft, whereby the display of an image projected by the image projector 202 in an automotive or aviation environment can depend at least in part on or be controlled by the user's eye gaze direction. In an automotive environment, a HUD employing the main components of an optical system according to the disclosed embodiments can be mounted in front of the driver of the vehicle, for example, integrated into the vehicle's dashboard or windshield. In an aviation environment, a HUD can be mounted in front of the pilot of an aircraft, for example, as part of the pilot's helmet in the front area of ​​the helmet.

[0082] The processing system 154 can be implemented using any suitable type of processing hardware and / or software known in the art, including but not limited to any combination of various dedicated computerized processors that operate under any suitable operating system and implement suitable software or firmware modules. The processing system 154 may also include various communication components for allowing wired or wireless communication with LAN and / or WAN devices for bidirectional transmission of information and graphical content. Figure 2A In the illustrated non-limiting example processing system 154, the processing subsystem 154 includes at least one computerized processor 156 coupled to a storage medium 158. The storage medium 158 may be one or more computerized memory devices, such as volatile data storage devices. The processor 156 may be implemented as any number of computerized processors, including but not limited to microprocessors, microcontrollers, graphics processors, display drivers, application-specific integrated circuits (ASICs), digital signal processors (DSPs), image processors, field-programmable gate arrays (FPGAs), field-programmable logic arrays (FPLAs), etc. Such computerized processors include computer-readable media storing program code or instruction sets, or media that can electronically communicate with the computerized processor, the program code or instruction set causing the computerized processor to perform actions when executed by the computerized processor. Types of computer-readable media include, but are not limited to, electronic devices, optical devices, magnetic devices, or other storage or transmission devices capable of providing computer-readable instructions to the computerized processor. Although the processing system 154 is shown to be arranged locally with the optical sensor 152, and in some cases integrated into the mechanical body of the optical system 100 (e.g. Figure 4 (In the middle), however, note that the processing system 154 may alternatively be arranged remotely from other major components of the optical system 100. For example, in some embodiments, the processing system 154 may be implemented as a remote processing server that receives signals representing those generated by the optical sensor 152 in response to sensing captured light. The signals can be transmitted to the remote processing system 154 via one or more wired and / or wireless communication networks using a network interface device connected to the optical sensor 152.

[0083] In some non-limiting embodiments, the optical system 100 obtains the gaze direction (the angular orientation of the eye or the line of sight of the eye) by imaging a pattern present on a specific area of ​​the eye 110. The position and movement of such a pattern indicate the current gaze direction and movement of the eye. The human eye includes a variety of trackable features, including, for example, patterns generated by the nerves of the cornea based on corneal reflections, the center of the pupil (i.e., corneal nerve patterns), and patterns generated by the blood vessels of the optic disc. These trackable features can be tracked using appropriate tracking algorithms implemented by appropriate image processing instructions executed by the processing system 154. In some non-limiting embodiments, the processing system 154 calculates the gaze direction based on the vector between the pupil center and the corneal reflection.

[0084] Typically, all background illumination introduces noise that degrades the quality of the eye-tracking image. To reduce the impact of external lighting sources (e.g., ambient light, natural sunlight, etc.), the illumination arrangement 214 can be configured to generate short light pulses (preferably less than 1 ms), and the optical sensor 152 is synchronized (by processing subsystem 154) to integrate light only during this short illumination duration. In this way, continuous background illumination can be significantly suppressed. Alternatively, a passband spectral filter can be arranged in the optical path from the second optical coupling configuration 136 to the optical sensor 152 to block light of wavelengths outside a given wavelength range from which eye-tracking illumination is generated. The spectral filter can ideally be positioned between the focusing optics 144 and the optical sensor 152, but can alternatively be arranged before the focusing optics 144.

[0085] In a non-limiting process for obtaining and tracking the gaze direction, a corneal pattern (optionally combined with an optic disc pattern and / or pupil) is mapped and trackable features are determined during an initial setup process, followed by a continuous tracking process. For example, an image marker may be displayed to the viewer for viewing during initialization. When the viewer looks toward the marker, illumination arrangement 214 fully illuminates the cornea and obtains a full image of the cornea (and pupil) (via optical sensor 152). This image is then processed by processing system 154 to identify trackable features (e.g., optic disc and fovea). During the continuous tracking process, selected regions of interest (ROIs) of the eye 110 are selectively illuminated by illumination arrangement 214, and the image of the ROI (obtained by optical sensor 152) is sampled and processed by (processing system 154) during the corresponding illumination pulse to determine the current gaze direction (line of sight), and this obtained gaze direction is used to update the position of the ROI for subsequent illumination cycles, and the continuous tracking process is repeated by illuminating the updated ROI. Assuming the frequency of tracking measurements is high compared to the speed of eye movement, this update process typically maintains continuous tracking effectively, optionally combined with tracking information from the other eye. The illuminated area changes when the gaze direction changes. ROI updates can be performed based on the “current” gaze direction determined from the last sampled image, or in some cases, using extrapolated predictions based on eye movements between two or more previous measurements. In the event of tracking failure, the size of the illuminated area can be temporarily increased until trackable features are recovered.

[0086] The light source of illumination arrangement 214 can be configured to emit eye-tracking light at approximately the same or different center wavelengths within the eye-tracking spectrum. Typically, in the NIR region, the glass material used to construct LOE 102 can have sufficiently low dispersion to avoid distortion within the spectral width of a single eye-tracking light source (for LEDs, the spectral width is typically in the range of 20 nm to 50 nm). However, employing light sources emitting eye-tracking light at two spectrally separate center wavelengths (while still within the same region of the eye-tracking electromagnetic spectrum) can provide certain advantages when imaging the eye. For example, arranging illumination arrangement 214 with a first light source and a second light source emitting light with centers around approximately 700 nm and 950 nm respectively can result in two different images of the eye formed on optical sensor 152, one shifted relative to the other. By applying appropriate image processing algorithms, such as correlation algorithms, processing subsystem 154 can achieve higher resolution in gaze direction calculation.

[0087] The optical structure and characteristics of the collimator-coupler 120 according to certain embodiments of this disclosure will now be discussed in more detail. (See again...) Figures 2A to 3In the embodiment shown, the collimator-coupler 120 includes an optical element 122 (in... Figure 2A and Figure 2B (Schematably represented by a lens), the optical element 122 performs both collimation and light deflection to couple light into the LOE. The collimation and coupling functions of the optical element 122 are achieved by an optical surface, which, in some preferred but non-limiting implementations, is a spherical (or near-spherical) surface or an aspherical surface. Besides spherical or aspherical (curved) surfaces, the optical element 122 can also be implemented in other ways, including, for example, holographic surfaces or dichroic gratings. To effectively collimate and couple light from the eye 110, the optical element 122 preferably has one or more of the following characteristics:

[0088] 1) Optical element 122 preferably exhibits light-resolving properties, for example, achieved by a coating that distinguishes certain types of light, such that optical element 122 reflects only eye-tracking light and transmits image light generated by the image projector. In some embodiments, the light resolving power of optical element 122 is achieved by the following: optical element 122 is spectrally selective, such that optical element 122 reflects light having wavelengths in the eye-tracking spectrum and transmits light in the photopic (visible) spectrum. As mentioned, the eye-tracking spectrum refers to the spectrum occupied by the light generated by illumination arrangement 214, which is preferably the NIR region of the electromagnetic spectrum (but may also be other regions of the spectrum including, for example, infrared or ultraviolet, as will be discussed). In other embodiments, the light resolving power of optical element 122 is achieved by the following: optical element 122 is polarization selective, such that optical element 122 reflects incident light polarized in one polarization direction and transmits incident light polarized in a second polarization direction orthogonal to the first polarization direction. The polarization selectivity of optical element 122 will be discussed in further detail below.

[0089] 2) Optical element 122 is arranged obliquely relative to surfaces 104 and 106, and is preferably appropriately sized to have an elongated shape (in the direction of extension perpendicular to the vector defined by the oblique arrangement angle), the elongated shape having sufficient length to collimate and couple all light from eye 110 (EMB 114), including beams 124 and 128 from the edges of EMB 114. Optical element 122 is also preferably appropriately sized to be narrow enough (measured along the vector defined by the oblique arrangement angle) to be enclosed within LOE 102 (between surfaces 104 and 106).

[0090] 3) When implemented as a curved (spherical or aspherical) surface, the optical element 122 has a curvature that is preferably optimized (or nearly optimized) to collimate the eye 110 (or EMB 114) up to infinity (i.e., to image the eye 110 at infinity). A specific curvature value can be calculated based on ER 116 (the distance between surface 104 and EMB 114).

[0091] 4) Preferably, the optical element 122 has its optical (collimating) surface arranged between two media having the same refractive index, such that light propagating through the collimating surface (e.g., spherical or aspherical) does not undergo a change in optical power.

[0092] 5) Optical element 122 is arranged at an angle relative to surfaces 104 and 106. The angle of the optical element 122 may differ from the angle of the partially reflective surface 108, but should be selected such that all light 124 from the eye 110 is deflected at an appropriate angle to ensure that the deflected light is coupled into the LOE 102 and captured by internal reflection. It has been found that an angle of 25° to 35° for the optical element 122 is suitable for achieving effective light capture. In some preferred but non-limiting implementations, the optical element 122 is arranged at an angle of approximately 30° relative to surfaces 104 and 106.

[0093] In addition to employing discrimination to selectively reflect and transmit incident light, optical element 122 is preferably also operated to employ discrimination to collimate only certain types of incident light. Specifically, optical element 122 is preferably configured to collimate only the light reflected from the eye (i.e., apply optical power only to the light reflected from the eye), but not to collimate (i.e., not apply optical power to) the image light generated by the image projector. A particular advantage of collimator-coupler 120 having optical element 122 with such characteristics is that optical element 122 can be arranged in other areas of LOE 102, for example, overlapping with optical coupling configuration 108 (e.g., across one or more partial reflective surfaces 108), rather than as Figures 2A to 3 The limited arrangement shown is located at or near the far end of LOE 102.

[0094] As described above, in some embodiments, optical element 122 may be polarization-selective. Polarization selectivity may replace the spectral selectivity described above, or polarization selectivity may be used in addition to the spectral selectivity described above. For example, optical element 122 may be configured to transmit incident light having a first polarization direction (e.g., s-polarized or p-polarized) relative to the surface of optical element 122, and to reflect incident light having a second polarization direction (e.g., p-polarized or s-polarized) orthogonal to the first polarization direction. In one example, illumination arrangement 214 may include one or more polarized light sources that produce p-polarized NIR light, such that light 124, light 126, and light 128 are p-polarized NIR light. An image projector may produce s-polarized light in the visible region, such that illumination 160 is s-polarized visible light. In such an example, optical element 122 may be designed to transmit s-polarized light in the visible region without applying optical power to s-polarized light in the visible region, and to apply optical power (for collimation) to NIR p-polarized light and reflect NIR p-polarized light. It should be understood that other combinations of spectral and polarization selectivity can be used to achieve the distinction between eye-tracking light and image light (from the image projector) via optical element 122.

[0095] Now for reference Figure 5 This illustrates an optical system according to another embodiment of the invention. Here, the optical system includes a second light-transmitting substrate 164 formed of a transparent material (e.g., glass) and having one pair of opposing (main surfaces) 166 and 168. An optical element 122 of an optical coupling configuration 120 is arranged in the substrate 164, wherein the long principal axis of the optical element 122 is parallel to surfaces 104 and 106 of the LOE 102. The substrate 164 is arranged such that one of its main surfaces 166 is associated with one of the main surfaces 106 of the LOE 102, such that the two surfaces 106 and 166 face each other, and the other surface 168 faces the real-world scene. As a result, the LOE 102 is located between the eye 110 and the substrate 164.

[0096] LOE 102 and substrate 164 are preferably made of materials having the same refractive index and can be attached to each other at surfaces 106 and 166 using optical attachment materials such as optical adhesives or gels. Preferably, the optical attachment material is a matching material, such that light is transmitted from LOE 102 to substrate 164 without reflection and refraction (and vice versa, light is transmitted from substrate 164 to LOE 102).

[0097] In the illustrated embodiment, the optical coupling configuration 120 further includes a second optical element 170, which is a partially reflective surface arranged at an angle relative to surfaces 104 and 106 in the LOE 102. Optical element 170 is used to deflect collimated light from optical element 122 so as to capture the deflected light within the LOE 102 by internal reflection. It has been found that an angle of arrangement of the second optical element 170 in the range of 25° to 35° is particularly suitable for achieving efficient light capture. In some preferred but non-limiting implementations, the second optical element 170 is arranged at an angle of approximately 30° relative to surfaces 104 and 106. The arrangement orientation of the optical element 170 is opposite to that of the partially reflective surface 108, the reason for which will become clear when discussing the passage of light from the eye 110 to the optical sensor 152. Additionally, as Figure 5 As shown, the optical element 170 can be arranged in an overlapping relationship with the partially reflective surface 108. Although the optical element 170 is shown as a plane, it can also be implemented as a curved surface.

[0098] Optical element 122 is preferably arranged in a region of substrate 164 generally in front of eye 110, such that the normal from the center of the optical collimating surface of optical element 122 reaches the center (or approximately the center) of pupil 112 of eye 110. This arrangement increases the collimating effectiveness employed by optical element 122 because... Figures 2A to 3 Compared to the edge rays in the illustrated embodiment, even the edge rays of beams 124, 126, and 128 arrive at the optical element 122 at an angle closer to the normal to the optical surface of the optical element 122.

[0099] Optical element 170 may be spectrally selective, such that it is partially reflective (and therefore partially transmissive) to light in the eye-tracking spectrum. The reflectivity or transmissivity of optical element 170 to light in the photopic spectrum can be configured according to the desired display characteristics of the optical system. For example, in some non-limiting embodiments, optical element 170 may be designed to reflect 50% of the light intensity in the NIR region and transmit 100% of the light intensity in the photopic region, such that image light propagating through LOE 102 or coupled from the partially reflective surface 108 (e.g., Figure 3 The light sources 160, 162A, 162B, and 162C are not affected by the optical element 170.

[0100] The following paragraphs describe the process of... Figure 5The optical coupling configuration 120 shows the passage of light from eye 110 to optical sensor 152. For simplicity, only a sample ray of light from each of beams 124, 126, and 128 will be shown here. Light 124A, 126A, and 128A from eye 110 enter LOE 102 via surface 104 and reach optical element 170. Depending on the angle of incidence of the incident light to surface 104, some rays from one or more of beams 124, 126, and 128 may undergo refraction at surface 104 before reaching optical element 170. A portion of the intensity of light 124A, 126A, and 128A is transmitted by optical element (partially reflective surface) 170. Figure 5 The transmitted light is schematically represented by rays 172A, 174A, and 176A. Transmitted light 172A, 174A, and 176A exit LOE 102 through surface 106 and then enter substrate 164 through surface 166. Rays 172A, 174A, and 176A then reach optical element 122, where the incident light is collimated and deflected, thereby generating collimated rays 130A, 132A, and 134A that propagate backward toward surface 166 and exit substrate 164 via surface 166. Collimated rays 130A, 132A, and 134A then enter LOE 102 through surface 106 and reach optical element 170. A portion of the intensities of collimated beams 130A, 132A, and 134A are reflected by optical element 170 to generate reflected light (in... Figure 5 (Schematably represented as light rays 178A, 180A, and 182A). Optical element 170 deflects collimated light rays 130A, 132A, and 134A at an appropriate angle, such that the resulting reflected light rays 178A, 180A, and 182A are captured (i.e., guided) within LOE 102 by internal reflection at surfaces 104 and 106. The captured light rays 178A, 180A, and 182A propagate through LOE 102 by internal reflection until they reach the second optical coupling configuration 136, which couples the light rays 178A, 180A, and 182A out of LOE 102, becoming coupled light rays 138A, 140A, and 142A. Then, the outgoing light 138A, outgoing light 140A, and outgoing light 142A reach the imaging module. On the imaging module, lens 144 collimates the outgoing light 138A, outgoing light 140A, and outgoing light 142A into one or more converging beams of capturing light (146A, 148A, 150A). Then, the one or more converging beams reach the optical sensor 152, which generates a signal. This signal is processed by the processing system 154 to obtain the gaze direction of the eye 110.

[0101] In some embodiments, instead of or in addition to spectral selectivity, optical element 170 may be polarization-selective. For example, optical element 170 may be designed to transmit all polarized light in the visible region and transmit s-polarized or p-polarized eye-tracking light (e.g., in the NIR region) and reflect p-polarized or s-polarized eye-tracking light. In such an example, light 172A, light 174A, and light 176A are s-polarized or p-polarized, and a delay plate, such as a half-wave plate (not shown), may be arranged parallel to surface 106 between substrate 164 and LOE 102, such that light 172A, light 174A, and light 176A pass through the delay plate and are converted into circularly polarized light. Optical element 122 collimates the circularly polarized light to produce circularly polarized collimated light 130A, circularly polarized collimated light 132A, and circularly polarized collimated light 134A. Then, the circularly polarized collimated beams 130A, 132A, and 134A pass backward through a delay plate, which converts them into p-polarized or s-polarized collimated beams. These collimated beams are then reflected by optical element 170 to produce p-polarized or s-polarized collimated beams 178A, 180A, and 182A, which are captured within LOE 102 by internal reflection.

[0102] Notice, Figure 5 A non-limiting configuration of substrate 164 is shown, wherein substrate 164 is implemented as a plate-shaped substrate. In such a configuration, surfaces 166 and 168 are parallel to each other and parallel to surfaces 104 and 106. However, the requirement for parallelism between the principal planes of substrate 164 (i.e., surfaces 166 and 168) is much less stringent than the requirement for LOE 102 for image projection into eye 110, where a parallelism on the order of about 1 arcminute may be required. Therefore, it should be understood that other non-limiting configurations of substrate 164 can be implemented, including configurations in which one or both of surfaces 166 and 168 are curved surfaces and / or in which substrate 164 is formed as a lens that provides optical power to incident light from a real-world scene that is directly visible to the viewer's eye 110. Alternatively, or in addition to such a curved / lens configuration, the substrate 164 may be configured with a reflection suppression component having a certain curvature in order to reduce the illusions caused by light from a real-world scene incident on the substrate 164 at a specific incident angle.

[0103] although Figure 5A specific configuration is shown in which optical element 122 is arranged in a second substrate 164 and partially reflective optical element 170 is arranged in LOE 102. However, other implementations are conceivable where the second substrate 164 is not used and both optical elements 122 and 170 are arranged in LOE 102. However, note that the size / size of optical element 122 may need to be reduced to ensure that both optical elements 122 and 170 are housed within LOE 102. This could reduce the angular range of illumination from eye 110 that can reach collimator-coupler 120, potentially reducing the accuracy of gaze direction determination.

[0104] Now for reference Figure 6 An optical system according to another embodiment of the present invention is shown. Similar to... Figure 5 Implementation method, Figure 6 The embodiment shown employs a second substrate 164. However, in Figure 6 In this embodiment, the second substrate 164 is arranged such that its main surface 168 is associated with the main surface 104 of the LOE 102, such that the substrate 164 is located between the eye 110 and the LOE 102. Therefore, surfaces 168 and 104 are facing each other, and another surface 166 is facing the eye 110. The LOE 102 and the substrate 164 can be attached to each other at surfaces 104 and 168 using optical attachment materials such as optical adhesives or gels. Preferably, the optical attachment material is a ratio material, such that light is transmitted from the LOE 102 to the substrate 164 without reflection or refraction (conversely, light is transmitted from the substrate 164 to the LOE 102).

[0105] exist Figure 6In this embodiment, the optical coupling configuration 120 further includes two optical elements 184 and 192, which are reflective or partially reflective surfaces. Although both optical elements 184 and 192 are shown as planar, either or both of optical elements 184 and 192 can be implemented as curved surfaces. Optical elements 122 and 184 are arranged obliquely relative to main surfaces 104 and 106 (and obliquely relative to main surfaces 166 and 168 when surfaces 166 and 168 are parallel to each other and parallel to surfaces 104 and 106). Optical element 192 is arranged obliquely relative to surfaces 104 and 106 in LOE 102. Generally, optical element 184 is used to redirect collimated light from optical element 122 toward the other optical element 192, which deflects the received redirected collimated light so as to couple the collimated light into LOE 102 by internal reflection. It has been found that an angled arrangement of optical element 184 in the range of 25° to 35° is particularly suitable for achieving effective redirection of light toward optical element 192. In some preferred but non-limiting implementations, optical element 184 is arranged at an angle of approximately 30° relative to surfaces 104 and 106. Similarly, it has been found that an angled arrangement of optical element 192 in the range of 25° to 35° is particularly suitable for achieving effective light capture. In some preferred but non-limiting implementations, optical element 192 is arranged at an angle of approximately 30° relative to surfaces 104 and 106. The arrangement orientation of optical elements 122, 184, and 192 is opposite to that of the partially reflective surface 108, the reason for which will become clear when discussing the passage of light from eye 110 to optical sensor 152. Preferably, optical elements 184 and 192 are aligned with each other such that the projections of optical elements 184 and 192 in a plane parallel to the main plane (surface 104, surface 106) of LOE 102 overlap each other completely.

[0106] The following paragraphs describe the process of... Figure 5 The optical coupling configuration 120 shows the passage of light from the eye 110 to the optical sensor 152. For simplicity, only one sample ray (124A, 126A, 128A) of each of beams 124, 126, and 128 will be shown here.

[0107] Light 124A, 126A, and 128A from eye 110 enter substrate 164 via surface 166 and reach optical element 122. Depending on the angle of incidence of the incident light to surface 166, some rays of one or more of beams 124, 126, and 128 may undergo refraction at surface 166 before reaching optical element 122. Light 124A, 126A, and 128A are collimated and deflected by optical element 122, thereby generating collimated light 130A, 132A, and 134A. Collimated light 130A, 132A, and 134A reach optical element 184, which redirects (reflects) the collimated light 130A, 132A, and 134A toward optical element 192. The redirected (reflected) light... Figure 5 The ray is schematically represented as ray 186A, ray 188A, and ray 190A.

[0108] Incidentally, some of the collimated beams 130A, 132A, and 134A may propagate via internal reflection at one or both of surfaces 166 and 168 before reaching optical element 184. For example, in Figure 5 In this process, light rays 124A and 126A are reflected by internal reflection at surfaces 166 and 168 before reaching optical element 184. Therefore, in order to ensure that optical element 184 properly redirects collimated light, portions of surfaces 166 and 168 in the region of the substrate 164 in which optical elements 122 and 184 are arranged should preferably be parallel or as close to parallel as possible.

[0109] Redirected light 186A, redirected light 188A, and redirected light 190A exit the substrate 164 via surface 168, enter the LOE 102 via surface 104, and reach the optical element 192. The optical element 192 reflects light 186A, light 188A, and light 190A to generate light 194A, light 196A, and light 198A. The optical element 192 deflects light 188A, light 190A, and light 192A at an appropriate angle, such that the resulting light 194A, resulting light 196A, and resulting light 198A are captured (i.e., guided) within the LOE 102 by internal reflection at surfaces 104 and 106. The captured light beams 194A, 196A, and 198A propagate through LOE 102 via internal reflection until they reach the second optical coupling configuration 136. The second optical coupling configuration 136 couples the light beams 194A, 196A, and 198A out of LOE 102 (via surface 104) and then out of substrate 164 (via surface 166), becoming coupled light beams 138A, 140A, and 142A. The coupled light beams 138A, 140A, and 142A then reach the imaging module. On the imaging module, focusing optics 144 converts the collimated coupled light beams into one or more converged beams of captured light. These converged beams then reach the optical sensor 152, which generates a signal. This signal is processed by the processing system 154 to determine the gaze direction of the eye 110.

[0110] Figure 6 In this embodiment, the substrate 164 can be a plate-type substrate. However, compared with the reference... Figure 5 Similarly, the parallelism requirement between the principal planes (i.e., surfaces 166 and 168) of substrate 164 is much less stringent than the requirement for the LOE 102 used for image projection into eye 110. However, as mentioned above, portions of surfaces 166 and 168 in the regions of substrate 164 in which optical elements 122 and 184 are arranged should be parallel or as close to parallel as possible. Keeping this in mind, substrate 164 may still exhibit a degree of curvature outside these regions. Therefore, similar to the reference... Figure 5 As discussed, substrate 164 can still be implemented with a certain degree of curvature or formed as a lens.

[0111] Although the embodiments discussed so far have involved the optical coupling configuration 136 as a reflective surface (or coupling prism or other coupling surface), other embodiments are conceivable where the optical coupling configuration is not a surface, but rather an opening end of the LOE 102. In such an embodiment, the opening end can be achieved by cutting or slicing the LOE 102 along a plane perpendicular to (or orthogonal to) surfaces 104 and 106. Figure 7 It was shown as Figure 2A and Figure 2B An example of such an embodiment, modified as shown (for simplicity, the partially reflective surface 108 is omitted), is provided, wherein the optical coupling configuration 136 includes a planar opening end 200 formed by cutting the substrate near or near the proximal end of the LOE 102 along a plane orthogonal to the two main surfaces 104 and 106 of the LOE 102. The cutting plane is also perpendicular to the elongation direction of the LOE 102. Figure 7 Along the horizontal axis. As can be seen, the collimated light from the eye (in...) Figure 7 The light rays 130A, 130B, 132A, 132B, 134A, and 134B reach the opening end 200 and naturally leave LOE 102 as the outgoing light rays 130A, 130B, 132A, 132B, 134A, and 134B.

[0112] Imaging module 145 is provided with a lens 144 associated with opening end 200. Lens 144 is arranged such that two images are formed on two corresponding sides (halves) of the detector surface of optical sensor 152. Specifically, lens 144 focuses outgoing light 130A, outgoing light 130B, outgoing light 132A, outgoing light 132B, outgoing light 134A, and outgoing light 134B, such that the light rays 130A, 132A, and 134A last reflected from surface 104 within LOE 102 are converted into different convergent beams of captured light reaching different corresponding portions of the lower half of the detector surface of optical sensor 152, and the light rays 130B, 132B, and 134B last reflected from surface 106 within LOE 102 are converted into different convergent beams of captured light reaching different corresponding portions of the upper half of the detector surface of optical sensor 152. Figure 7 In this configuration, lens 144 converts light 130A, light 132A, and light 134A into corresponding converging beams 146A, 148A, and 150A, and converts light 130B, light 132B, and light 134B into corresponding converging beams 146B, 148B, and 150B. Optical sensor 152 generates a signal, which is processed by processing system 154 to determine the gaze direction. In determining the gaze direction, processing system 154 first combines the two images to produce a single image with a uniformly distributed intensity, thereby improving the quality of the determined gaze direction.

[0113] although Figure 7 The configuration of LOE 102 shown in the embodiment is illustrated as follows: Figure 2A and Figure 2B Modifications to the implementation method, but Figure 7The configuration can be used with any of the foregoing embodiments of the optical coupling configuration 120. However, it should be noted that when used with reference... Figure 2A and Figure 2B When used with the described embodiment of the optical coupling configuration 120, employing the planar opening end 200 may be most efficient to avoid unnecessary cutting of the substrate 164. Furthermore, in embodiments, the use of the opening end 200 can also be used in conjunction with a separate optical coupling configuration—such as a reflective surface or coupling prism that couples collimated image light from the image projector into the LOE 102—provided that the coupling configuration transmits collimated light from the eye.

[0114] It should be noted that the eye-tracking device of the embodiments described so far can be advantageously used independently of a display system, for example, in non-AR / VR applications where it is desirable to determine the direction of eye gaze. For example, the eye-tracking device of the embodiments of this disclosure can be used in conjunction with applications associated with computers or mobile devices, where the direction of a user's eye gaze can be used for navigation displays, web pages, menus, etc., or for interacting with computerized games played on computer devices (e.g., video game systems, mobile devices, laptops, desks, etc.). In such applications, "LOE" can include the display screen of the computer device, and the collimator-coupler 120 can be suitably associated with a portion of the display screen to collimate and deflect eye-tracking light reflected from the eye toward the display screen.

[0115] Despite the above description, various eye-tracking devices according to embodiments of the present invention are particularly suitable for AR display system applications and / or VR display system applications, wherein scene images are generated by a small optical image generator (image projector) and displayed to the viewer's eyes using an optical waveguide / substrate (i.e., LOE) having a partially reflective surface (or another type of optical coupling arrangement), the small optical image generator having a small aperture that is multiplied to generate a large aperture. The following paragraphs describe embodiments of optical systems combining eye-tracking and display functions, with particular attention to the structure and operation of the image projector that generates the image to be displayed to the eyes, and imaging / camera systems for imaging the eyes for eye-tracking purposes.

[0116] refer to Figure 8 An optical system according to an embodiment of the present invention is shown, which is similar to Figures 2A to 3The illustrated embodiment. In the illustrated embodiment, image light generated by image projector 202 is coupled into LOE 102 via optical coupling configuration 210 (schematically shown as a reflective surface in the drawings, but it can also be implemented as a coupling prism or other coupling optical device), which is separate from optical coupling configuration 136. Here, optical coupling configuration 136 is only used to couple eye-tracking light out of LOE 102 toward imaging module 145. For simplicity, in Figure 8 The middle LOE 102 is shown as a portion having only a partial reflective surface 108, an optical coupling configuration 210, and an optical coupling configuration 136.

[0117] In the illustrated embodiment, the image projector 202 includes a light source for generating image light (in... Figure 8 The image is shown as an image light with sample rays 159A, 159B, and 159C. It includes a spatial light modulator (SLM) 204 (e.g., an LCoS chip) and collimating optics 206 for collimating the images 159A, 159B, and 159C to infinity to produce a collimated beam having sample rays 160A, 160B, and 160C across it. An illumination source, such as one or more LEDs (not shown), is typically used to illuminate the SLM 204 to drive image generation. The SLM 204, collimating optics 206, and illumination source may be suitably arranged on the surface of one or more polarization beams (PBS) cubes or other prism arrangements.

[0118] Beams 160A, 160B, and 160C are coupled into LOE 102 via optical coupling configuration 210, such that the coupled beams 160A, 160B, and 160C are captured within LOE 102 by internal reflection. Images 160A, 160B, and 160C propagate along LOE 102 in the positive direction via repeated internal reflections between surfaces 104 and 106 until they reach a partially reflective surface 108, where a portion of the image intensity is reflected so that it is coupled out of LOE 102 towards eye 110 as rays 162A, 162B, and 162C. Rays 159A, 159B, 159C, 160A, 160B, 160C, 162A, 162B, and 162C... Figure 8 The dashed arrows are used to indicate the light source and the eye-tracking light, so as to more clearly distinguish the image light from the eye-tracking light.

[0119] Lighting from the eye (for simplicity, in) Figure 8Only beams 126A and 126B emanating from the center of EMB 114 are shown in the image, collimated by a coupler. Figure 8 Not shown, but it can be any collimator-coupler discussed herein, such as Figure 2A The collimator-coupler 120 collimates and couples the collimated light 132A and 132B generated by the collimator-coupler into the LOE 102, so that the collimated light 132A and 132B generated by the collimator-coupler are captured within the LOE 102 by internal reflection. The light 132A and 132B propagate in opposite directions along the LOE 102 by repeated internal reflection between surfaces 104 and 106 until they reach the optical coupling configuration 136, which reflects the light 132A and 132B so as to couple the light 132A and 132B out of the LOE 102 as the coupled light 140A and coupled light 140B. Then, the coupled light 140A and coupled light 140B are focused onto the optical sensor 152 by the focusing optics 144 (which generates converging beams 148A and 148B).

[0120] In order for light 132A and light 132B to propagate along LOE 102 to reach optical coupling configuration 136, optical coupling configuration 210 (arranged between optical coupling configuration 136 and partially reflective surface 108) should be selectively reflective, that is, distinguishing between eye-tracking light 132A and eye-tracking light 132B and image light 160A, image light 160B, and image light 160C, such that optical coupling configuration 210 transmits eye-tracking light and reflects image light. In a non-limiting example, the light discrimination of optical coupling configuration 210 can be achieved by implementing optical coupling configuration 210 as a spectrally selective surface (e.g., a dichroic surface), which reflects light in the photopic region (e.g., images 160A, 160B, and 160C) and transmits light in the NIR region (e.g., light 132A and image 132B). In another non-limiting example, the optical discrimination of the optical coupling configuration 210 can be achieved by implementing a polarization-selective surface that reflects s-polarized or p-polarized light (where image light 160A, image light 160B, and image light 160C are s-polarized or p-polarized) and transmits p-polarized or s-polarized light (where eye-tracking light 132A and eye-tracking light 132B are p-polarized or s-polarized). In still some non-limiting examples, a combination of spectral selectivity and polarization selectivity can be used.

[0121] In some implementations, the positions of optical coupling configuration 210 and optical coupling configuration 136 can be interchanged. In such an implementation, the positions of image projector 202 and imaging module 145 should also be interchanged, and optical coupling configuration 210 no longer needs to be selectively reflective (light-resolved). However, for proper coupling in and out of LOE 102, optical coupling configuration 136 should be selectively reflective (light-resolved), such that optical coupling configuration 136 transmits image light corresponding to the collimated image (e.g., light 160A, light 160B, light 160A) and reflects collimated eye-tracking light (e.g., light 132A, light 132B).

[0122] Figure 9 Another embodiment of the optical system is shown, employing a compact configuration in which the imaging module and image projector share common components for integration into a single imaging and projector module 224. In such an embodiment, optical coupling configuration 136 is used to couple collimated image light 160A, collimated image light 160B, and collimated image light 160C into LOE 102, and also to couple eye-tracking light 132A and eye-tracking light 132B out of LOE 102 as coupled-out light 140A and 140B. For simplified presentation, LOE 102 is shown in... Figure 9 The portion shown is only the one with optical coupling configuration 136.

[0123] Module 224 includes an SLM 202 (e.g., an LCoS chip), optics 226, and an optical sensor 152. Optics 226 performs the dual function of collimating image light 159A, image light 159B, and image light 159C generated by the SLM 202 to generate collimated beams 160A, 160B, and 160C, and focusing the coupled eye-tracking light 140A and eye-tracking light 140B onto the optical sensor 152 as a converging beam.

[0124] Optical sensor 152 and SLM 202 are arranged such that their principal planes are orthogonal to each other. Module 224 also includes a light-discriminating surface 212 (which may be a spectrally selective surface and / or a polarization selective surface), which is inclined relative to the principal planes of optical sensor 152 and SLM 202, preferably at a 45° angle. The light-discriminating properties of surface 212 cause it to reflect or transmit image light 159A, image light 159B, and image light 159C, and transmit or reflect eye-tracking light 140A and eye-tracking light 140B.

[0125] exist Figure 9In the non-limiting example configuration shown, optical sensor 152 is arranged with its principal plane parallel to surface 104 (and SLM 202 is arranged with its principal plane orthogonal to surface 104). Surface 212 is configured to reflect image light 159A, image light 159B, and image light 159C and transmit eye-tracking light 140A and eye-tracking light 140B. The reflected image light 159A, image light 159B, and image light 159C reach optics 226, which collimates the image light 159A, image light 159B, and image light 159C to generate collimated light 160A, collimated light 160B, and collimated light 160C. Then, collimated beams 160A, 160B, and 160C are coupled into LOE 102 via optical coupling configuration 136, thereby guiding the collimated beams through LOE 102 in the positive direction via internal reflection until they reach, for example, a partially reflective surface. Figure 9 An optical coupling configuration (not shown) is used to couple a certain proportion of the light intensity out of the LOE 102.

[0126] Lighting from the eye (for simplicity, Figure 9 Only beams 126A and 126B emanating from the center of EMB 114 are shown in the image, collimated by a coupler. Figure 9 Not shown, but it could be, for example Figure 2A The collimator-coupler 120 collimates and couples the collimated light 132A and 132B generated by the collimator-coupler into the LOE 102, so that the collimated light 132A and 132B generated by the collimator-coupler are captured within the LOE 102 by internal reflection. The light 132A and 132B propagate in opposite directions along the LOE 102 by repeated internal reflections between surfaces 104 and 106 until they reach the optical coupling configuration 136, which reflects the light 132A and 132B to couple the light 132A and 132B out of the LOE 102 as coupled-out light 140A and coupled-out light 140B. The coupled-out light 140A and coupled-out light 140B are then focused by the optics 226 to generate converging beams 148A and 148B. Converging beams 148A and 148B reach surface 212, which transmits the converging beams 148A and 148B, so that the converging beams 148A and 148B reach optical sensor 152.

[0127] although Figure 9A specific, non-limiting arrangement of module 224 is shown, in which optical sensor 152 is arranged with its main plane parallel to surface 104; however, an arrangement in which the positions of optical sensor 152 and SLM 202 are interchanged is also possible. In such a configuration, surface 212 operates to transmit image light 159A, image light 159B, and image light 159C and reflect eye-tracking light 140A and eye-tracking light 140B.

[0128] It should also be noted that, since a single optical coupling configuration 136 is used to couple image light into LOE 102 and to couple eye-tracking light out of LOE 102, the optical coupling configuration 136 can have general reflective characteristics, making it reflective for all types of incident light, regardless of the spectral and / or polarization state of the incident light.

[0129] Although the embodiments of the optical system described so far involve an optical coupling configuration implemented for coupling image light (from the image projector) out of the LOE 102 using a set of partially reflective surfaces 108, the partially reflective surfaces 108 represent only one non-limiting optical coupling configuration, and other optical coupling configurations can be used to couple eye-tracking light into the LOE 102 and image light out of the LOE 102. The optical coupling configuration can be any optical coupling arrangement that deflects a portion of the image incident radiation (from the image projector) that has already propagated within the LOE 102 via internal reflection at an angle, such that a portion of the deflected image incident radiation exits the LOE 102. Other examples of such suitable optical coupling arrangements include, but are not limited to, one or more diffractive optical elements arranged on either surface 104 or surface 106. Furthermore, although only two partial reflective surfaces 108 are shown for simplicity, the optical coupling configuration (when implemented as a set of partial reflective surfaces) can include any number of such partial reflectors supported by the optical design of the device, including implementations using five or more partial reflectors or ten or more partial reflectors.

[0130] Thus far, an implementation of the optical system has been described in the context of a light-guiding optical element (LOE) configured to guide image light (injected from image projector 202) via internal reflection. Such an implementation has particular value when used in AR and / or VR applications, where AR / VR images are generated by a compact image projector with a small aperture, which is multiplied by the LOE to produce a large aperture. As described in the background section, aperture multiplication in one dimension is developed based on a plate of parallel planes of transparent material in which the image propagates via internal reflection. Note that aperture multiplication in two dimensions has also been developed using various optical waveguide configurations.

[0131] An example of a two-dimensional (2D) aperture multiplier employs a pair of optical waveguides. The first waveguide has two pairs of parallel main outer surfaces forming a rectangular cross-section. A first set of mutually parallel partially reflective surfaces is inclined along the elongation direction of the first waveguide. A second waveguide, optically coupled to the first waveguide, has a pair of parallel main outer surfaces forming a plate-like waveguide. A second set of mutually parallel partially reflective surfaces is inclined along the main outer surface of the second waveguide. Furthermore, the plane containing the first set of partially reflective surfaces is preferably inclined to the plane containing the second set of partially reflective surfaces. The optical coupling between the two optical waveguides, and the arrangement and configuration of the two sets of partially reflective surfaces, ensure that when an image is coupled into the first optical waveguide at an initial propagation direction with coupling angles inclined to the two pairs of parallel main outer surfaces of the first optical waveguide, the image propagates along the first optical waveguide (i.e., in two dimensions) through quadruple internal reflection, wherein a portion of the intensity of the image reflected at the first set of partially reflective surfaces couples out of the first optical waveguide and into the second optical waveguide, and then propagates through double internal reflection within the second optical waveguide (i.e., in one dimension), wherein a portion of the intensity of the image reflected at the second set of partially reflective surfaces couples out of the second optical waveguide, becoming the visible image seen by the viewer's eye. Further details of such a two-dimensional aperture multiplier can be found in various patent documents, including, for example, U.S. Patent No. 10,564,417, the entire contents of which are incorporated herein by reference.

[0132] In another example of a two-dimensional aperture multiplier, a first optical waveguide has two pairs of parallel main outer surfaces forming a plate-shaped waveguide. A first set of mutually parallel internal partial reflective surfaces traverses the first optical waveguide at an angle to the two pairs of parallel main outer surfaces. A second optical waveguide also has two pairs of parallel main outer surfaces. A second set of mutually parallel internal partial reflective surfaces traverses the second optical waveguide at an angle to the two pairs of parallel main outer surfaces. Furthermore, the plane containing the first set of partial reflective surfaces is inclined or perpendicular to the plane containing the second set of partial reflective surfaces. The optical coupling between the two optical waveguides, and the arrangement and configuration of the two sets of partially reflective surfaces, ensure that when an image is coupled into the first optical waveguide, it propagates in a first guiding direction via double internal reflection within the first optical waveguide between one of the two outer surfaces. A portion of the image intensity reflected at the first set of partially reflective surfaces is coupled out of the first optical waveguide and into the second optical waveguide. It then propagates in a second guiding direction (inclined to the first guiding direction) via double internal reflection within the second optical waveguide between the outer surface of the second optical waveguide and one of the outer surfaces. A portion of the image intensity reflected at the second set of partially reflective surfaces is coupled out of the second optical waveguide, becoming the visible image seen by the viewer's eye. Further details of such a two-dimensional aperture multiplier can be found in various patent documents, including, for example, U.S. Patent No. 10,551,544, the entire contents of which are incorporated herein by reference.

[0133] The eye-tracking technology according to embodiments of this disclosure can be applied to two-dimensional aperture multipliers. For example, Figure 2A The collimator-coupler 120 and optical coupling configuration 136 can be arranged in any of the aforementioned second optical waveguides (which can be similar to the LOE 102 described herein) for 2D aperture multiplication (expansion). The eye-tracking light will propagate through the second optical waveguide via internal reflection and will be coupled into the corresponding first optical waveguide via optical coupling configuration 136, whereby the eye-tracking light travels through the first optical waveguide via internal reflection. Another optical coupling configuration (similar to optical coupling configuration 136) can be arranged in the first optical waveguide to couple the eye-tracking light toward the imaging module located at the exit aperture of the first optical waveguide.

[0134] Although embodiments of this disclosure have been described in the context of illumination arrangement 214, which is configured to illuminate the eye with light preferably in the NIR region of the electromagnetic spectrum, embodiments of this disclosure should not be limited to illumination arrangements that emit eye-tracking light in any particular region of the electromagnetic spectrum. The use of NIR light for eye-tracking purposes is for illustrative purposes to provide a clearer explanation of the construction and operation of various devices of this disclosure. Other types of light may also be used for eye-tracking purposes, including but not limited to visible light, light in the infrared region, and ultraviolet (UV) light. In embodiments of illumination arrangement 214 that illuminate the eye with visible light, it may be advantageous to arrange the light source to concentrate illumination on areas of the eye less sensitive to visible light (e.g., the sclera) to avoid bombarding the eye with non-image visible light. In the embodiment of lighting arrangement 214 that uses UV light to irradiate the eyes, precautions should be taken to reduce or minimize the eyes’ exposure to harmful UV radiation, for example by setting limits on the intensity / power of the UV beam received in a given area of ​​the eye for a given duration (e.g., less than 1 milliwatt per square centimeter for a period of more than 1000 seconds for UV light with wavelengths in the range of 315 nm to 400 nm).

[0135] According to certain non-limiting implementations, various optical systems of this disclosure can be replicated to simultaneously track both eyes of an object and project images onto both eyes. Enhanced tracking stability and continuity can be achieved by combining data from two eye trackers. For example, when the eyes move, the trackable portion of the eye may be visible to the tracker in one eye but not in the other. If a tracking algorithm that tracks trackable features is used, simultaneous tracking of both eyes allows continuous tracking to be maintained during periods when only one eye tracker can track blind spots.

[0136] In the case of a binocular optical system, each eye has its own image projection and eye-tracking device, and various processing and power components can optionally be shared between the two eye-tracking systems. As described above, eye-tracking information collected by the binocular eye-tracking device can be fused to provide enhanced tracking stability and continuity.

[0137] Various embodiments of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0138] As used in this article, unless the context clearly indicates otherwise, the singular form and “the” include plural references.

[0139] The word "exemplary" is used here to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations and / or as excluding combinations of features from other implementations.

[0140] It should be understood that certain features of the invention described in the context of a single embodiment for clarity may also be provided in combination in a single embodiment. Conversely, various features of the invention described in the context of a single embodiment for brevity may also be provided individually or in any suitable sub-combination or as suitably as in any other described embodiment of the invention. Certain features described in the context of various embodiments are not considered essential features of those embodiments unless the embodiment would be inoperable without these elements.

[0141] In drafting the appended claims without multiple references, this is done only to accommodate the formal requirements of jurisdictions that do not permit such multiple references. It should be noted that all possible combinations of features implied by making multiple references in the claims are explicitly contemplated and should be considered part of the invention.

[0142] Although the invention has been described in conjunction with specific embodiments thereof, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be apparent. Therefore, the invention is intended to include all such alternatives, modifications, and variations falling within the spirit and broad scope of the appended claims.

Claims

1. An optical system, comprising: A light-transmitting substrate having at least two main surfaces, wherein the at least two main surfaces are arranged such that a first main surface of the main surfaces faces the viewer’s eye, for guiding light through internal reflection between the two main surfaces of the light-transmitting substrate; An optical coupling configuration associated with the light-transmitting substrate is used to couple image light, which corresponds to the collimated image and is guided by internal reflection between the two main surfaces, out of the light-transmitting substrate for viewing by the viewer's eye. A first optical coupling configuration includes a curved surface embedded within the light-transmitting substrate, the curved surface being reflective to at least one wavelength of light reflected from the eye, the curved surface having a curvature sufficient to collimate the light reflected from the eye to produce collimated light, and being oriented to couple the collimated light into the light-transmitting substrate so as to propagate within the light-transmitting substrate by internal reflection. A second optical coupling configuration associated with the light-transmitting substrate, the second optical coupling configuration being configured to couple the collimated light out of the light-transmitting substrate as coupled light; An optical sensor is arranged to sense the coupled light; as well as At least one processor, which communicates with the optical sensor and is configured to process signals from the optical sensor to obtain the current gaze direction of the eye.

2. The optical system according to claim 1, wherein, The optical coupling configuration includes a plurality of partially reflective surfaces, which are arranged obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate.

3. The optical system according to claim 1, wherein, The optical coupling configuration includes a diffraction element associated with one of the two main surfaces of the light-transmitting substrate.

4. The optical system according to claim 1, wherein, The light from the eye is in a first spectrum, and the image light is in a second spectrum.

5. The optical system according to claim 1, further comprising: An image projector is used to generate the collimated image.

6. The optical system according to claim 1, wherein, The second optical coupling configuration is further configured to couple the image light corresponding to the collimated image into the light-transmitting substrate so that it propagates within the light-transmitting substrate via internal reflection.

7. The optical system according to claim 6, further comprising: A selectively reflective surface that transmits or reflects image light corresponding to the collimated image toward the second optical coupling configuration, and reflects or transmits coupled light from the second optical coupling configuration toward the optical sensor.

8. The optical system according to claim 1, further comprising: An optical coupling configuration associated with the light-transmitting substrate is used to couple the image light corresponding to the collimated image into the light-transmitting substrate so that it can propagate within the light-transmitting substrate by internal reflection.

9. The optical system according to claim 8, wherein, The optical coupling configuration reflects the image light corresponding to the collimated image and transmits the collimated light propagating within the light-transmitting substrate toward the second optical coupling configuration.

10. The optical system according to claim 8, wherein, The optical coupling configuration is coupled into the light-transmitting substrate so that the image light corresponding to the collimated image propagating within the light-transmitting substrate reaches the second optical coupling configuration via internal reflection, wherein the second optical coupling configuration transmits the image light corresponding to the collimated image propagating within the light-transmitting substrate.

11. The optical system according to claim 1, further comprising: An optical device arranged in an optical path from the second optical coupling configuration to the optical sensor for forming at least one image of at least a portion of the eye on the optical sensor.

12. The optical system of claim 11, further comprising: An image projector comprising a spatial light modulator for generating image light, wherein the optics form part of the image projector and collimate the image light generated by the spatial light modulator to generate the collimated image.

13. The optical system according to claim 1, wherein, The first optical coupling configuration reflects light from the eye and transmits image light corresponding to the collimated image.

14. The optical system according to claim 1, wherein, The curved surface has a curvature sufficient to make the light from the eye accurate up to infinity.

15. The optical system according to claim 14, wherein, The curvature is a function of the distance between the eye and the first main surface of the main surface of the light-transmitting substrate.

16. The optical system according to claim 1, wherein, The first optical coupling configuration is arranged obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate.

17. The optical system according to claim 1, further comprising: The second light-transmitting substrate has at least two main surfaces, wherein one of the two main surfaces of the second light-transmitting substrate is associated with another of the two main surfaces of the light-transmitting substrate, and wherein at least one optical element of the first optical coupling configuration is arranged within the second light-transmitting substrate.

18. The optical system according to claim 17, wherein, The at least one optical element of the first optical coupling configuration is arranged in the region of the second light-transmitting substrate located in front of the eye, such that the normal of the at least one optical element of the first optical coupling configuration approximately reaches the center of the pupil of the eye.

19. The optical system according to claim 17, wherein, The two main surfaces of the light-transmitting substrate are parallel to each other, and the two main surfaces of the second light-transmitting substrate are parallel to each other and parallel to the two main surfaces of the light-transmitting substrate.

20. The optical system according to claim 17, wherein, At least one of the main surfaces of the second light-transmitting substrate is a curved surface.

21. The optical system according to claim 17, wherein, The second light-transmitting substrate is formed as a lens for applying optical power to light from a real-world scene.

22. The optical system according to claim 1, further comprising: A second light-transmitting substrate has at least two main surfaces, including a first main surface and a second main surface, wherein the first main surface of the second light-transmitting substrate is associated with the second main surface of the light-transmitting substrate, and wherein the first optical coupling configuration includes: At least one collimating element, said at least one collimating element being disposed within the second light-transmitting substrate, and Partial reflectors, wherein the partial reflectors are arranged obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate, wherein the partial reflectors: Light from the eye is transmitted toward the at least one collimating element such that the at least one collimating element produces collimated light based on the light from the eye. The collimated light generated by the at least one collimating element is reflected so as to couple the collimated light into the light-transmitting substrate so as to propagate within the light-transmitting substrate by internal reflection.

23. The optical system according to claim 1, further comprising: A second light-transmitting substrate has at least two main surfaces, including a first main surface and a second main surface, wherein the second main surface of the second light-transmitting substrate is associated with the first main surface of the light-transmitting substrate, and wherein the first optical coupling configuration includes: At least one collimating element, disposed within the second light-transmitting substrate, is provided for collimating light from the eye to produce collimated light. A first reflector is disposed obliquely within the second light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate. The first reflector deflects the collimated light out of the second light-transmitting substrate and into the light-transmitting substrate. A second reflector is disposed obliquely within the light-transmitting substrate relative to the two main surfaces of the light-transmitting substrate. The second reflector deflects light from the first reflector to couple the collimated light into the light-transmitting substrate so that it can propagate within the light-transmitting substrate by internal reflection.

24. The optical system according to claim 1, further comprising: An illumination arrangement is configured to illuminate the eye with illumination light such that the eye reflects the illumination light as reflected light, wherein the reflected light corresponds to light from the eye collimated by the first optical coupling configuration.

25. The optical system according to claim 1, wherein, The second optical coupling configuration includes a reflective surface that deflects the collimated light out of the light-transmitting substrate.

26. The optical system according to claim 1, wherein, The second optical coupling configuration includes a planar opening end of the light-transmitting substrate, wherein the opening end is formed by cutting the light-transmitting substrate along a plane orthogonal to the two main surfaces of the light-transmitting substrate.

27. The optical system according to claim 1, wherein, The at least one processor is configured to receive signals from the optical sensor via one or more communication networks.

28. An optical system comprising: A light-guiding optical element having at least two main surfaces, the at least two main surfaces being arranged such that a first main surface of the main surfaces faces the viewer’s eye, for guiding light through internal reflection between the two main surfaces of the light-guiding optical element; Multiple partially reflective surfaces are arranged obliquely within the light guide optical element relative to the two main surfaces, for coupling image light corresponding to the collimated image, guided by internal reflection between the two main surfaces, out of the light guide optical element for viewing by the viewer's eye; A curved surface, disposed within the light-guiding optical element, is selectively reflective and selectively applies optical power to incident light. The curved surface is reflective of at least one wavelength of light reflected from the eye. The curved surface has a curvature sufficient to collimate the light reflected from the eye to produce collimated light, and is oriented to reflect the collimated light so as to couple the collimated light into the light-guiding optical element for guidance via internal reflection between the two main surfaces. The image light corresponding to the collimated image is transmitted through internal reflection guided between the two main surfaces and substantially without applying optical power to the image light; Imaging system; as well as An optical coupling configuration associated with the light-guiding optical element, the optical coupling configuration being configured to couple the collimated light from the light-guiding optical element to the imaging system, wherein the imaging system forms an image of the eye based on the collimated light coupled from the light-guiding optical element through the optical coupling configuration.

29. The optical system according to claim 28, wherein, The imaging system includes an optical sensor for sensing collimated light coupled from the light-guiding optical element via the optical coupling configuration, and wherein the optical system further includes a processing system that communicates with the optical sensor and is configured to process signals from the optical sensor to obtain the current gaze direction of the eye.

30. An optical system comprising: A light-transmitting substrate having at least two main surfaces, wherein the at least two main surfaces are arranged such that a first main surface of the main surfaces faces the viewer’s eye, for guiding light through internal reflection between the two main surfaces of the light-transmitting substrate; A first optical coupling configuration includes a curved surface embedded in the light-transmitting substrate, the curved surface being reflective to at least one wavelength of light reflected from the eye, the curved surface having a curvature sufficient to collimate the light reflected from the eye to produce collimated light, and being oriented to couple the collimated light into the light-transmitting substrate so as to propagate within the light-transmitting substrate by internal reflection. A second optical coupling configuration associated with the light-transmitting substrate, the second optical coupling configuration being configured to couple the collimated light out of the light-transmitting substrate as coupled light; A focusing optics device, which is associated with the second optical coupling configuration and operates to convert the coupled-out light into a converging beam of trapped light; An optical sensor is arranged to sense the captured light; as well as At least one processor, which communicates with the optical sensor and is configured to process signals from the optical sensor to obtain the current gaze direction of the eye.

Images