Eye tracking device and method based on eye imaging through a light-guided optical element

Abstract

The light-transmitting substrate has a parallel plane in which a first surface is positioned among the parallel planes facing the eye. An optical element is associated with the first surface and applies an optical output to the first type of incident light to collimate the incident light, and substantially does not apply an optical output to the second type of incident light. An optical coupling assembly is associated with the substrate and is configured to internally couple a portion of the first type of collimated light incident on the first surface to propagate within the substrate, and externally couple a portion of the second type of light propagating within the substrate. The optical element associated with the substrate converts the first type of collimated light into a converged beam of light detected by an optical sensor. A processor processes the signal from the optical sensor to derive the current line of sight of the eye.

Description

Technology Field

[0001] Cross-reference of related applications

[0002] This application claims priority from U.S. Provisional Application No. 62 / 867,249 filed on June 27, 2019 and U.S. Provisional Application No. 62 / 869,582 filed on July 2, 2019, the disclosures of which are incorporated herein by reference in their entirety.

[0003] Technology field

[0004] The present invention relates to eye tracking. Background Technology

[0005] Optical arrays for Near Eye Displays (NEDs), Head Mounted Displays (HMDs), and Head Up Displays (HUDs) require a large aperture to cover the area where the observer's eyes are located (commonly referred to as the eye motion box or EMB). To implement a compact device, the image to be projected onto the observer's eyes is generated by a small optical image generator (projector) with a small aperture that is multiplied to create a large aperture.

[0006] An approach to aperture multiplication in one dimension has been developed based on a slab of parallel planes of transparent material in which an image is propagated by internal reflection. A portion of the image wavefront is coupled to the outside of the slab using an angled partial reflector or by using a diffractive optical element on one surface of the slab. Such a slab is referred to herein as a light-guided optical element (LOE), a light-transmitting substrate, or a waveguide. The principle of such aperture multiplication is schematically illustrated in FIG. 1, which shows a light-guided optical element (20) having a pair of parallel planes (26, 26A) for guiding light by internal reflection. A projected image (18), as schematically illustrated herein with an illumination beam (18) containing sample rays (18A and 18B) that expand the beam, is coupled within the light-guided optical element (20), as schematically illustrated herein with a first reflective plane (16), to generate a reflected ray (28) that is trapped by internal reflection within the substrate and also generates a ray (30). The image propagates along the substrate by repeated internal reflection, and at an angle of inclination (26, 26A) with respect to the parallel plane (26, 26A) α sur It strikes a series of partial reflective surfaces (22) and, in this case, a portion of the image intensity is reflected and coupled to the outside of the substrate as rays (32A, 32B) directed toward the observer's eye (24). To minimize unwanted reflections that may result in a ghost image, it is preferable that the partial reflective surfaces (22) be coated to have a low reflectivity for a first angle of incidence range and a desired partial reflectivity for a second angle of incidence range, in this case (here, angle β ref A ray with a small slope relative to the normal to the partial reflection surface (22) (represented by) is split to produce a reflected ray for external coupling, while a ray with a high slope (relative to the normal) is transmitted with negligible reflection.

[0007] The projected image (18) is a collimated image, that is, in this case, each pixel is represented by a parallel beam of light at the same corresponding angle as light from a scene far from the observer (the collimated image is referred to as "colimated to infinity"). The image is simply represented here by a corresponding beam at a single point of the image, typically at the center of the image, but actually creates a field of view corresponding to parts of the image reaching the observer's eye (24) in different directions, including angle ranges for each side of this central beam that are combined within the substrate at a corresponding angle range and similarly combined at a corresponding angle.

[0008] Optical features that can be useful for NED, HMD, or HUD designs are eye tracking, or detecting the direction in which an observer's eyes are looking relative to head direction (generally referred to as gaze direction). Past eye tracking approaches relied on imaging the EMB using one or more off-axis cameras viewed from the side toward the EMB. To minimize user discomfort, the cameras must be relatively small, which can limit EMB imaging performance. Due to the small camera size, coupled with the general difficulty of deriving gaze direction from EMB images sampled at high off-axis angles, the performance of these eye tracking approaches is relatively low.

[0009] An aspect of the present invention provides an eye tracker and a corresponding method for tracking the direction of a human eye's gaze based on eye imaging through a light-guided optical element, and is particularly suitable for integration as part of a NED, HMD, or HUD.

[0010] According to the teaching of one embodiment of the present invention, a light-transmitting substrate having at least two parallel principal surfaces for guiding light by internal reflection, wherein the first surface among the principal surfaces is positioned to face an eye; an optical element associated with the first surface among the principal surfaces, wherein the optical element is configured to apply a light output to the incident light according to at least one characteristic of the incident light, such that the optical element applies a light output to the first type of incident light to collimate the first type of incident light and the optical element does not substantially apply a light output to the second type of incident light; an optical coupling configuration associated with the substrate, wherein the optical coupling configuration is collimated by the optical element to propagate within the substrate and is configured to internally couple a portion of the first type of light incident on the first surface among the principal surfaces and externally couple a portion of the second type of light propagating within the substrate; an optical body associated with the substrate and configured to convert the first type of collimated light into a convergent beam of captured light; and an optical sensor positioned to detect the captured light. A device is provided comprising at least one processor that is electrically associated with an optical sensor and configured to process a signal from the optical sensor to derive the current direction of gaze of the eye.

[0011] Optionally, at least one characteristic of the incident light includes the polarization direction of the incident light.

[0012] Optionally, at least one characteristic of the incident light includes a region of the electromagnetic spectrum occupied by the incident light.

[0013] Optionally, at least one characteristic of the incident light includes the polarization direction of the incident light and the region of the electromagnetic spectrum occupied by the incident light.

[0014] Optionally, the first type of light includes a component of light polarized in a first polarization direction, and the second type of light is polarized in a second polarization direction.

[0015] Optionally, the first type of light is within the first optical spectrum, and the second type of light is within the second optical spectrum.

[0016] Optionally, the first type of light includes a component of light polarized in a first polarization direction and is within the first optical spectrum, and the second type of light is polarized in a second polarization direction and is within the second optical spectrum.

[0017] Optionally, the device further includes a polarizer associated with a second surface among the main surfaces of the substrate.

[0018] Optionally, a first surface among the principal surfaces is disposed on the substrate at pupil distance from the eye, and the optical element has a focal length approximately equal to pupil distance.

[0019] Optionally, the device further comprises a second optical coupling configuration associated with an optical body, wherein the second optical coupling configuration is configured to externally couple a portion of a first type of light propagating within a substrate, the externally coupled light is received by the optical body, and is configured to internally couple a portion of a second type of light from a display source to propagate within the substrate by internal reflection.

[0020] Optionally, the device further includes a lighting array arranged to illuminate the eye with a first type of light.

[0021] Optionally, the device further includes an image projector coupled to the substrate to introduce a second type of collimating light corresponding to the image into the substrate, wherein the second type of internally coupled collimating light propagates by internal reflection within the substrate and is coupled to the outside of the substrate toward the eye by an optical coupling configuration.

[0022] Optionally, the image projector includes a reflective display device that generates polarized light in response to illumination from a polarizing light source, and the polarized light generated by the reflective display device is collimated by an optical body.

[0023] Optionally, the optical coupling configuration includes a plurality of partial reflective surfaces disposed within the substrate that are inclined relative to the main surface of the substrate.

[0024] Optionally, a first type of light propagates within the substrate in a first propagation direction, and a second type of light propagates within the substrate in a second propagation direction opposite to the first propagation direction.

[0025] Additionally, according to one embodiment of the teaching of the present invention, a light-transmitting substrate having a pair of parallel principal surfaces for guiding light by internal reflection, wherein the first surface among the principal surfaces is positioned to face the eye of a viewer; a lens associated with the first surface among the principal surfaces, wherein the lens is configured to apply a light output to a first type of incident light to collimate the first type of incident light, wherein the first type of incident light includes a component of light having polarization in a first polarization direction within a first optical spectrum and is configured not to substantially apply a light output to a second type of incident light, wherein the second type of incident light has polarization in a second optical spectrum and is polarized in a second polarization direction; an illumination array arranged to illuminate the eye with the first type of light such that a portion of the first type of light is reflected back toward the lens by the eye and collimated by the lens; an optical module comprising a reflective display device that generates a second type of light corresponding to an image in response to illumination from a light source, an optical body configured to collimate the light generated by the reflective display device to generate a second type of collimated light, and an optical sensor; An optical coupling structure configured to combine a second type of collimated light with a substrate and propagate within the substrate by internal reflection in a first propagation direction; a plurality of partial reflective surfaces disposed within the substrate inclined with respect to a main surface of the substrate, wherein the plurality of partial reflective surfaces are configured to externally combine a portion of the second type of light propagating in a first propagation direction within the substrate and are configured to internally combine a portion of the first type of collimated light incident on a first surface among the main surfaces to propagate in a second propagation direction within the substrate, and the optical coupling structure is further configured to externally combine the first type of propagation light;An apparatus is provided comprising at least one processor electrically coupled to an optical sensor, wherein the optical body of the optical module is further configured to receive external coupled light by an optical coupling configuration and convert the external coupled light into a converged beam of captured light, the optical sensor is configured to detect captured light, and at least one processor is configured to process a signal from the optical sensor to derive the current gaze direction of the eye.

[0026] Additionally, according to one embodiment of the teaching of the present invention, a first light-transmitting substrate having at least two substantially parallel principal surfaces for guiding light by internal reflection, wherein the first surface among the principal surfaces is positioned to face the eye of a viewer; at least a partial reflective surface disposed within the first substrate inclined with respect to the principal surface, wherein the at least partial reflective surface is configured to combine an incident light ray incident on the first surface among the principal surfaces within an internal coupling region and propagate by internal reflection within the first substrate, wherein the incident light ray is in a first optical spectrum and radiates from the eye in response to illumination of the eye, and the incident light ray comprises at least a first set of light rays and a second set of light rays, wherein the first set of light rays has an angular distribution that extends at least a portion of the internal coupling region in a first dimension and the second set of light rays extends at least a portion of the internal coupling region in a second dimension; and an external coupling array configured to externally couple the light ray propagating within the first substrate. An optical module comprising: at least one lens having a first focal length in a first dimension of the lens and a second focal length in a second dimension of the lens, configured to convert an external coupling ray corresponding to a first set of rays into a non-converging beam of captured light having an angular distribution representing the angular distribution of the first set of rays, and configured to convert an external coupling ray corresponding to a second set of rays into a convergent beam of captured light; and an optical sensor positioned at a distance substantially equal to the first focal length from the lens and configured to detect captured light; and at least one processor electrically coupled to the optical sensor and configured to process a signal from the optical sensor to derive the current line of sight of the eye.

[0027] Optionally, the device further includes a lighting array arranged to illuminate the eye with light of a first optical spectrum.

[0028] Optionally, the device comprises a second light-transmitting substrate having a plurality of surfaces including mutually parallel first and second principal surfaces for guiding light by internal reflection, wherein the first principal surface of the second substrate is positioned facing the eye and the second principal surface of the second substrate is positioned facing the first surface among the principal surfaces of the first substrate; and further comprises an external coupling configuration associated with the second substrate, wherein the external coupling configuration is configured to couple a portion of light of a second optical spectrum, which is different from the first optical spectrum and propagates within the second substrate, to the outside of the second substrate toward the eye.

[0029] Optionally, the device further comprises an image projector coupled to a second substrate and configured to generate collimation light of a second optical spectrum corresponding to an image, wherein the collimation light propagates by internal reflection within the second substrate and is coupled to the outside of the second substrate toward the eye by an external coupling component.

[0030] Optionally, the device further includes an internal coupling array associated with a second substrate configured to combine an image projector and collimation light generated by the image projector with the second substrate.

[0031] Optionally, the external coupling configuration includes a plurality of partial reflective surfaces disposed within the second substrate that are inclined with respect to the main surface of the second substrate.

[0032] Optionally, the external coupling configuration includes a diffraction optical element associated with one of the main surfaces of the second substrate.

[0033] Unless otherwise defined herein, all technical and / or scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. Methods and materials similar or equivalent to those described herein may be used to carry out or test embodiments of the invention, but exemplary methods and / or materials are described below. In the event of a conflict, the patent specification containing definitions shall prevail. Furthermore, materials, methods, and examples are merely illustrative and are not intended to be limiting. Brief explanation of the drawing

[0034] Some embodiments of the present invention are described herein merely by way of example with reference to the accompanying drawings. Specifically, with detailed reference to the drawings, it is emphasized that the details depicted are for illustrative purposes only and are intended for exemplary discussion of embodiments of the present invention. In this regard, the description taken together with the drawings makes it clear to those skilled in the art how embodiments of the present invention may be practiced. Now, pay attention to drawings in which similar reference symbols or characters represent corresponding or similar components. In the drawings: FIG. 1 is a schematic side view of a conventional light-guided optical element utilizing a partial reflective surface for use in a near-eye display; FIG. 2 is a schematic side view of a device configured and operated according to an embodiment of the present invention for displaying an image and tracking the direction of a person's gaze, illustrating the propagation of light from the eye to an image projector through a light-transmitting substrate; FIG. 3 is a schematic side view of an alternative configuration of the device of FIG. 2 in which an optical coupling structure for coupling light between a light-transmitting substrate and an image projector is implemented as a reflective surface; FIG. 4 is a schematic side view of the device of FIG. 2 illustrating the propagation of image light from an image projector and the propagation of light of an external scene to the eye; FIG. 5 is a schematic exploded plan view of the image projector of FIG. 2 to 4 illustrating the propagation of image light to the output section of the image projector; FIG. 6 is a schematic exploded plan view of the image projector of FIG. 5 illustrating the propagation of eye-tracking light by an optical sensor; FIG. 7 is a schematic exploded plan view of the image projector of FIG. 5 and 6, modified by the addition of a delay plate and illustrating the propagation of gaze-tracking light to an optical sensor; FIG. 8 is a schematic partial perspective view of the device of FIG. 1 implemented in a glasses form factor; FIG. 9 is a schematic side view of a device for displaying an image and tracking the direction of a person's gaze, having a first light-transmitting substrate for gaze tracking and a second light-transmitting substrate for image projection, configured and operated according to another embodiment of the present invention, illustrating the propagation of light from the eye to an image projector through the second light-transmitting substrate; FIG. 10 is a schematic partial side view of the first light-transmitting substrate of FIG. 9 illustrating the coupling of a light beam from an eye within the first light-transmitting substrate; FIG. 11 is a schematic partial side view similar to FIG. 10, illustrating a plurality of rays expanding a light beam from an eye in a first dimension; FIG. 12 is a schematic side view of the optical module of FIG. 9, illustrating a ray corresponding to the ray of FIG. 11 directed toward an optical sensor by a lens; FIG. 13 is an isometric view of a first light-transmitting substrate of FIG. 9 and 10 illustrating a plurality of rays extending a light beam from an eye in a second dimension; FIG. 14 is a schematic side view of the optical module of FIG. 9, illustrating that a ray corresponding to the ray of FIG. 13 is focused by a lens onto an optical sensor; FIG. 15 is a schematic exploded plan view of the image projector of FIG. 9, illustrating the propagation of image light and gaze tracking light to the output section of the image projector. Specific details for implementing the invention

[0035] Embodiments of the present invention provide various devices and corresponding methods for tracking the gaze direction of a human eye based on imaging the eye and / or identifying the angular distribution of light reflected by the eye through a light-guided optical element.

[0036] The principles and operation of various eye-tracking devices according to the present invention can be better understood by referring to the drawings accompanying the explanation.

[0037] Before describing at least one embodiment of the present invention in detail, it should be understood that the present invention is not necessarily limited to the details of the configuration and the details of the configuration and arrangement of the components and / or methods presented in the following description and / or illustrated in the drawings and / or examples. The present invention may be applicable to other embodiments or may be practiced or performed in various ways.

[0038] As an introduction, in various application fields, particularly in the context of head-up or near-eye displays, it is useful to provide an eye-tracking array for determining the direction of a user's gaze. One general approach to performing eye tracking is to typically sample an image of the eye to determine the position of the pupil within the image and derive the direction of the eye. It would be particularly advantageous to use a light-guided optical element that operates on a principle similar to that of FIG. 1 to sample the eye-tracking image.

[0039] A gaze tracking solution utilizing a light-guided optical element operating according to this principle or a similar principle is described herein. In a set of solutions according to a specific embodiment of the present invention, the eye is imaged by recombining light reflected from the eye (referred to as light of the first type) into a light-guided optical element, thereby propagating the light along a reverse path through the light-guided optical element in the reverse propagation direction of image light (referred to as light of the second type) from the image projector and focusing it on an optical sensor placed in the image projector, wherein a signal generated by the optical sensor in response to the detection of the light is processed by a processing system to derive the gaze direction. Since the eye is not located at infinity from the light-guided optical element (rather, typically located at a pupil distance of about 20 mm), the light reflected from the eye is collimated by the optical element, preferably by a polarization and / or a spectrum selection lens that distinguishes between the first and second types of light, before being combined into the light-guided optical element to accurately derive the gaze direction from the light focused on the optical sensor.

[0040] In another set of solutions according to an embodiment of the present invention, the line of sight direction is preferably determined through a special partial reflective surface in a dedicated light-guided optical element separated from the LOE through which a projected image is propagated, in which uncolloced light reflected from the eye is coupled within the light-guided optical element, and thereby the internally coupled light is coupled to an optical module comprising two focal lengths in each orthogonal dimension, which propagates along a reverse path through the light-guided optical element and guides the externally coupled light to an optical sensor.

[0041] Now, referring to the drawings, FIGS. 2 through 8 illustrate various aspects of the structure and operation of a device, generally denoted as 100, configured and operated according to various embodiments of the present invention for displaying an image through a collimating optical element (112) (hereinafter referred to as lens (112)) placed between an eye (110) and a light-guiding optical element (LOE) (102) and deriving the direction of gaze of a human eye (110). The LOE (102) is formed of a transparent material and has a pair of parallel planes (main surfaces of a plane) (104, 106) for guiding light by internal reflection (preferably internal total reflection). One of the parallel planes (104) facing the eye (110) is disposed on the LOE (102), in which case the eye (110) is located at an eye pupil distance (ER) (111) from the corresponding plane (104) at the EMB (109). An optical coupling configuration implemented by a set of partial reflective surfaces (108) is configured to internally couple a portion of light incident on a surface (104) within an internal coupling region so as to be propagated by internal reflection (total reflection) within the LOE (102) associated with the LOE (102). In particular, the partial reflective surfaces (108) are positioned within the LOE (102) that is inclined with respect to parallel planes (104, 106) (i.e., between the surfaces (104, 106)). The internal coupling region of the LOE (102), also referred to as the "active region" or "active zone," is generally defined as an area extended by the projection of the partial reflective surfaces (108) from the plane of the surface (104).

[0042] The lens (112) is associated with the surface (104) (via optical attachment to the LOE (102)) and the lens (112) is positioned between the LOE (102) and the eye (110). It is preferable that the lens (112) have a focal length approximately equal to that of the ER (111). Light reflected from the eye (110) (in response to illumination of the eye (110) by the illumination array (138)) is collimated by the lens (112), and as a result, the collimated light is incident on the surface (104) and coupled within the LOE (102) by the partial reflective surface (108) so as to propagate within the LOE (102) by internal reflection. An optical element (140) (hereinafter referred to as the lens (140)) is associated with the LOE (102) to receive captured light propagating within the LOE (102) and to convert collimated light (a set of parallel rays) propagating within the LOE (102) into a converged beam of captured light. Preferably, the lens (140) is integrated into an optical module (126) together with an optical sensor (128) configured to detect captured light, and the lens (140) is associated with the LOE (102) through an optical coupling configuration (124) that couples captured light propagating from within the LOE (102) to outside the LOE (102) to the optical module (126). A processing system (130) comprising at least one computer-type processor (132) coupled to a storage medium (134) (e.g., computer memory, etc.) is electrically associated with an optical sensor (128) and is configured to process a signal from the optical sensor (128) to derive the current gaze direction of the eye (110).

[0043] The optical coupling configuration (124) may be any coupling array that deflects incident light out of the LOE (102) and into the optical module (126). Suitable optical coupling configurations include, but are not limited to, a reflective surface (schematically illustrated in FIG. 2) and a prism (schematically illustrated in FIG. 3).

[0044] Generally speaking, the eye (110) is illuminated by light from a light array (138). As will be discussed, the light array (138) is configured to illuminate the eye (110) with light having wavelengths outside the photoadaptation region of the electromagnetic spectrum. In other words, the light array (138) is configured to illuminate the eye (110) with light that is invisible to the human eye. Reflection from the human eye, particularly from the retina of the eye, is much higher in the near-infrared region than in the visible wavelength region. Therefore, it is preferable that the light array (138) be configured to illuminate the eye (110) with light having wavelengths in the near-infrared (NIR) region of the electromagnetic spectrum. Additionally, as will be discussed in detail in a subsequent section of the present disclosure, the lighting array (138) is also configured to illuminate the eye (110) in response to illumination from the lighting array (138), such that the light reflected by the eye (110) includes a minimum component of light having a specific polarization direction (typically p-polarized) with respect to the surface of the lens (112).

[0045] Now, specifically referring to FIGS. 2 and 3, this illustrates the traversal of a light beam from the eye (110) through the LOE (102) to the optical sensor (128). Generally, light propagating from the eye (110) to the optical sensor (128) within the LOE (102) is referred to as propagating in the reverse direction (referred interchangeably as the first / second propagation direction, first / second direction, or reverse direction) within the LOE (102), whereas image light propagating from the image projector to the eye (110) within the LOE (102) is referred to as propagating in the forward direction (referred interchangeably as the second / first propagation direction, second / first direction, or forward direction), which is opposite to the reverse direction within the LOE (102). A portion of the light intensity from the illumination array (138) incident on the eye (110) is reflected by the eye (110). The reflected light emitted from the eye (110) is schematically represented in FIGS. 2 and FIGS. 3 as sample rays (114A-114F). The light emitted from the eye (110) is collimated by a lens (112), in which case the collimated light is schematically represented as rays (116A-116F) (each ray (114A-114F) has a corresponding collimated ray (116A-116F)). Collimation rays (116A-116F) are generally incident on the surface (104) of the LOE (102) which is perpendicular to the surface (104), and are coupled within the LOE (102) by a partial reflective surface (108) to generate a reflected ray (118) (downward ray) that is trapped by internal reflection within the LOE (102), and also generate an (upward) ray (120). The light reflected from the eye (110) propagates along the substrate until it reaches an optical coupling configuration (124) (schematically illustrated as a reflective surface in FIG. 2 and a prism in FIG. 3) which couples the light (rays (118 and 120)) outside the LOE (102) to an optical module (126) as rays (122A, 122B, and 122C). The lens (140) converts the collimated external combined light (rays (122A, 122B, and 122C)) into a converged beam of captured light and focuses the external combined light (rays (122A, 122B, and 122C)) onto the optical sensor (128).

[0046] In addition to having an internally integrated lens (140) and an optical sensor (128), the optical module (126) preferably includes a component for generating an image and projecting it onto the LOE (102) for viewing by the eye (110) (similar to the projected image (18) in FIG. 1), so that the optical module (126) performs the dual functions of image projection and light focusing and detection. As to be discussed, the lens (140) also functions to collimate the light rays generated by the display device of the optical module (126).

[0047] Now, referring to FIG. 4, this illustrates the propagation of light in the forward direction within the LOE (102). Similar to FIG. 1, the projected image (142), schematically represented here as an illumination beam (142) containing sample rays (142A, 142B, and 142C) that expand the beam, is generated by an optical module (126) and coupled within the LOE (102) through an optical coupling configuration (124) (as schematically illustrated herein as a reflective surface) to generate a reflected ray (144) (upward ray) that is trapped by internal reflection within the LOE (102), and also generates a ray (146) (downward ray). The image (142) propagates along the LOE (102) by repeated internal reflection between the surfaces (104, 106), strikes the partial reflection surface (108), and in this case, a portion of the image intensity is reflected and combined outside the LOE (102) into rays (148A, 148B, and 148C) toward the eye (110). However, before reaching the eye (110), the rays (148A-148C) must pass through the lens (112).

[0048] It is important that the lens (112) applies light output to the light emitted from the eye (110) so that the lens (112) can accurately detect the light captured (by the optical sensor (128)) and process the signal from the optical sensor (128) (by the processing system (130)) to derive the current direction of gaze of the eye (110), but it is equally important that the lens (112) does not apply light output to the image light propagating from the optical module (126) to the eye (110) through the LOE (102), because if light output is applied to the light rays (148A, 148B, and 148C), the projected image (142) will be distorted when viewed by the eye (110). Therefore, it is a specific feature of the present embodiment to design the lens (112) to distinguish between two types of light represented by the lens (light reflected from the eye, represented by rays (114A-114F) propagating to the focusing and sensing components of the optical module (126), referred to as the first type of light, and image light from the image projection component, represented by rays (142A-142C), referred to as the second type of light) and to apply light output only to one of the corresponding types of light (i.e., the first type of light, i.e., the reflected light from the eye). Within the context of this document, the terms “first type of light,” “first type of light wave,” “first type of light,” “first type of light wave,” and variations thereof are used interchangeably. Additionally, within the context of this document, the terms "Type 2 light," "Type 2 light wave," "Type 2 light," "Type 2 light wave," and their variations are used interchangeably.

[0049] According to a specific preferred embodiment, a distinction is made based on at least one characteristic of the light incident on the lens (112). In other words, the lens (112) is designed to selectively apply a light output to the incident light according to at least one characteristic (feature) of the incident light. In a specific embodiment, one characteristic—e.g., the wavelength of the incident light (i.e., optical spectrum)—is used as a criterion for distinguishing between the first type and the second type of light, in another embodiment, another characteristic—e.g., the polarization direction or the polarization direction of the incident light component—is used as a criterion for distinguishing between the first type and the second type of light, and in yet another preferred embodiment, both the optical spectrum (wavelength) and the polarization direction of the incident light are used as a criterion for distinguishing between the first type and the second type of light.

[0050] Generally, it should be noted that, in contrast to the light illuminating the eye (110), the image light (142) (second type of light) has a wavelength in the photo-adapting region of the electromagnetic spectrum (i.e., 380 nanometers (nm) to about 700 nm). Therefore, the lens (112) can be designed so that the light output is applied only to light having a wavelength outside the photo-adapting region of the electromagnetic spectrum. Additionally, in various applications, it is desirable that the image light projected by the optical module (126) be linearly polarized (preferably s-polarized) in a specific polarization direction. Thus, the lens (112) can be designed to apply a light output to polarized light having a polarization direction rotated with respect to the polarization direction of the externally coupled image light projected by the optical module (126). Accordingly, it is preferable that the lens (112) be designed to be polarized and spectrally selected such that an optical output is applied to a first type of incident light wave to collimate the first type of incident light wave, and the lens (112) does not apply an optical output to a second type of incident light wave, wherein the first type of incident light wave has a component of a first polarization direction (e.g., p-polarization) and has a wavelength of a first optical spectrum (e.g., the NIR region of the electromagnetic spectrum), and the second type of incident light wave has a second polarization direction rotated with respect to the first polarization direction (e.g., s-polarization) and has a wavelength of a second optical spectrum (e.g., the photoadaptation (or visible light) region of the electromagnetic spectrum). To this end, for the first type of incident light wave, the lens (112) has a focal length approximately equal to that of the ER (111).

[0051] In the aforementioned exemplary configuration of the lens (112), the light rays (114A-114F) (first type of light) represent the p-polarized component of the light emitted from the eye (110) (with respect to the surface of the lens (112)) and have wavelengths in the NIR region of the electromagnetic spectrum, whereas the light rays (148A-148C) (second type of light) externally coupled from the LOE (102) are s-polarized (with respect to the surface of the lens (112)) and have wavelengths in the visible region of the electromagnetic spectrum. As a result of the polarization and wavelength-dependent light output distinction performed by the lens (112), the lens (112) applies light output to p-polarized NIR light waves to collimate light rays (116A-116F) (first type of light), and does not apply light output to s-polarized visible image light waves coupled outside the LOE (102) so that light rays (148A-148C) (second type of light) coupled outside the LOE (102) (by the partial reflective surface (108)) pass through the lens (112) without being distorted by the lens (112). Furthermore, the lens (112) does not apply light output to any s-polarized component of NIR light reflected from the eye (110).

[0052] One specific class of material exhibiting birefringence (polarization) and / or spectral characteristics is a liquid crystal, which has different effects on light of different polarizations and, in certain examples, different wavelengths. For example, nematic phase liquid crystal molecules respond differently to incident light of two different linear polarizations (s-polarization and p-polarization). In an exemplary but non-limiting embodiment, the lens (112) is implemented as a nematic phase liquid crystal lens composed of a layer of liquid crystal material. The layer of liquid crystal material is assumed to provide a tunable focal length, thereby the lens (112) has a defined focal length for polarized light of one polarization direction (e.g., p-polarization) and acts as a collimator for that light, and the lens (112) does not apply a light output to light of orthogonal polarization (e.g., s-polarization). Because each liquid crystal molecule of the nematic phase liquid crystal has a different responsiveness to each linear polarization, different refractive indices of the liquid crystal molecule can be induced. In this way, incident light in one polarization direction is "observed" as having no change in refractive index, whereas incident light in the other polarization direction is "observed" as having a change in refractive index, thereby inducing a lens effect for the light of that polarization.

[0053] In a twisted nematic liquid crystal, each liquid crystal molecule has a different responsiveness to each circular polarization (e.g., right circular polarization (or RHP) and left circular polarization (or LHP)). Typically, the responsiveness to the twisted nematic liquid crystal induces a positive power lens effect in the case of RHP, while in the case of LHP, a negative power lens effect is induced. By introducing another isotropic lens having the same focal length induced by the liquid crystal lens (112), the light output for one polarization can be doubled and no light output can be produced for the other polarization. Since the light output is applied differently to the RHP and LHP light, it is noted that it is desirable to place a quarter-wave plate (150) between the eye (110) and the lens (112) to properly rotate the circular polarization direction of the reflected light from the eye (110).

[0054] A lens composed of liquid crystal material is generally composed of a thin diffraction grating structure (similar to that in a Fresnel lens) that generates diffraction dispersion of incident light. Each grating can be designed to have greater intensity for a specific diffraction order of that grating. High intensity for that specific diffraction order is chromatic (i.e., wavelength dependent). Therefore, the grating can be designed so that for wavelengths in the NIR region, the relative intensity of diffraction nodes of order 1 or higher is higher than the intensity of diffraction nodes of order 0. In the photoadaptation region, the higher nodes should have low or no intensity. The grating orientation is spatially changed so that the aforementioned conditions for light in the NIR and photoadaptation regions are satisfied, thereby creating a lens effect so that the lens (112) effectively collimates the light, and incident light having wavelengths of the second optical spectrum (e.g., the photoadaptation (or visible light) region of the electromagnetic spectrum) is essentially unaffected by the lens (112). Here, it should be noted that the lattice direction of the liquid crystal molecules is changed to spatially alter the refractive index of the liquid crystal molecules without utilizing the birefringence properties of the liquid crystal material.

[0055] Generally, the lens (112) may be designed to distinguish based on a combination of wavelength and polarization. However, if the spectral separation between the first optical spectrum and the second optical spectrum is sufficiently large without negatively affecting the light from the image projector, it may be sufficient to distinguish between the first type and the second type of light based only on wavelength. Generally speaking, the effect of the lens (112) on the light from the image projector may be evaluated based on one or more image quality metrics, such as, for example, MTF, haze, checkerboard contrast, etc.

[0056] Note that the device (100) of the present disclosure is particularly applicable when used in an augmented reality (AR) system, in which case an image projected by the optical module (126) is overlaid on a real-world scene that can be seen by an observer through the surface (104, 106) and the partial reflective surface (108). Accordingly, it is also desirable that light waves from the real-world scene passing through the surface (104, 106) of the LOE (102) are not distorted by the lens (112) before reaching the eye (110). To prevent light waves from the real-world scene from being distorted by the lens (112), a polarizer (136) that transmits only the component of the incident light in a second polarization direction (e.g., s-polarization) is associated with the surface (106). It is desirable that the polarizer (136) and the LOE (102) have a common elongation direction (optionally depicted herein as corresponding to the x-axis). Preferably, the polarizer (136) is positioned to extend over the entire (or nearly the entire) surface (106) so that light from the entire real-world field of view (corresponding to a wide angular distribution of incident light) is appropriately polarized by the polarizer (136) before striking the surface (106).

[0057] The effect of the polarizer (136) on a real-world scene is schematically illustrated in FIG. 4. As illustrated, an image of a real-world scene (152), as schematically depicted here, with an illumination beam (152) containing sample rays (152A and 152B) that expand the beam, strikes a polarizer (136) that transmits only the s-polarized light components of the rays (152A and 152B). Since the rays (152A and 152B) are s-polarized and have wavelengths in the light-adapting region, the s-polarized rays (152A and 152B), similar to rays (148A-148C), pass through the lens (112) and reach the eye (110) without being distorted by the lens (112) (i.e., the lens (112) does not apply any light output to the rays (152A and 152B).

[0058] As discussed in the background section, to minimize unwanted reflections that may result in ghost images, it is desirable for the partial reflective surface to be coated to have low reflectivity for a first angle of incidence range and a desired partial reflectivity for a second angle of incidence range. In the prior art configuration of FIG. 1, such coating is typically specific to the wavelength range and polarization of the projected image. For example, if the projected image consists of s-polarized light having a wavelength in the photoadaptation region of the electromagnetic spectrum, the partial reflective surface is coated to have low reflectivity for the s-polarized light in the photoadaptation region in the first angle of incidence range and to have a desired partial reflectivity for the s-polarized light in the photoadaptation region in the second angle of incidence range. This coating scheme is ideal for the configuration of FIG. 1 because the light propagates only in the forward direction and the partial reflective surface (22) is used only to combine the light outside the LOE (20). However, in the configurations of FIGS. 2 to 4, where the first type of light (the p-polarized component of the NIR light reflected from the eye (110)) propagates in the reverse direction and the second type of light (s-polarized photo-adapting light from the image projector) propagates in the forward direction and the partial reflective surface (108) is configured to combine the first type of light within the LOE (102) and combine the second type of light outside the LOE (102), a modified coating scheme must be followed to ensure an appropriate desired reflectance for the first type of light. Specifically, the partial reflective surface (108) is preferably coated as in the configuration of FIG. 1 and is further coated to have a desired reflectance for the p-polarized light in the NIR region within a predetermined angle of incidence range.

[0059] As previously discussed, the optical module (126) performs the dual roles of image projection and light focusing and detection. The following paragraph describes the structure and operation of the optical module (126) in both the role of an image projector for projecting an image (142) and a focusing and detection device for focusing light reflected from the eye (110) to an optical sensor (128).

[0060] Referring first to FIG. 5, the optical module (126) (also referred to as the image projector (126)) comprises an illumination prism (160) and a collimating prism (180), each formed of a light-transmitting material. The illumination prism (160) has a plurality of outer surfaces including a light-incident surface (168), an image display surface (170), a light-incident surface (172), and a light-outcident surface (174). A polarization-selective beam splitter configuration (166) is placed within the prism (160) on a plane inclined with respect to the light-incident surface (168). The prism (160) is based on two configuration prisms, namely a first configuration prism (162) and a second configuration prism (164), wherein at least one of the prisms (162, 164) is provided on the hypotenuse side, and a polarizing beam splitter (e.g., a wire grid beam splitter) forms at least part of a polarization-selective beam splitter configuration (166) that reflects s-polarized light and transmits p-polarized light (incident on the surface of the beam splitter). The two hypotenuse sides of the prisms (162, 164) are joined together to form a joined single illumination prism assembly. This single joined prism is used to illuminate a reflective display device (for image projection) and also to guide incident light reflected from the eye (110) to an optical sensor (128) associated with a light wave emission surface (174). A polarizing beam splitter can be provided through a direct polarization-selective coating on one of the hypotenuses, or through a thin piece of material, such as a sheet, foil, or glass plate, on which a polarization-selective coating is deposited, thereby attaching the thin piece of material to one of the hypotenuses.

[0061] In a specific preferred embodiment, surfaces (170 and 172) are parallel to each other, and surfaces (168 and 174) are parallel to each other. In a specific particularly preferred embodiment, the prism (160) is a rectangular prism, that is, it has rectangular faces that are orthogonal to each other, and in a specific particularly preferred example shown herein, it is a cubic prism, in which case each constituent prism (162 and 164) has a 45-degree right-angle cross-sectional shape.

[0062] The collimation focusing prism (180) also has a plurality of outer surfaces including a first light wave input / output surface (190) (aligned and parallel with the light wave input / output surface (172)), a second light wave input / output surface (194), a collimation-focusing surface (192), and a fourth surface (188). A polarization and spectrum-selective beam splitter configuration (186) is disposed within the prism (180) on a plane inclined with respect to the surface (188). As can be seen in FIG. 5, the beam splitter configurations (166 and 186) are in parallel planes. The prism (180) is based on two constituent prisms, namely a first constituent prism (182) and a second constituent prism (184), wherein at least one of the prisms (182, 184) is provided on the hypotenuse side, and the polarization and spectrum selective beam splitter forms at least part of a polarization and spectrum selective beam splitter configuration (186) that reflects p-polarized light and transmits s-polarized light having a wavelength of the first optical spectrum (e.g., NIR region) and reflects s-polarized light and transmits p-polarized light having a wavelength of the second optical spectrum (e.g., photoadaptation (or visible light) region). The two hypotenuse sides of the prisms (182, 184) are joined together to form a joined single collimated focusing prism assembly. These single junction prisms are used to guide light from a reflective display device toward an optical element (a lens (140) which is a collimating focusing component) to collimate the display light, and also to guide incident light reflected from the eye (110) toward the optical element to focus the light to an optical sensor (128) through an illumination prism (160). A polarization and spectrum-selective beam splitter may be provided through a polarization and spectrum-selective coating in the form of a direct dielectric coating on one of the hypotenuses.

[0063] In a specific preferred embodiment, surfaces (190 and 192) are parallel to each other, and surfaces (188 and 194) are parallel to each other. In a specific particularly preferred embodiment, the prism (180) is a rectangular prism, that is, it has rectangular faces that are orthogonal to each other, and in a specific particularly preferred example shown herein, it is a cubic prism, in which case each constituent prism (182 and 184) has a 45-degree right-angle cross-sectional shape.

[0064] A polarizing light source (176) (which may be a combination of a polarizer and a light source (e.g., an LED)) is associated with a light incident plane (168). The polarizing light source (176) is configured to emit polarized light of a second optical spectrum (i.e., the visible region), which is schematically represented by the incident beam (158). A reflective display device (178) (preferably implemented as a liquid crystal on silicon (LCoS) microdisplay) that generates spatial modulation of reflected light corresponding to an image is associated with an image display plane (170). The reflective display device (178) is illuminated by the incident beam (158) from the polarizing light source (176) reflected from the beam splitter configuration (166). The reflective display device (178) is configured so that the reflected light corresponding to the bright region of the desired image has polarization rotated with respect to the polarizing light source. Accordingly, as illustrated in FIG. 5, polarized illumination (158) is incident on the prism (160) through the light wave incident surface (168) with first polarization, typically s-polarization with respect to the surface of the beam splitter configuration (166), and is reflected toward the image display surface (170) that strikes the reflective display device (178). Pixels corresponding to bright areas of the image are reflected with modulated rotational polarization (typically p-polarized), and radiation from the bright pixels is transmitted through the beam splitter configuration (166) and exits from the prism (160) through transmission through the light wave inlet / outlet surface (172).Then, light is incident on a prism (180) through a light wave input / output surface (190) with second polarization (typically p-polarized with respect to the surface of the polarization and spectrum-selective beam splitter configuration (186)) and reaches a collimation focusing surface (192), in which case the light passes through at least one delay plate (196), preferably a quarter-wave plate associated with at least a portion of the collimation focusing surface (192), and is incident on at least one light wave collimation focusing component, i.e., a lens (140), placed over at least a portion of the delay plate (196), and is reflected back through the delay plate (196) by the reflective surface (141) of the lens (140). Double passage through a delay plate (196) aligned with the fast axis at 45 degrees with respect to the polarization axis rotates the polarization (e.g., converting p-polarization to s-polarization) so that collimated image illumination is reflected from a polarization and spectrum-selective beam splitter configuration (186) toward the light wave input / output plane (194) and emitted from a prism (180) as an illumination beam (142). Subsequently, the illumination beam (142) is coupled within the LOE (102) by an optical coupling configuration (124).

[0065] FIG. 6 schematically illustrates an optical path followed by collimation light reflected from the eye (110) through an optical module (126) after being coupled outside the LOE (102) by an optical coupling configuration (124). Referring again to FIG. 2 and 3, collimation light (116A-116F) reflected from the eye (110) propagating in the reverse direction through the LOE (102) is coupled outside the LOE (102) by the optical coupling configuration (124) as light rays (122A-122C) representing an illumination beam (122). Illumination (122) (typically in the NIR region) may include two orthogonal polarization components (i.e., a collimated first polarization component (e.g., p-polarization for the surface of the beam splitter configuration (186)) and a second polarization component (e.g., s-polarization for the surface of the beam splitter configuration (186) that is not collimated)). Illumination (122) is incident on the prism (180) through the light wave input / output surface (194). As discussed, the polarization and spectrum-selective beam splitter configuration (186) reflects p-polarized light and transmits s-polarized light having a wavelength of the first optical spectrum (e.g., NIR region). Therefore, the second polarization component of the illumination (122) is transmitted by the beam splitter configuration (186) and exits the prism (180) through the surface (188). The first polarization component of the (collimation) illumination (122) (typically p-polarization with respect to the surface of the beam splitter configuration (186)) is reflected by the polarization and spectrum-selective beam splitter configuration (186) and reaches the collimation focusing surface (192), whereby the component passes through the delay plate (196), is incident on the collimation focusing component (140), and is reflected again through the delay plate (196) by the reflective surface (141) of the lens (140).While the collimation focusing component (140) serves to collimate the uncolimated illumination (158) of FIG. 5, the collimation focusing component (140) performs the opposite function for the collimated illumination (122) of FIG. 6, that is, to convert a set of parallel rays (colimated rays (122A, 122B, and 122C)) into a converged beam of captured light, that is, to focus the illumination (122) to the optical sensor (128), by applying an optical output to the incident collimated rays. Additionally, similar to what was previously described with reference to FIG. 5, a double pass through a delay plate (196) aligned with the fast axis at 45 degrees with respect to the polarization axis rotates the polarization of the illumination (122) (e.g., converting p-polarization to s-polarization), and the focused illumination is transmitted through a polarization and spectrum-selective beam splitter configuration (186) and exits from a prism (180) through transmission through a light wave input / output surface (190). Then, the light is incident on the prism (160) through the light wave input / output surface (172) with a first polarization (typically s-polarization with respect to the surface of the beam splitter configuration (166)). As previously discussed, the beam splitter configuration (166) reflects s-polarized light and transmits p-polarized light. Since these reflection and transmission characteristics are based solely on the polarization of the incident light, both NIR and visible light are processed in the same way by the beam splitter configuration (166). Thus, s-polarized NIR light is emitted from the prism (160) as a focused illumination beam (198) that is reflected from the beam splitter configuration (166) toward the light wave emission plane (174) and strikes the optical sensor (128).

[0066] Additionally, for each case following a specific polarization path in the examples described herein, the polarization is interchangeable, so that, for example, when changing the polarization selection characteristics of the beam splitter configuration (166, 186) and the lens (112), each reference to p-polarized light can be replaced with s-polarized light and vice versa. For example, the lens (112) can be configured to collimate the s-polarized component of (NIR) light. In this configuration, the polarizing light source (176) is configured to emit a p-polarized incident beam (158), the beam splitter configuration (166) reflects the p-polarized light and transmits the s-polarized light (in both the photoadaptation and NIR regions), and the beam splitter configuration (186) reflects the s-polarized light and transmits the p-polarized light having a wavelength in the NIR region, and reflects the p-polarized light and transmits the s-polarized light having a wavelength in the photoadaptation (visible light) region.

[0067] The polarization and spectrum-selective beam splitter configuration (186) illustrated in FIGS. 5 and 6 may have certain drawbacks, particularly the complexity of the coating design that provides the appropriate spectrum of the incident light and polarization-selective transmission and reflection. An alternative to the beam splitter design illustrated in FIGS. 5 and 6 is illustrated in FIG. 7. In the present invention, the beam splitter configuration (186) is implemented as a polarization-selective beam splitter configuration (similar to beam splitter configuration (166)), that is, it reflects s-polarized light and transmits p-polarized light in both the first and second optical spectra (i.e., visible light and NIR light are treated equally by the beam splitter configuration (186)). Since the beam splitter configuration (186) illustrated in FIG. 7 does not distinguish between light of the first or second optical spectrum, two additional delay plates are placed to handle polarization rotation for NIR light. Specifically, the delay plate (195) is associated with at least a portion of the light wave input / output surface (194), and the other delay plate (197) is associated with the light wave input / output surface (172) and the light wave input / output surface (190) and is placed between the prisms (160 and 180). The delay plates (195, 197) act as half-wave plates for the incident light of the first optical spectrum (i.e., NIR light) to rotate the polarization of the incident NIR light, and act as full-wave plates for the incident light of the second optical spectrum (i.e., photo-conforming (visible) light) so as not to affect the polarization state of the incident photo-conforming light.

[0068] Accordingly, the first polarization (typically p-polarization) component of the illumination (122) striking the delay plate (195) has a polarization that is rotated to a second orthogonal polarization (e.g., converted from p-polarization to s-polarization) by the delay plate (195), and the second polarization (typically s-polarization) component of the illumination (122) striking the delay plate (195) has a polarization that is rotated to a first orthogonal polarization (e.g., converted from s-polarization to p-polarization) by the delay plate (195). (After passing through the delay plate (195)) the illumination (122) is incident on the prism (180) through the light wave input / output surface (194). The component of the illumination (122) incident on the prism as p-polarized (with respect to the surface of the polarization-selective beam splitter configuration (186)) is transmitted by the beam splitter configuration (186) and exits from the prism (180) through the surface (188). A component of the illumination (122) incident on the prism as s-polarized (on the surface of the polarization-selective beam splitter configuration (186)) is reflected by the beam splitter configuration (186) and reaches the collimation focusing surface (192), in which case the component passes through the delay plate (196), is incident on the collimation focusing component (i.e., lens) (140), and is reflected again through the delay plate (196) by the reflective surface (141) of the lens (140) to rotate the polarization (e.g., convert s-polarization to p-polarization), and the focused illumination is transmitted through the polarization-selective beam splitter configuration (186) and exits from the prism (180) through transmission through the light wave input / output surface (190). After that, light strikes the delay plate (197) with first polarization (typically, p-polarization) and is rotated by the delay plate (197) into second orthogonal polarization (e.g., converting p-polarization into s-polarization), so that the illumination (122) is incident on the prism (160) through the light wave input / output surface (172) with s-polarization against the surface of the beam splitter configuration (166).Now, the s-polarized light is emitted from the prism (160) as a focused illumination beam (198) that is reflected by the beam splitter configuration (166) toward the light wave emission surface (174) and strikes the optical sensor (128).

[0069] Note that since the delay plate (195, 197) acts as a wave plate for light adaptation light, the traverse path through the prism (160, 180) from the polarizing light source (176) to the output part of the prism (180) (light wave input / output surface (194)) and the polarization direction of the traverse light are not affected by the delay plate (195, 197).

[0070] Additionally, it should be noted that the configuration of the optical module (126) illustrated in FIG. 7 is applicable to a situation where the lens (112) distinguishes a first type of light, i.e., eye-tracking light (light from the eye (110)), and a second type of light, i.e., image light (light from the reflective display device (178), at least partially based on polarization separation. In a configuration where the lens (112) distinguishes these two types of light based only on spectral separation, a delay plate (195) is not required. This is due to the fact that the eye can be illuminated such that the eye-tracking light generally includes s and p polarization components that are collimated by the lens (112), for example, in the NIR region regardless of polarization, because the lens (112) collimates the light of the optical spectrum occupied by the eye-tracking light. Accordingly, the illumination (122) externally coupled to the optical module (126) from the LOE (102) is collimated with the s-polarized and p-polarized components (with respect to the surface of the beam splitter configuration (186). Here, the p-polarized component will be incident on the prism (180) through the surface (194), transmitted by the beam splitter configuration (186), and exit from the prism (180) through the surface (188). The s-polarized component is incident on the prism (180) through the surface (194), reflected by the beam splitter configuration (186), exits the prism (180) through the surface (192), and reaches the collimation focusing surface (192). In this case, the component passes through the delay plate (196), is incident on the collimation focusing component (i.e., lens) (140), and is reflected again through the delay plate (196) by the reflective surface (141) of the lens (140) to rotate the polarization (e.g., converting s-polarized light to p-polarized light), and the focused illumination is transmitted through the beam splitter configuration (186) and exits the prism (180) through transmission through the light wave input / output surface (190).

[0071] For example, other implementations of the beam splitter configurations (166, 186) of the optical module (126) are considered herein by implementing one or both of the beam splitter configurations (166, 186) as simple 50-50 beam splitters, which reflect about half of the incident light intensity and transmit about half of the incident light intensity. Alternatively, both beam splitter configurations may be implemented as a polarization-selective beam splitter configuration for incident light of the second optical spectrum (visible light) and a simple 50-50 beam splitter for incident light of the first optical spectrum (NIR light). For example, the beam splitter configuration may reflect s-polarized visible light and transmit p-polarized visible light, and may reflect about half of the incident NIR light intensity and transmit about half of the incident NIR light intensity. However, it should be noted that in this 50-50 beam splitter configuration, only about 25% of the initial incident light intensity reaches the output.

[0072] Various configurations of the illumination array (138) are considered herein. In all configurations of the illumination array, the illumination array (138) comprises one or more light sources configured to illuminate the eye (110) with a first type of light (i.e., light of a first optical spectrum containing a component of light polarized in a first polarization direction (e.g., p-polarized) (e.g., NIR light)). Ideally, the light source(s) of the illumination array (138) are positioned to illuminate the eye (110) in an illumination direction as close as possible to the normal to the EMB (109). In an alternative configuration, the light source(s) are positioned around the eye's (110) field of view to illuminate the eye (110) from the side. In another configuration, a lighting array (138) is positioned as part of an optical module (126) and may be configured to inject light from the lighting array (138) into the LOE (102) so as to propagate in a forward direction in addition to generating and projecting an image (142) onto the LOE (102) for viewing by the eye (110), and to be coupled outside the LOE (102) by a partial reflective surface (108) in an external coupling direction perpendicular to the EMB (109).

[0073] The following paragraph describes various placement options for the lighting array (138) with particular reference to FIG. 8. Non-limiting embodiments of the device (100) shown in FIG. 8 are intended to provide context regarding general placement options for the lighting array (138). In a specific non-limiting embodiment shown herein, the device (100) is implemented as an eyeglass form factor having a head-mounted mechanical body implemented as an eyeglass frame (154) having a side arm (156) for engaging with the observer's ear. It should be noted that other form factors, such as a helmet-mounted form factor, a vehicle windshield form factor, and other head-up display and near-eye display form factors, are also clearly within the scope of the invention. The illumination array (138) may include at least one light source (138A) positioned close to the active area of ​​the LOE (102) (represented as two light sources in FIG. 8) (e.g., via direct or indirect attachment to the surface (204)), so that the light emitted by the light source (138A) reaches the EMB (109) at a normal to the EMB (109). Alternatively, or in addition to the light source (138A), the illumination array (138) may include at least one other light source (138B) positioned near the side of the observer's head (in FIG. 8, this is shown attached to an optical coupling configuration (124) attached to the side arm (156)). In this configuration, the light emitted by the polarized light source (138B) reaches the EMB (109) at an out-of-axis angle. As discussed, light reflected from the eye (110) may contain two orthogonal polarization components of light (i.e., an s-polarization component and a p-polarization component), and the lens (112) is configured to collimate only one of these two polarization directions. In the example described herein, it is preferable that the lens (112) is configured to apply light output to the p-polarization component of the light reflected from the eye (110) (to collimate the p-polarization light) and not to apply light output to the s-polarization component of the light reflected from the eye (110).

[0074] The illumination array (138) may be configured to illuminate a specific area of ​​the eye (110) or the entire eye (110) with NIR light. As discussed in detail, the illumination reflected by the eye (110) (i.e., a first type of light represented by rays (114A-114F)) is collimated (by the lens (112)), coupled within the LOE (102) by the partial reflective surface (108), and then coupled outside the LOE (102) (by the optical coupling configuration (124)), in which case the illumination is focused (by the lens (140)) by the optical sensor (128). The optical sensor (128) generates a signal in response to detecting the focused light, and the signal is transmitted to a processing system (130) configured to process the signal and derive the current line of sight of the eye (110). In a specific, non-limiting embodiment, the device (100) obtains a line of sight (an angular direction of the eye (110), or a line of sight of the eye (110)) by imaging a pattern present in a specific region of the eye (110). The location and movement of this pattern indicate the current line of sight and movement of the eye. The human eye contains various traceable features, for example, patterns generated by the blood vessels of the retina. These traceable features can be tracked using a suitable tracking algorithm implemented by a suitable image processing instruction performed by the processing system (130).

[0075] In the non-limiting process of deriving and tracking the direction of gaze, a retinal pattern is mapped and traceable features are determined during initial setup, after which a continuous tracking process is performed. For example, an image marker may be displayed to the observer so that the observer can see it during initialization. While the observer faces the marker, the illumination array (138) fully illuminates the fundus (visible part of the retina) with short pulses and a full image of the fundus acquired (via the optical sensor (128)). Subsequently, this image is processed by a processing system (130) to identify traceable features (e.g., the optic disc and fovea). During a continuous tracking process, a selected region of interest (ROI) of the eye (110) is selectively illuminated by a lighting array (138), and an image of the ROI (acquired by an optical sensor (128)) is sampled and processed (by a processing system (130)) during a corresponding lighting pulse to determine the current line of sight (line of sight), and using this derived line of sight, the position of the ROI for a subsequent lighting cycle is updated, and the continuous tracking process is repeated by illuminating the updated ROI. Assuming the frequency of tracking measurements is high relative to the speed of the eye's movement, this update process is typically effective for maintaining continuous tracking and is optionally combined with tracking information from another eye. When the line of sight changes, the illuminated area also changes. The update of the ROI can be performed according to the "current" line of sight determined from the last sampled image, or in certain cases, predictive extrapolation based on eye movement between two or more previous measurements can be used. In the event of a tracking failure, the size of the illuminated area can be temporarily increased until the traceable features are recovered.

[0076] Referring again to FIG. 8, the processing system (130) may be implemented using any suitable type of processing hardware and / or software as known in the art, including but not limited to any combination of various dedicated graphics processors, display drivers, and computer-type processors (collectively designated as processors (132)) that operate under any suitable operating system and implement suitable software or firmware modules. The storage medium (134) may be one or more computer-type memory devices, such as volatile data storage. The processing system (130) may further include various communication components to enable wired or wireless communication with LAN and / or WAN devices for bidirectional transmission of information and graphic content. The device (100) is powered by a suitable power source, which may be any combination of a battery and / or provided external power source, which is schematically illustrated herein as a power source (131) connected via a cable (133). If a battery power source is used, the battery may be integrated as part of a glasses or helmet-mounted structure.

[0077] Optical components associated with surfaces (104, 106) of the LOE (102), such as a lens (112) and a polarizer (136), are optically attached to the LOE (102) using any suitable attachment technique, such as mechanical attachment to the LOE (102), while maintaining an air gap or material (e.g., gel) gap between the optical components and the surfaces of the LOE (102). The material occupying this air gap or material gap has a refractive index low enough to preserve internal total reflection conditions within the LOE (102). Other suitable alternatives for optically attaching the optical components to the LOE (102) include the placement of an air gap film having an ultrafine structure between the surfaces of the LOE (102) and the optical components, or the placement of a transparent layer of a low-refractive-index material (e.g., a thin plate of a low-index material). Further details regarding this optical attachment methodology can be found in the applicant’s jointly owned U.S. Patent No. 10,520,731 and U.S. Patent Publication No. 2018 / 0067315. A quarter wave plate (150) can be attached to a lens (112) using a similar optical attachment technique.

[0078] The embodiment of the device (100) described so far relates to an optical coupling configuration implemented with a set of partial reflective surfaces (108) for coupling eye-tracking light within the LOE (102) and image light (from the optical module (126)) outside the LOE (102), but the partial reflective surfaces (108) are merely examples of one non-limiting optical coupling configuration, and other optical coupling configurations may be used to coupling eye-tracking light within the LOE (102) and image light outside the LOE (102). The optical coupling configuration may be any optical coupling array that deflects a portion of the eye-tracking incident radiation from the lens (112) into an angle in which it propagates through internal reflection within the LOE (102), and likewise deflects a portion of the image incident radiation (from the optical module (126)) already propagating within the LOE (102) into an angle in which the deflected portion of the image incident radiation exits the LOE (102) through internal reflection. Other examples of such suitable optical coupling arrays include, but are not limited to, one or more diffractive optical elements placed on either of the planes (104, 106).

[0079] An embodiment of the device described in connection with FIGS. 2 through 8 relates to the use of a collimation element (lens (112)) sensitive to polarization and / or spectrum (wavelength) so as to collimate only specific components of the eye-tracking light (i.e., light reflected from the observer's eye) to focus the internally coupled collimated light onto an optical sensor integrated in an image projector (optical module (126)). To image the eye and determine the angle at which light is emitted from the eye, other eye-tracking solutions are considered herein in which light not collimated from the eye is coupled within a light-guided optical element and guided to an optical sensor by a biconic lens having different radii of curvature with respect to two orthogonal axes. Such a solution preferably utilizes a specialized at least partial reflective surface placed in a dedicated light-guided optical element separated from the LOE through which the projected image propagates.

[0080] Now, referring to FIGS. 9 through 15, various aspects of the structure and operation of a device, generally denoted as 200, configured and operated according to various embodiments of the present invention for displaying an image and deriving the line of sight of a human eye (210) through an internal coupling configuration associated with a light-guided optical element (LOE). In a preferred but non-limiting embodiment illustrated herein, the internal coupling configuration is implemented as a surface (208) that reflects at least partially light emitted from the eye (210). The surface (208) is hereinafter referred to interchangeably as at least partial reflective surface (208). The surface (208) is associated with a first LOE (202) configured to propagate light reflected from the eye (in response to illumination by the illumination array (242), which is separate from a second LOE (212) configured to propagate a projected image to be externally coupled for viewing by the eye (210). The LOE (202) is formed of a transparent material and has a pair of parallel planes (principal surfaces of the plane) (204, 206) for guiding light by internal reflection (preferably internal total reflection). Surface (208) is configured to internally combine a portion of the light incident on the plane (204) within the internally combined region (243) so that it propagates by internal reflection (total reflection) within the LOE (202). In particular, the surface (208) is positioned within the LOE (202) inclined with respect to the parallel planes (204, 206) (i.e., between the planes (204, 206)) so that the internally combined light is trapped within the substrate (202) by internal reflection from the planes (204, 206). The internally combined region (243) of the LOE (202), also referred to as the "active region" or "active zone," is a two-dimensional region of the plane (204).

[0081] Light internally coupled by the surface (208) propagates in reverse through the LOE (202) until it reaches the external coupling optical configuration (207) (schematically illustrated as a prism in FIG. 9, but which may be implemented, for example, as a reflective surface). Preferably, a mixer (205) implemented as a partial reflective surface is positioned within the LOE (202) in a plane parallel to the upstream and adjacent surfaces (204, 206) from the external coupling optical configuration (207) (preferably, an intermediate plane between surfaces (204 and 206)) to mitigate the non-uniformity of the light propagating through the LOE (202). The light is coupled to the optical module (236) outside the LOE (202) by the external coupling optical configuration (207). The optical module (236) includes a lens (240) (a biconic lens) and an optical sensor (238) (the sensor (238) may be located outside the optical module (236). External coupled light passes through the lens (240), which guides the light to the optical sensor (238) configured to detect light reflected from the eye (210). A processing system (230) comprising at least one computer-type processor (232) coupled to a storage medium (234) (e.g., computer memory, etc.) is electrically associated with an optical sensor (238) and is configured to process a signal from the optical sensor (238) to derive the current gaze direction of the eye (210).

[0082] A second LOE (212) configured to propagate a projected image to be externally coupled for viewing by the eye (210) is formed of a transparent material and has a pair of parallel planes (principal surfaces of a plane) (214, 216) for guiding light by internal reflection (preferably internal total reflection). One of the parallel planes (214) facing the eye (210) is positioned in the LOE (212), in which case the eye (210) is located at an eye-distance (ER) (213) from the corresponding plane (214) in the EMB (209). An image projector (228) is configured to project an image (222) (colimed to infinity) as schematically illustrated here by an illumination beam (222) comprising sample rays (222A, 222B, and 222C) that expand the beam. The projected image (222) is coupled within the LOE (212) by an internal coupling optical configuration (224) as schematically illustrated here by a reflective surface (however, other configurations such as, for example, a prism are considered herein), generating a ray (223) captured by internal reflection within the substrate and a ray (225) is also generated. The image propagates along the substrate by repeated internal reflection and strikes an optical external coupling configuration associated with a second LOE (212), which is composed of a series of partial reflective surfaces (218) at an angle of inclination with respect to parallel planes (214, 216), in which case a portion of the image intensity is reflected and coupled outside the substrate as a ray (226A, 226B, and 226C) toward the observer's eye (210). Note that the partial reflective surface (218) is merely an example of one non-limiting optical external coupling configuration suitable for use with the LOE (212), and that other optical coupling configurations may be used to coupling image light outside the LOE (212). The optical external coupling configuration may be any optical coupling array that deflects a portion of the image propagating within the LOE (212) by internal reflection at an angle at which a deflected portion of the image exits the LOE (212).Other examples of such suitable optical coupling arrays include, but are not limited to, one or more diffractive optical elements placed on any one of the planes (214, 216).

[0083] One of the parallel planes (204) facing the eye (210) is disposed on the LOE (202), but the LOE (212) is interposed between the eye (210) and the LOE (202), and the planes (204 and 216) are parallel (or approximately parallel), aligned, and adjacent to each other. The eye (210) is located at an eye-distance (ER) (211) from the plane (204). In a non-limiting configuration illustrated in the drawings, the LOEs (202 and 212) are disposed such that the planes (204, 206, 214, 216) are parallel to each other and the LOEs (202 and 212) have a common elongation direction (optionally depicted herein as corresponding to the x-axis). The LOEs (202 and 212) are optically attached to each other on the planes (204, 216) to form an interface plane. Any suitable mechanism, including but not limited to mechanical arrays and optical bonds, may be used to optically bond the LOEs (202 and 212) to each other. For example, the surfaces (204, 206) may be bonded to each other by providing an optical bonding layer on at least a portion of at least one of the surfaces (204, 206) to form a bonded single optical structure formed from two light guides that perform separate functions.

[0084] As described in the embodiment with reference to FIGS. 2 through 8, in this embodiment, the eye (210) is illuminated by light of a first optical spectrum (preferably in the NIR region) so that the light of the first optical spectrum is coupled within the LOE (202) by the surface (208), and the illumination (222) (i.e., projected image) is in the second optical spectrum (photoadaptation, i.e., visible region). As discussed in the previous embodiment, the partial reflective surface (218) is preferably coated to have a low reflectance for the first angle of incidence range and a desired partial reflectance for the second angle of incidence range. Additionally, the surfaces (214, 216) and the partial reflective surface (218) are preferably coated to have a high transmittance for light of the first optical spectrum so that the light reflected by the eye (210) passes through the LOE (212) with minimal loss of intensity before being coupled within the LOE (202).

[0085] In contrast to the incident light from the eye coupled within the LOE (102) in the previously described embodiments (Figs. 2 to 8), in this embodiment, the incident light from the eye (210) coupled within the LOE (202) is not collimated, resulting in an angular distribution of incident light rays striking the surface (208) at different angles of incidence. Particularly with reference to Fig. 10, incident light from two different points within the EMB (209) along the first dimension of the EMB (209) (the first dimension along the x-axis in the XYZ coordinate system arbitrarily labeled in the drawing) strikes the surface (208) and is coupled within the LOE (202). Here, the incident light, schematically represented by the first illumination beam (244) and the second illumination beam (250), is light (preferably in the NIR region) reflected by the eye (210) in response to illumination from the illumination array (242). Note that the two beams (244, 250) are merely examples of beams from the EMB (209) coupled within the LOE (202) by the surface (208), and additional beams from each additional point within the EMB (209) are also coupled within the LOE (202) by the surface (208). As can be seen, each beam (244, 250) generally reaches the surface (208) at a different angle of incidence, so that the beams (244, 250) generate each reflected beam (245, 251) that is trapped within the LOE (202) by internal reflection, but which propagates within the LOE (202) at a different angle with respect to the surface (204, 206).

[0086] As illustrated in FIG. 11, each beam (244, 250) includes a ray that extends the beam. In the illustrated example, the beam (244) includes sample rays (246A, 246B, and 246C) that extend the beam (244) along at least a portion of the internal coupling region (243) in the first dimension (along the x-axis), in which case the rays (246A and 246C) are the peripheral rays of the beam (244). Similarly, the beam (250) includes sample rays (252A, 252B, and 252C) that extend the beam (244) along at least a portion of the internal coupling region (243) in the first dimension (along the x-axis), in which case the rays (252A and 252C) are the peripheral rays of the beam (250). Each of the rays (246A, 246B, 246C, 252A, 252B, and 252C) is incident on the surface (204) at different points of the surface (204) along the first dimension (x-axis in the drawing) of the surface (204), and thus is incident on the surface (208) at different angles of incidence. Therefore, each of the rays (246A, 246B, 246C, 252A, 252B, and 252C) reaches the surface (208) at different angles of incidence, and each of the reflected beams (245, 251) includes a spaced reflected ray (extending each beam) that propagates within the LOE (202).

[0087] The angular distribution of light coupled within the LOE (202) by the surface (208) (extended by beams (244, 250) and by rays extending beams (244, 250)) is a function of the aperture width of the surface (208) (the width is projected onto a plane parallel to the EMB (209)). The aperture width is the placement angle of the surface (208) (measured with respect to the face (204)). βInversely proportional to the slope, in the case of a steep placement angle, the aperture width is effectively reduced, providing high resolution in the angle extension dimension (x-axis of the drawing). In this embodiment, the surface (208) is placed at a steeper angle than the partial reflective surface (218), and is placed at a sufficiently steep angle such that the aperture width of the surface (208) is sufficiently narrow compared to the distance between the LOE (202) and the EMB (209), so that light covering only a narrow angle distribution of the angle is coupled within the LOE (202) by the surface (208).

[0088] The resolution can be roughly approximated by the width of the surface (208) projected onto a plane parallel to the plane (204). In FIG. 9, the width is w Displayed as h / tan(β) It can be calculated as, and in this case h is the thickness of the LOE (202) (i.e., the minimum distance between planes (204, 206)). For example, h = 1mm and β In the case of = 65°, w It is 0.47 mm. Since a resolution of 0.47 mm is smaller than the pupil of the human eye, h and β These parameters for can provide a high-resolution image from the optical sensor (238), (placement angle β Increase and / or thickness h By reducing the width (by reducing it), a much higher resolution image can be produced. However, it should be noted that as the width is reduced, the intensity of the signal output by the optical sensor (238) is also reduced, thereby reducing the overall signal-to-noise ratio of the output signal. Therefore, care must be taken to find a proper balance between a small aperture width corresponding to a significant signal-to-noise ratio in the optical sensor (238).

[0089] Now, referring to FIG. 12, external combined rays (247A, 247B, and 247C) and external combined rays (253A, 253B, and 253C) are shown being received in the optical module (236). The rays (247A, 247B, and 247C) and the rays (253A, 253B, and 253C) correspond to incident rays that expand the angular distribution of the first dimension (width) of the surface (208). In particular, the external combined rays (247A, 247B, and 247C) correspond to incident rays (246A, 246B, and 246C), and the external combined rays (253A, 253B, and 253C) correspond to incident rays (252A, 252B, and 252C). The external coupling rays (247A, 247B, and 247C) and the external coupling rays (253A, 253B, and 253C) pass through a lens (240) that applies light output to the rays and guides the rays to an optical sensor (238).

[0090] As mentioned, the lens (240) is biconic, which in this context refers to having different radii of curvature for different axes. Different radii of curvature have two focal lengths in each of the two dimensions (orthogonal dimensions), i.e., in the first dimension f 1 At the first focal length and in the second dimension (orthogonal to the first dimension) f 2 It leads to a lens (240) having a second focal length. The optical sensor (238) is from the lens (240). f 1 It is placed at a distance. The first focal length ( f 1 ) and the first focal length ( f 1Positioning of the optical sensor (238) in the ) is configured such that the lens (240) converts the rays (247A, 247B, and 247C) (and rays (253A, 253B, and 253C)) into non-converging beams of captured light reaching different respective regions of the optical sensor (238), and the angular distribution of the rays (247A, 247B, and 247C) (and rays (253A, 253B, and 253C)) represents the angular distribution (in the width dimension of the surface (208)) of the corresponding beam (244) and (beam (250)). Furthermore, the rays (247A, 247B, and 247C) and the rays (253A, 253B, and 253C) reach different respective regions of the optical sensor (238), and the total angular separation between the set of rays (247A, 247B, and 247C) and the rays (253A, 253B, and 253C) in the optical sensor (238) represents the angular separation (in the dimension of the width of the surface (208)) between the beams (244 and 250). Therefore, the optical sensor (238) can measure the relative angle of light (beams (244, 250)) emitted from the EMB (209) coupled within the LOE (202) by the surface (208) with significantly high angular resolution. The angular resolution is generally a function of the effective aperture width (previously described) and pupil distance (ER(211)) of the surface (208) and sin -1 ( w / ER It can be expressed as ). An effective opening width of 0.47 mm ( w For an eye distance (ER) of ) and 27 mm, the angular resolution provided by the optical sensor (238) is approximately 1 degree. Additionally, as a result of the small angular resolution, the requirement for parallelism between the principle planes of the LOE (202) is much more lenient than that of the LOE (212) used for image projection on the eye (210), and in this case, parallelism of about 1 arcmin may be required.

[0091] The lens (240) has a second focal length (in a dimension orthogonal to the first focal length dimension) f2 It has a biconic mode of the lens (240) that enables imaging of the eye (210) through incident light (reflected from the eye (210)) that extends an internal coupling region (243) along two orthogonal dimensions. Imaging through the guidance (by the lens (240) of an external coupling light corresponding to an incident ray extending a first dimension (along the x-axis) was discussed with reference to FIGS. 10 through 12. The next paragraph will describe imaging of the eye (210) through the focusing of an external coupling light corresponding to an incident ray extending a second dimension (along the z-axis) by the lens (240) to an optical sensor (238).

[0092] Now, referring to FIG. 13, the beam (244, 250) also includes rays that extend each beam along at least a portion of the internal coupling region (243) in a second dimension (along the z-axis). In the illustrated example, the beam (244) includes sample rays (248A, 248B, and 248C) that originate from a common point of the EMB (209) and extend the beam (244) along at least a portion of the internal coupling region (243) in a second dimension (along the x-axis), in which case the rays (248A and 248C) are peripheral rays of the beam (244). Similarly, the beam (250) includes sample rays (254A, 254B, and 254C) that originate from a common point of the EMB (209) and extend the beam (250) along at least a portion of the internal coupling region (243) in a second dimension (along the x-axis), in which case the rays (254A and 254C) are peripheral rays of the beam (250). The projection of the rays (246A, 246B, and 246C) in the XY plane is incident on the surface (208) at a common angle of incidence. Thus, the rays (246A, 246B, and 246C) generate a set of spaced parallel rays that propagate through the LOE (202) when coupled within the LOE (202) by the surface (208). Similarly, the projection of rays (254A, 254B, and 254C) in the XY plane is incident on the surface (208) at a common angle of incidence. Therefore, the rays (254A, 254B, and 254C) generate a set of spaced parallel rays that propagate through the LOE (202) when combined within the LOE (202) by the surface (208).

[0093] Now, referring to FIG. 14, external coupling rays (249A, 249B, and 249C) and external coupling rays (255A, 255B, and 255C) are shown being received in the optical module (236). The rays (249A, 249B, and 249C) and the rays (255A, 255B, and 255C) correspond to incident rays that extend the first dimension (height along the z-axis) of the surface (208). In particular, the external coupling rays (249A, 249B, and 249C) correspond to incident rays (248A, 248B, and 248C), and the external coupling rays (255A, 255B, and 255C) correspond to incident rays (254A, 254B, and 254C). The external combined rays (249A, 249B, and 249C) and the external combined rays (255A, 255B, and 255C) pass through a lens (240) that applies an optical output to the rays to focus the rays (249A, 249B, and 249C) to a common area (or spot) on the image plane of the optical sensor (238) and focus the rays (255A, 255B, and 255C) to another common spot on the image plane of the optical sensor (238). In other words, the lens (240) converts the set of rays (249A, 249B, and 249C) into a converged beam of captured light, and likewise converts the set of rays (255A, 255B, and 255C) into a converged beam of captured light. The ability to focus this set of rays onto an image plane involves positioning the lens (240) at the output aperture of the LOE (202) and positioning the lens (240) at an appropriate second focal length ( f 2 It is possible by designing with ).

[0094] Generally, the lens (240) has a second focal length ( f 2 )go f 2= uf 1 / (( u - f It is desirable to design it so as to be given as 1), and in this case uis the in-plane distance that the ray travels from the surface (208) to the lens (240) along the second dimension. u = ER + L 2 / cos( θ It can be given as ), and L 2 is an in-plane distance from the surface (208) to the external coupled optical configuration (207), and θ is the angle at which light propagates (measured with respect to the plane (204)).

[0095] As in the embodiment described with reference to FIGS. 2 through 8, the optical sensor (238) of the present embodiment generates a signal in response to detecting a light beam reaching the sensor, and the signal is transmitted to a processing system (130) configured to process the signal and derive the current gaze direction of the eye (110). Derivation of the gaze direction may be performed using steps similar to those previously described with reference to FIGS. 2 through 8. Additionally, the ability of the optical sensor (238) to measure the relative angle of the incident light (beam (244, 250)) may be used to improve the derivation of the gaze direction in the present embodiment.

[0096] Possible arrangement configurations of the lighting array (242) are generally similar to the configuration of the lighting array (138) described with reference to FIG. 8. For example, the device (200) may be implemented as an eyeglass form factor having a head-mounted mechanical body implemented as an eyeglass frame having side arms for engaging with the observer's ears. Other form factors, such as a helmet-mounted form factor, a vehicle windshield form factor, and other head-up display and near-eye display form factors, may also be considered herein. The lighting array (242) may include one or more NIR light sources that can be placed close to the active area of ​​the LOE (202), for example, through direct or indirect attachment to the surface (214), so that the light emitted by the light source reaches the EMB (209) close to the normal to the EMB (209). Alternatively, or in addition to the configuration described above, the illumination array (242) may include at least one other NIR light source attached to an image projector (228) or an internal coupling optical configuration (224) (preferably attached to one of the side arms of the eyeglass frame) positioned near the side of the observer's head. In this configuration, a beam emitted by the NIR light source reaches the EMB (209) at an out-of-axis angle.

[0097] Additionally, the LOE (212) can be used to illuminate the eye (210) in a direction perpendicular to the EMB (209). In this configuration, the illumination array (242) is integrated as part of the image projector (228) as shown in FIG. 15. The image projector (228) is generally similar to the image projector (126) shown in FIG. 5, except that the image projector (228) does not include an optical sensor (since the optical sensor (238) is placed in an optical module (236) that is optically separated from the image projector (228). Also, since the optical sensor (238) is not part of the image projector (228), there is no need to design the beam splitter configuration (166 and 186) with a suitable coating for spectral selectivity. Therefore, in a non-limiting example of the image projector (228) illustrated in FIG. 15, both beam splitter configurations (166 and 186) are polarization-selective beam splitters that reflect incident light in a first polarization direction (e.g., s-polarized with respect to the surface of the beam splitter configurations (166, 186)) and transmit incident light in a second polarization direction (e.g., p-polarized with respect to the surface of the beam splitter configurations (166, 186)) for incident light of a first optical spectrum and incident light of a second optical spectrum (i.e., visible light and NIR light are processed equally by the beam splitter configurations (166 and 186)). Additionally, since light propagates only in the forward direction through the LOE (212) and light does not enter the image projector (228) from the LOE (212), the surface (172) is the light wave emission surface (172), the surface (190) is the light wave incidence surface (190), and the surface (194) is the light wave emission surface (194).

[0098] Similar to what is described with reference to FIG. 5, the polarizing light source (176) emits polarized light of a second optical spectrum (i.e., visible region), schematically represented by the incident beam (158). The polarizing illumination (158) is incident on the prism (160) through the light wave incident plane (168) as a first polarization, typically s-polarization for the surface of the polarization-selective beam splitter configuration (166), and is reflected toward the image display plane (170) by the polarization-selective beam splitter configuration (166) which strikes the reflective display device (178). Pixels corresponding to bright areas of the image are reflected as modulated rotational polarization (typically p-polarized), and radiation from the bright pixels is transmitted through the beam splitter configuration (166) and exits from the prism (160) through transmission via the light wave exit plane (172). After that, light is incident on the prism (180) through the light wave incident plane (190) as a second polarization (typically p-polarized with respect to the surface of the polarization-selective beam splitter configuration (186)) and reaches the collimation plane (192), where the light passes through the delay plate (196), is incident on the lens (140), and is reflected back through the delay plate (196) by the reflection plane (141) of the lens (140). The double passage through the delay plate (196), which is aligned with the fast axis at 45 degrees with respect to the polarization axis, rotates the polarization (e.g., converting p-polarization to s-polarization) so that the collimated image illumination is reflected from the polarization-selective beam splitter configuration (186) toward the light wave exit plane (194) and exits from the prism (180) as an illumination beam (222). After that, the illumination beam (222) is coupled into the LOE (212) by an internal coupling optical configuration (224) (shown in FIG. 9).

[0099] For example, an illumination array (242) implemented as a polarized NIR light source (which may be a combination of a polarizer and an NIR light source) is associated with a surface (188) which is the light wave incident plane (188) in this configuration. The polarized NIR light source is configured to emit polarized light of a first optical spectrum (i.e., polarized NIR light) schematically represented by an incident beam (268). The polarized illumination (268) is incident on a prism (180) through the light wave incident plane (188) as a first polarization, typically as p-polarization for the surface of the polarization-selective beam splitter configuration (188), transmitted through the polarization-selective beam splitter configuration (186), and exits from the prism (180) as an illumination beam (270) through transmission through the light wave exit plane (194). Subsequently, a p-polarized illumination beam (270) is coupled within the LOE (212) by an internal coupling optical configuration (224) (similar to the illumination beam (222)). The p-polarized illumination (270) propagates through the LOE (212) (similar to the illumination (222)) and is coupled outside the LOE (212) by a partial reflective surface (218). In this configuration, care must be taken to ensure that the NIR illumination propagating within the LOE (212) is externally coupled by the partial reflective surface (218), and that the NIR light emitted from the eye (210) (in response to illumination by the externally coupled NIR light) is not coupled back into the LOE. To this end, the partial reflective surface (218) is preferably coated to have a desired reflectance for s-polarized light in the NIR region within a defined angle of incidence range, so that s-polarized NIR illumination propagating within the LOE (212) is externally coupled by the partial reflective surface (218), but s-polarized NIR illumination emitted from the eye (210) is incident on the partial reflective surface (218) at an angle of incidence outside the defined angle of incidence range and passes through the partial reflective surface without reflection.

[0100] Although an embodiment of the device (200) has been described so far in the context of LOEs (202 and 212) having a common (parallel) stretching direction, other embodiments in which LOEs have mutually orthogonal stretching directions are also possible. For example, LOE (212) may be positioned to have a stretching direction in the x-axis direction (as shown in FIG. 9), whereas LOE (202) may be positioned to have a stretching direction in the z-axis direction. Additionally, embodiments of the device (100 and 200) have been described in the context that the LOE (102 and 212) is a "one-dimensional waveguide" or "1D waveguide," which means that each LOE (102 and 212) has a single pair of parallel principal surfaces (surfaces (104, 106) and surfaces (214, 216)) forming a "slab-type waveguide" that guides image light (from image projectors (126, 228)) to perform aperture expansion in one dimension. However, the eye-tracking device according to the present embodiment is equally applicable to other waveguide configurations, including a configuration in which an additional slab-type waveguide is coupled to each LOE (102, 212) that guides image light in an orthogonal dimension to perform aperture expansion in an orthogonal dimension, thereby creating a total two-dimensional aperture expansion effect. Alternatively, one or both of the LOEs (102, 212) are "two-dimensional waveguides" or "2D waveguides," which means having two pairs of mutually orthogonal principal surfaces that serve to guide the image light in two dimensions as the image light (from the image projector (126, 228)) propagates along the LOEs and performs aperture expansion in two dimensions using a single waveguide.

[0101] Although embodiments of the present disclosure have been described within the context of a lighting array positioned to illuminate the eye with light in the near-infrared region of the electromagnetic spectrum, embodiments of the present disclosure should not be limited to lighting arrays emitting eye-tracking light in any specific region of the electromagnetic spectrum. Description of using NIR light for eye-tracking purposes is for illustrative purposes to provide a clearer description of the configuration and operation of the various devices of the present disclosure. Other types of light, including but not limited to light in the infrared region and ultraviolet light emitted with low intensity and short pulse duration, may also be used for eye-tracking purposes.

[0102] According to certain non-limiting embodiments, various eye-tracking devices of the present disclosure may be duplicated to simultaneously track both eyes of an object and project images onto both eyes. For example, the device (100) and / or the device (200) may be duplicated for both eyes. By combining data from two eye trackers, the stability and continuity of tracking can be improved. For example, while the eyes are moving, the trackable part of the eye is visible to the tracker in one eye but not in the other. If a tracking algorithm is used that utilizes tracking of trackable features, tracking can be maintained for a period during which only one eye tracker can track the blind spot through simultaneous tracking for both eyes.

[0103] If the device is a pair of binoculars, each eye has its own image projection and eye tracking device, and various processing and power supply components can optionally be shared between the two eye tracking systems. The eye tracking information collected by the binocular eye tracking devices can be fused to provide enhanced stability and tracking continuity as mentioned above.

[0104] Although descriptions of various embodiments of this disclosure are provided for illustrative purposes, they are not intended to be complete or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been chosen to best describe the principles, actual applications, or technical improvements of the embodiments found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0105] As used herein, the singular forms (“a,” “an,” “the”) may include plural objects unless the context clearly indicates otherwise.

[0106] The word “exemplary” is used herein to mean “serving as an example, case, or illustration.” Any embodiment described as “exemplary” must not be interpreted as being preferable or advantageous over other embodiments and / or as excluding the integration of features from other embodiments.

[0107] It is understood that for clarity, specific features of the invention described in the context of separate embodiments may be provided in combination in a single embodiment. Conversely, for brevity, various features of the invention described in the context of a single embodiment may be provided separately, in any suitable sub-combination, or as suitable in any other described embodiment of the invention. Specific features described in the context of various embodiments are not considered essential features of such embodiments unless the embodiment would not operate without such elements.

[0108] The appended claims are drafted without multiple dependencies, and this was done solely to accommodate the formal requirements of jurisdictions that do not permit such multiple dependencies. It should be noted that the features of all possible combinations implied by implementing claims that rely on multiple dependencies are also explicitly considered and are deemed to be part of the invention.

[0109] Although the present invention has been described in relation to specific embodiments, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to encompass all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

Claims

Claim 1 delete Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 As a device, a light-transmitting substrate having a pair of parallel principal surfaces for guiding light by internal reflection, wherein the first surface among the principal surfaces is positioned to face the eye of a viewer; a lens associated with the first surface among the principal surfaces, configured to selectively apply light output to incident light according to a polarization direction, wherein the lens applies light output to incident light reflected from the eye polarized in a first polarization direction to collimate the incident light reflected from the eye polarized in the first polarization direction, and the lens substantially does not apply light output to incident light coupled to the outside of the substrate polarized in a second polarization direction orthogonal to the first polarization direction; an illumination array arranged to illuminate the eye with light polarized in the first polarization direction so that a portion of the light polarized in the first polarization direction is reflected back toward the lens by the eye and collimated by the lens; as an optical module, the optical module is a reflective display device that, in response to illumination from a light source, generates light polarized in the second polarization direction and corresponding to an image. An optical module comprising an optical body configured to collimate the light generated by the reflective display device to generate collimated light polarized in the second polarization direction, and an optical sensor; an optical coupling configuration configured to combine the collimated light polarized in the second polarization direction with the substrate and propagate within the substrate by internal reflection in the first propagation direction; and a plurality of partial reflective surfaces disposed within the substrate inclined with respect to the main surface of the substrate, wherein the plurality of partial reflective surfaces are configured to externally combine a portion of the light polarized in the second polarization direction propagating within the substrate in the first propagation direction, and are configured to internally combine a portion of the collimated light polarized in the first polarization direction incident on the first surface among the main surfaces to propagate within the substrate in the second propagation direction, and the optical coupling configuration is further configured to externally combine the propagation light polarized in the first polarization direction.A device comprising at least one processor electrically coupled to the optical sensor, wherein the optical body of the optical module is further configured to receive external coupled light by the optical coupling configuration and convert the external coupled light into a convergent beam of captured light, the optical sensor is configured to detect the captured light, and the at least one processor is configured to process a signal from the optical sensor to derive the current gaze direction of the eye. Claim 17 As a device, a light-transmitting substrate having at least two parallel principal surfaces for guiding light by internal reflection, wherein the first surface among the principal surfaces is positioned to face the eye of a viewer; an image projector that generates light corresponding to a collimated image and polarized in a first polarization direction; a first optical coupling configuration that incidents light corresponding to the collimated image onto the substrate so that the light corresponding to the collimated image propagates within the substrate by internal reflection from the parallel principal surfaces; a second optical coupling configuration that deflects a portion of the light corresponding to the collimated image propagating within the substrate; and an optical element associated with the first surface among the principal surfaces, wherein the optical element is configured to selectively apply a light output to the incident light according to at least one characteristic of the incident light including the polarization direction of the incident light, the optical element applies a light output to the incident light reflected from the eye that is polarized in a second polarization direction orthogonal to the first polarization direction to collimate the incident light reflected from the eye that is polarized in the second polarization direction, and the optical element collimates the incident light coupled to the outside of the substrate An apparatus comprising: an optical element in which, without substantially applying an optical output to the light corresponding to the image, the second optical coupling component deflects collimating light polarized in the second polarization direction toward the substrate so as to propagate within the substrate by internal reflection; an optical body associated with the substrate and configured to convert collimating light polarized in the second polarization direction propagating within the substrate by internal reflection into a convergent beam of capturing light, wherein the first optical coupling component deflects a portion of collimating light polarized in the second polarization direction propagating within the substrate toward the optical body; an optical sensor configured to detect the capturing light; and at least one processor electrically coupled to the optical sensor and configured to process a signal from the optical sensor to derive the current gaze direction of the eye. Claim 18 delete Claim 19 delete Claim 20 delete Claim 21 delete Claim 22 delete Claim 23 delete

Images