Optical systems for eye tracking
By using holographic optical elements with large numerical apertures in eye tracking systems, infrared light is converged to specific positions of the eyes, and using deflection reflection technology, the low refresh rate, high power consumption and window limitations in the prior art are solved. The problem has achieved efficient and accurate eye tracking, improving the user experience of AR and VR applications.
Patent Information
- Application Number
- CN202280042190.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-06-14
- Filing Date
- 2022-06-08
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-06-08
AI Technical Summary
Existing optical eye tracking systems have problems such as low refresh rate, high power consumption, high computing requirements and window limitations in amplified reality (AR) and virtual reality (VR) applications, resulting in accuracy and User acceptance is limited.
Using a holographic optical element (HOE) design with large numerical aperture, the light from an infrared light source is converged to the center of curvature of the eye through HOE, and the scintillation spot is focused on the photodetector using the deflection reflection technology to achieve low time Delay and high refresh rate eye tracking.
A compact, cheap, low latency, and high accuracy eye tracking system is realized, which can provide efficient eye tracking in AR or VR, reducing computing requirements and power consumption, and improving user experience and system efficiency.
Smart Images

Figure CN117546073B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an optical system for eye tracking, such that it can form part of an eye tracking system. The optical system can form part of an augmented reality (AR) or virtual reality (VR) system. Background Art
[0002] Eye tracking has many applications in user interface research or optimization, health, psychology, automotive safety, military and accessibility, and is also used to understand user behavior in response to stimuli such as advertising. An emerging field is that of augmented reality (AR) and virtual reality (VR), where eye tracking is expected to be an important component, allowing user interface control and helping to provide high-quality virtual images to users. AR provides virtual images overlaid on the real world, and more and more applications are being developed for this technology.
[0003] Eye tracking techniques are generally split into three groups: optical eye tracking; eye-attached tracking; and potentiometric measurement. Optical eye tracking is most commonly used because it is non-invasive and inexpensive, and is the technique discussed within this disclosure.
[0004] There are two main types of optical eye tracking systems: table-mounted with a remote camera; or head-mounted in glasses or helmet-sized form factors. Eye tracking is understood to measure the rotation of the eyes. In contrast, gaze tracking is usually understood to include determining what the user is looking at in the real world, and includes additional tracking of the head position. This is usually achieved using an outward-facing camera (world camera) or accelerometer on the head-mounted device. Ideally, calibration is used to correlate eye rotations with gaze direction. Eye tracking is discussed in textbooks such as "Eye Tracking: A comprehensive guide to methods and measures" by Holmqvist et al. (Oxford University Press, 2011).
[0005] Optical eye tracking head-mounted systems typically have an infrared (IR) light source to illuminate the eye, and an IR camera (e.g., a charge-coupled device, CCD) to capture images of the eye, both mounted on a head-mounted structure (e.g., a glasses frame) near the eye. The machine vision algorithm can then determine the position of the corneal reflection from the light source (also known as a glint or first Purkinje image or spot), and the position of the pupil. The vector between the center of the pupil and the glint determines the eye rotation and therefore the direction of sight. In principle, there are multiple Purkinje reflections that can be used, but in practice other reflections are usually too weak. However, the computational requirements are high, so most eye trackers run at relatively low frame rates or refresh rates (30-200Hz, 4-33ms latency). Such a low refresh rate limits the application of eye tracking in AR. There are two main types of eye movement: focus; and saccade. Saccades are fast eye rotation movements (up to 900 degrees per second), so tracking saccade movements requires significantly higher refresh rates.
[0006] The accuracy of optical eye tracking, i.e. the angular deviation of the calculated gaze point relative to the true gaze point, is also important. This accuracy is typically between 0.5 and 2 degrees.
[0007] The power consumption of optical eye tracking systems is particularly significant for wireless AR glasses, which are designed to be worn for long periods of time and are therefore expected to be lightweight and have a long battery life. To this end, computational requirements should be kept to a minimum. Current head-mounted eye tracking devices have high power consumption (up to 1W).
[0008] For AR, machine vision computation is made more difficult by the eye's off-axis view from the camera, which is often necessary to provide compact form factors and meet transparency requirements of AR glasses. For VR, the camera can be off-axis or on-axis (e.g., using a hot mirror beam splitter). Eye tracking in AR or VR allows foveated rendering and other computational effects to reduce computational power, increase visual comfort, and reduce motion sickness. However, this depends on low-latency eye tracking.
[0009] AR products are still to achieve consumer acceptance. One reason for this problem is the trade-off between brightness, field of view (FOV), and viewport (defined as the area within which the eye can see a virtual image) inherent in existing AR optical system designs. Current compact AR products are typically in one of two configurations: a waveguide design; or a virtual retinal display (VRD). The waveguide design has a large viewport created by replicating the virtual image, but this reduces brightness and has low power efficiency, and is expensive to manufacture. The VRD design is cheaper and more compact, has a bright image and high power efficiency, but has a small viewport. The small viewport can be alleviated by equipping each user with a pair of customized glasses, but this is expensive. Using low-latency eye tracking, the VRD design can be improved to overcome this viewport limitation. For example, a single viewport can be manipulated via eye tracking via a microelectromechanical system (MEMS) mirror, or multiple separate viewports can be generated and turned on or off depending on where the eyes are looking. This requires improving existing eye tracking systems in view of the problems raised above.
[0010] Holographic optical elements (HOEs) are used in AR because they are largely transparent and can reflect virtual images from the projector to the eye. Typically, HOEs reflect visible light (400-700nm), but they can be designed to reflect infrared (IR) light (700-1500nm). IR and visible transparent holograms can be overlaid on each other to combine functions. Preferably, IR light illuminates the eye because it is invisible to the user (especially wavelengths > 820nm). US-10,838,362B2 is an example of an optical eye tracking system that uses HOEs to act as reflectors and thereby provide on-axis viewing of the eye to the IR camera, allowing lower computational requirements. However, the computational requirements are still higher than the desired computational requirements. These HOEs use "free space" holograms, so called because they act like transparent concave mirrors, reflecting flashes to the camera. Due to the reciprocal nature of these HOEs, IR light can also be directed to the eye through the hologram and reflected back to the camera. The commercialization of HOE-based systems has been hampered by the lack of holographic materials sensitive to IR light, but new suitable materials are now available.
[0011] In order to reduce the computational requirements, US-8,939,579B2 proposes to measure only the glint reflection from the cornea. Here, the curve of the cornea acts as a convex mirror, which reflects the illumination IR point source (LED or laser) back as a light spot toward the camera or detector via a lens. The collimated beam is directed to the eye, and a transparent waveguide and HOE are used to redirect the glint reflection (or the first Pukin field image), and this light spot is focused onto the camera through a lens. In principle, the position of this light spot on the camera corresponds to the rotation of the eye, and requires less calculation than calculating the relative position of the pupil and the glint. However, it is difficult to achieve high accuracy, especially by focusing the light spot to a small size.
[0012] US-10,712,571 B2 takes a similar approach, but uses technology from an optical mouse sensor, which works by detecting the relative movement of a laser speckle pattern relative to a diffuse surface as the object or sensor moves. A collimated beam is directed to the eye, and the speckle pattern is redirected to the camera via a lens using a waveguide and HOE. This may require significantly less computation (and therefore have lower latency) than other methods. However, tracking the speckle pattern from the eye with a curved, watery surface remains challenging.
[0013] US-2021 / 0041693A1 and WO-2020 / 043472A1 suggest using HOE to illuminate the eye, and incorporating an IR source into a VRD-based AR projection system, using a laser beam scanner and MEMS mirrors and holograms to focus the image onto the retina via the pupil (or the center of the iris). These two documents describe scanning IR light focused at the pupil and measuring the reflected light via the self-interference effect in the IR laser.
[0014] An optical system for eye tracking that is compact and allows low computational requirements, low latency (allowing high refresh rates), and high eye tracking accuracy is therefore desirable. Summary of the invention
[0015] In this context, an optical system for eye tracking according to claim 1 is provided, as well as an augmented reality (AR) or virtual reality (VR) system according to claim 25. Further optional and / or advantageous features are defined in the dependent claims.
[0016] A light source and a light detector (which are preferably but not necessarily co-located and / or integrated) can be used to indicate position information of one or more scintillation reflections. A holographic optical element (HOE) arrangement (which preferably uses a single HOE, but multiple HOEs may be used in some embodiments) receives light from the light source, directs the light to the eye (advantageously, focuses or converges the light on the eye), receives one or more scintillation reflections from the eye and directs these reflections to be focused at the light detector. The light source is advantageously a point light source and / or emits divergent light. The light is preferably infrared light, but in some cases it can be visible light. The HOE arrangement (or at least the portion that redirects the scintillation to the detector) has a large numerical aperture (NA), specifically large enough that ambient or stray light is out of focus at the detector. For example, the NA may be at least (or greater than) 0.2 or 0.3, preferably about 0.5 and typically not more than 0.8. This can be achieved at least in part by sizing and / or setting the HOE (hologram) to a size and / or optical power sufficient to achieve a large NA.
[0017] An eye tracking system may thus be provided with a free space HOE that acts as an off-axis transparent reflector with a large numerical aperture to focus the IR light to the eye, specifically to a point behind the surface of the cornea, for example between the center of the cornea and sclera, more preferably behind the iris (e.g., the center of curvature of the sclera), and most preferably the center of curvature of the cornea (or somewhere between the center of curvature of the sclera and cornea). The same HOE (or in some cases another HOE) may then reflect back and focus an image of the IR source (glint) onto a photodetector. A large NA means that only the glint is in focus, while any ambient or stray light is out of focus. This allows for extremely low computational requirements and therefore low latency eye tracking. Reducing ambient or background light may increase the signal-to-noise ratio of detected glint, thereby improving tracking. In contrast, many existing systems involve focusing light into a narrow beam to achieve accuracy. In laser MEMS or VRD systems (as discussed above), a narrow pseudo-collimated laser beam spot is used, and a mirror moves to scan this spot across the HOE or rasterize this spot onto the eye, so there is only a low instantaneous numerical aperture. In contrast, increasing the numerical aperture of the HOE has been found to improve tracking in view of noise reduction. The design described herein can have an accuracy of about 0.1 degrees and a latency of about 1 ms or less (refresh rate of 1000fps or more), which is a significant improvement over existing designs.
[0018] The optical system can be optimized to provide a compact, inexpensive, low latency, high accuracy, low power eye tracking system with a large numerical aperture (NA) to enable fast eye tracking preferably for use with head-mounted augmented reality (AR) or virtual reality (VR) devices.
[0019] Position information of the eye (e.g., rotation of the eye) may be determined based on, for example, flickers detected at a processor. Specifically, a light detector may detect an image (as a camera and / or array). Then, the position information may be determined based on the position of the detected flicker in the detected image. The light detector may include one or more of: a position sensitive detector; a photoelectric sensor or a photodiode array; and an optical flow sensor. Options for light sources may include vertical cavity surface emitting lasers (VCSELs), laser diodes, and broadband LEDs with one or more filters to reduce spectral bandwidth. Preferably, there is no lens between the HOE arrangement and the light detector. This may be advantageous in reducing the overall NA of the system and thereby avoiding stray or ambient IR light (specifically, sunlight) from affecting performance. Alternatively, in some embodiments, an autofocus lens may be disposed between the HOE arrangement and the light detector.
[0020] Advantageously, the HOE arrangement may include one or more point-to-point holograms, each of which is recorded to act as an ellipsoidal mirror and / or to focus the received light to two finite focal points. Each hologram or HOE may be configured to focus the light to a different point and / or at a different angle.
[0021] The HOE in the arrangement (or the HOE if only one is used) need not simply redirect the glint, but may use the (or each) glint to replay a hologram, which is then focused onto a photodetector. Additionally or alternatively, the HOE may replay multiple holograms at the eye using light from a light source (typically a spot), each converging at a slightly different angle to generate multiple glints (and / or different holograms may converge to different points, e.g. to the center of the cornea or the center of the sclera). In another option, multiple distinct (infrared) light sources may be used to generate multiple glints.
[0022] The HOE arrangement may include multiple HOEs. For example, a first HOE may reflect light from a light source toward the eye. A second HOE may receive one or more scintillation reflections and focus the light onto a photodetector. Optionally, multiple HOEs may be formed on the same substrate and / or multiplexed (angled). In another embodiment, multiple HOEs are used to couple light into and out of a waveguide, for example a first HOE (or HOE configuration) incouples light from the light source and outcouples scintillation light to a photodetector, and a second HOE (or HOE configuration) arranged along the waveguide and spatially separated from the first HOE outcouples light to the eye and incouples scintillation light.
[0023] The light source and light detector may be integrated by modulating a light emitting diode (LED) to function as a sensor when not emitting light, or by integrating separate light sources and light detectors into a single device.
[0024] In another embodiment, the light source and light detector are spatially separated. For example, a beam splitter can be used to keep the light from the light source and the received scintillation along the same path between the beam splitter and the HOE, but bifurcate the path from the beam splitter to the light source and light detector.
[0025] The light source, light detector and HOE arrangement may be mounted on a frame structure (e.g., eyeglass frame, helmet, shield or the like) for the user to wear, in particular, on his head. The frame may be configured so that the distance between the HOE and the eye is small to provide a sufficiently large numerical aperture for the optical system so that ambient or stray light is out of focus at the detector and / or so that the HOE focuses the IR light behind the cornea and preferably to the center of curvature of the cornea and / or sclera of the eye, as discussed above.
[0026] The optical eye tracking system may form part of an AR or VR system that also includes an optical display system for displaying AR or VR images. The optical eye tracking system may provide eye measurement indications that may then be used to control the optical display system.
[0027] Combinations of various aspects or features from various aspects are also contemplated, where such combinations are feasible. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The present disclosure can be put into practice in several ways and a preferred embodiment will now be described by way of example only and with reference to the accompanying drawings, in which:
[0029] Figure 1 depicts a top-down schematic diagram of a first embodiment of an optical system according to the present disclosure;
[0030] Figure 2 A shows a side view of an eye receiving reflected light from a holographic optical element (HOE);
[0031] Figure 2 B Display Figure 2 Front view of A's eye;
[0032] Figure 3 A shows when the eyes are in the first position Figure 1 Example imaging locations of scintillation reflections in embodiments of;
[0033] Figure 3 B shows when the eyes are in the second position Figure 1 Example imaging locations of scintillation reflections in embodiments of;
[0034] Figure 4 Schematically depict an ellipsoidal mirror;
[0035] Figure 5A schematic diagram showing a second embodiment of an optical system according to the present disclosure using a beam splitter;
[0036] Figure 6 Schematically illustrates a system for recording a hologram according to an embodiment of the present disclosure;
[0037] Figure 7 A schematic diagram showing a third embodiment of an optical system according to the present disclosure using a waveguide;
[0038] Figure 8 A schematically depicts changes on a portion of an optical system according to the present disclosure using multiplexed HOEs for generating more than one scintillating reflection;
[0039] Figure 8 B Display according to Figure 8 Example imaging locations of two scintillation reflections in the system of A;
[0040] Fig. 9 a flowchart depicting a procedure according to the present disclosure;
[0041] Fig.10 showing ray paths from a ray tracing model according to a design of the present disclosure; and
[0042] Fig.11 A top-down schematic diagram depicting a fourth embodiment of an optical system according to the present disclosure using spatially separated light sources and light detectors. DETAILED DESCRIPTION
[0043] A free-space HOE (or HOE system) can act as an off-axis transparent reflector with a large numerical aperture. The numerical aperture of an optical system can be understood with reference to the depth of field of a conventional camera. If the camera reduces its aperture (decreased diameter of the iris, high f-number, e.g., f / 11), the depth of field increases because the light rays entering the camera are largely parallel and each object plane is largely focused at the image plane. Alternatively, if the camera increases its aperture (increased diameter of the iris, low f-number, e.g., f / 1.4), the depth of field decreases (is shallow) because the light rays entering the camera are largely divergent and only one object plane is focused at the image plane. Therefore, the large numerical aperture system described herein will have a shallow depth of field, and any stray and / or ambient light will be out of focus. As a result, the signal-to-noise ratio at the photodetector will be significantly increased, thereby reducing the computational requirements and latency of the eye tracking system.
[0044] First see Figure 1, depicts a top-down schematic diagram of a first embodiment of an optical system according to the present disclosure, typically designed for AR applications. This optical system includes: an eyeglass frame 101; a HOE 102; an IR light source 105; and an IR camera 106 (e.g., a photodetector array, which may be, for example, a CCD). The IR light source 105 and the IR camera 106 are co-located and are therefore shown as a single block in the figure. Preferably, the IR light source 105 and the IR sensor or camera 106 form part of a single device, and they may be integrated, but it is also possible that they are two separate devices that are adjacent and ideally as close to each other as possible. For completeness, two bidirectional light rays 103 and the user's eye 104 are shown.
[0045] Divergent light from IR light source 105 is reflected by HOE 102 to be focused on eye 104. HOE 102 also advantageously converges light to eye 104, as will be discussed further below. Eye 104 provides glint, the image of which then reciprocally converges to a focal spot on IR camera 106 (referred to as retroreflection). For clarity, Figure 1 A monocular version of the embodiment is shown, but a binocular version would provide higher accuracy gaze tracking and can be easily inferred from the diagram.
[0046] The optical system is dimensioned so that the user wears a pair of 'normal' glasses, shown by frame 101. HOE 102 is a transparent hologram laminated onto or encapsulated in the lens of the glasses (the lenses are not shown, but can be either optometry or non-optometry lenses, and can be curved or flat). An IR sensor 106 (CCD or photodiode array) is placed at the temple of the frame, pointing toward the hologram. The IR sensor 106 is optimally sensitive to 850nm IR light, preferably with a filter to exclude undesirable light (wavelengths less than 840nm and / or greater than 860nm). An IR light source 105 (LED or IR laser) is directed toward the eye 104, and as discussed above is placed as close to the IR sensor 106 as possible (a beam splitter can be used to effectively overlap the sensor and IR source, examples of which are discussed below).
[0047] The hologram or HOE 102 reflects and focuses the divergent IR light 103 onto the eye (thus it converges the IR light onto the eye) so that the light converges or focuses 8 mm behind the front end of the cornea (specifically, at the center of curvature of the cornea). The divergence and power of the IR source are designed to remain eye safe. The form factor of a VR headset can be similar, with the HOE or hologram laminated onto the VR lens between the screen and the user (the hologram may need to be compensated for the curvature of the VR lens).
[0048] When wearing the glasses, the user's eye 104 is in close proximity to the hologram 102, with a typical eye relief (distance from the eye to the hologram) of 20 to 30 mm. Thus, with a typical square hologram with a side of 25 to 50 mm, most of the IR light reflected from the eye 104 will be incident on the hologram 102, resulting in a large numerical aperture optical system. The hologram 102 reflects the IR light from the eye 104 and focuses it onto the IR sensor 106. This appears as a small spot on the IR sensor 106, and as the eye 104 rotates, the spot moves across the IR sensor 106, allowing the effective rotation of the eye 104 to be measured.
[0049] In effect, the divergent IR point source 105 has been retroreflected and refocused back onto itself, so the position of this small point source can be measured on the IR detector 106 to provide a low latency measurement of the rotation of the eye 104. This can be considered equivalent to a point source (object) at the radius of curvature of a spherical concave mirror, with a diffuser placed next to the point source so that the retroreflected spot (image) can be viewed. As the spherical mirror rotates, the spot will translate laterally across the diffuser. Within a limited range of rotation, the spot size will remain small enough so that the position of the spot can be measured, and therefore the rotation of the mirror determined.
[0050] refer to Figure 2 A, shows a side view of the eye 201 receiving and reflecting light from the HOE 203. Also refer to Figure 2 B, which shows Figure 2 Front view of the eye of A. The eye 201 is shown together with the cornea 202, the sclera (eyeball) 205, the iris 206 and the pupil 207.
[0051] HOE 203 converges the IR light onto the center of curvature R 209 of cornea 202, and the IR light is retro-reflected back to HOE 203. The surface of cornea 202 acts as a convex mirror. Focal point F 210 is shown if the eye is illuminated with collimated light. Typical eye tracking systems illuminate the eye with divergent IR light, or less collimated or slowly converging IR light, and as will be explained below, the problem with this is that IR light reflected from sclera 205, iris 206, and the retina of the eye (back side of sclera 205) is also focused along with the desired IR light reflected from the surface of cornea 202 (glint 208). This reduces the signal-to-noise ratio and increases computational requirements.
[0052] Typically, the center of curvature R 209 is 8 mm behind the surface of the cornea 202, the iris 206 is 2 mm behind the surface of the cornea 202, and the F 210 is 4 mm behind the surface of the cornea 202. The proximity of the iris 206 to the F 210 indicates that illuminating the eye with converging light, particularly converging to the center of curvature R 209, is optimal for blurring the undesirable image of the iris (especially compared to divergent or collimated illumination).
[0053] As described above, the cornea 202 has a radius of curvature R 209 of approximately 8 mm. And, the sclera 205 has a radius of curvature of approximately 12 mm. The cornea 202 acts as a convex mirror and can therefore be analyzed using the mirror equation 1 / f = 1 / o + 1 / i (where f is the focal length, o is the object distance and i is the image distance). If the cornea is illuminated with collimated light (so that the object distance o is infinity), the focal length (f) is at f = R / 2 = 4 mm. Because the iris 206 is 2 mm behind the cornea, both the iris 206 and the glint 208 (focal point, 4 mm from the corneal surface) are in focus, thereby reducing the signal-to-noise ratio.
[0054] In contrast, the cornea 202 may be illuminated with a fast converging beam, specifically converging to 8 mm behind the surface of the cornea 202 (center of curvature R 210 of the cornea 202). In this case, the object and (virtual) image positions are virtually the same, 8 mm behind the surface of the cornea 202 at the radius of curvature R 210. This can be compared to a concave mirror with a point source at the radius of curvature, thereby forming a retro-reflective system. As a result, the iris 206 (2 mm behind the surface of the cornea 202) is further away from the image (8 mm behind the surface of the cornea 202). The light reflected from the iris 206 is largely defocused at the HOE 203, thereby advantageously increasing the signal-to-noise ratio.
[0055] Thus, converging to the center of curvature of the cornea may mean that the wavefront of the converging beam perfectly matches the wavefront of the reflected beam from the surface of the cornea. In other words, the reflection from the surface is a back reflection. Beneficially, this may give the lowest possible aberrations and therefore the smallest possible spot size on the detector or camera. Even if there is some divergence from the center of the cornea (e.g., due to eye movement), it may still be beneficial if the light from the HOE converges to the center of curvature of the sclera (or instead between the two centers of curvature). Converging to the center of the sclera (eyeball) may not necessarily produce the minimum aberrations, but the aberrations may still be significantly reduced compared to other configurations, and as the eye rotates, the aberrations may be lower at the edge of the rotation compared to the corneal reflection.
[0056] It should be noted that although the above description focuses the IR light behind the cornea 202, the light does not actually reach the center of curvature R 210 of the cornea 202. In fact, this focus represents a virtual image because the imaged IR light reflects from the surface of the cornea 202, which acts as a convex mirror. Therefore, the refractive index of the eye is not relevant to the reflected glint 208.
[0057] Figure 2 The front view of the eye in B shows the sclera 205, iris 206, pupil 207, and glint 208 (first Pukin field image). As the eye rotates, the position of glint 208 will appear to move laterally across the surface of the eye at a different apparent rate than the center of the pupil. Typical eye tracking systems calculate the vector between glint 208 and the center of the pupil to determine the line of sight. In the design described herein, light reflected from the eye (sclera 205, iris 206, or retina) is blurred at the image plane and only glint 208 (spot) is visible.
[0058] HOE 203 can then focus the reflected light onto an IR detector (see Figure 1 discussed), such as CCD ( Figure 2 (not shown in FIG. 2A or 2B ) Angle θ212 is the half angle at which rays from HOE 203 converge to the center of curvature of cornea 202 , which is the angle used in object space numerical aperture (NA) calculations.
[0059] Typical existing eye tracking systems have an NA of about 0.01 to 0.1 (where NA = n*sin(θ), where θ is the half angle between the eye and the hologram, and n is the refractive index of air, 1). An optical system will have an object space NA, which is the cone of angles of light captured from the object (i.e., how far or close the lens or mirror is to the object); and an image space NA, which is the cone of angles captured by the detector (i.e., how far or close the lens is to the detector). The HOE NA is the range of angles used when recording the hologram, i.e., how much the lens diverged or converged when recording it.
[0060] In theory, the HOE NA and object space NA should be the same, and although they are different in practice, the HOE NA, along with the eye relief, is a key component of the object space NA. For example, an object space NA of 0.45 can be achieved by having the eye relief and HOE size be the same. An eye relief of 25 mm (which is typical) and a 25 mm diameter HOE can therefore achieve this object space NA, but the dimensional specifications can vary with different designs, and this will assume that the HOE is recorded so that it can utilize the maximum angle.
[0061] The object space NA may define the depth of field: the larger the object space NA, the shallower the depth of field, so the only object in focus will be the desired reflected glint.The image space NA is ideally as large as possible, but any reduction may not be as critical to performance as the object space NA and / or HOE NA.
[0062] A typical camera aperture will be about 1 to 3 mm, and the eye-to-camera distance will be about 35 to 50 mm. VRD-type eye tracking systems with a pseudo-collimated scanning beam (e.g., as described in US-2021 / 0041693A1) typically have a very small NA of about 0.01. The design described herein has a NA of about 0.5 (ideally greater than 0.2 or 0.3, and preferably less than 0.8) because the HOE (hologram) captures most of the IR light angles reflected from the eye. The NA also remains large when no lens is used between the HOE and the IR detector (which will act as a limiting aperture). In fact, the hologram size can determine the exit pupil of the system, which is a significant improvement.
[0063] Because the NA of the system is at least an order of magnitude larger than typical eye tracking systems, the only part of the image that is sharply focused is the glint from the corneal surface, while the rest of the image (e.g., reflections from the iris) is blurry. The NA can be either object space or image space NA, but for this application, the object space NA, which is effectively defined by the angle between the eye and the hologram, is probably the most relevant. Thus, shortening the eye relief and increasing the hologram active area size can increase the object space NA (assuming the hologram is recorded to match this).
[0064] Existing VRD-type systems (e.g., as described in US-2021 / 0041693A1) use a converging beam onto the eye that is reflected back to the sensor, but the IR light is generated by a narrow beam (laser beam scanning, not a diverging point source) and therefore has a lower numerical aperture. In these systems, the point source is actually at the laser beam scanning mirror, and the sensor is physically separated from the mirror, which also limits the numerical aperture.
[0065] Eye tracking systems desirably focus on eye safety requirements, which state that IR radiation must be below a certain power threshold. Because IR (at the eye tracking wavelength) is invisible to the eye, the blink reflex (which protects the eye from high energy light) does not apply. Focusing IR light on the retina can cause photothermal damage and therefore presents a significant danger. Eye tracking systems therefore maintain a balance between eye safety and having enough power to create a suitably bright image (signal-to-noise ratio). Advantageously, embodiments according to the present disclosure focus the IR light to a point within the sclera that is less sensitive to photothermal damage. Obviously, system efficiency is increased because a large portion of the IR light is utilized due to the retro-reflective design, and noise is significantly reduced. Therefore, a relatively low power IR source can be used, so that the risk of photothermal damage is mitigated.
[0066] Typical eye tracking systems have a lens in front of the CCD to capture a wide FOV image of the eye. The physical lens aperture limits the numerical aperture and also focuses undesirable ambient IR light onto the CCD, which reduces the signal-to-noise ratio. This can be a significant problem when using such an eye tracking system outdoors, as the sun emits a large amount of ambient IR radiation. Therefore, the absence of a lens in the novel design described herein is advantageous.
[0067] Next reference Figure 3 A and 3B show when the eyes are in the first and second positions respectively Figure 1 302. Example imaging positions of glints in embodiments of the present invention. In both figures, a portion of a pixelated CCD 302 is shown. The eye in the first position 301 is facing forward, and the approximate center position of the glint spot 303 on the pixelated CCD 302 is shown. As the eye rotates, the position of the glint in the image plane translates laterally. The eye in the second position 305 rotates from the first position, and the approximate center position of the glint reflection spot 304 on the pixelated CCD 302 translates accordingly. As long as the spot size is smaller than the pixel size, the rotation of the eye can be accurately measured (+ / - 0.1 degrees). This readout (where only the highest pixel value can be indicated) allows extremely high frame rates (1000fps or more) and equivalent low latency (1ms or less).
[0068] Figure 1 Embodiments of (which may be optimal for a pair of compact AR glasses) may have limits on the eye rotation that can be measured because the magnification is close to 1:1. Once the eye rotates far enough, the spot will move outside the photodiode active area and will not be detected. Photodiode arrays typically have an active area side size of about 1 to 10 mm (and can be significantly larger while still fitting into the eyeglass frame).
[0069] A numerical example (approximation) of eye rotation is that the eye rotates comfortably naturally by 15 degrees, up to a maximum of 45 degrees, and then the head rotates instead. Considering the curvature of the eye relative to the virtual glint position and the distance of the hologram from the eye and CCD, an approximate example is that as the eye rotates 15 degrees, the eye shifts 3mm laterally, and the glint shifts 1mm laterally at the eye. On the CCD, the spot then shifts 2mm. Therefore, for an average CCD width of 4mm, approximately + / - 15 degrees of eye rotation can be tracked before the imaged glint moves out of the CCD active area.
[0070] The detectable rotation angle of the eye may thus be limited to approximately + / - 15 degrees. This may not be a problem for situations where detection of the eye across its entire rotation angle is not required (e.g., exit pupil steering for a VRD system with limited FOV). It may also be possible to use larger photodiodes or multiple photodiode arrays in other eye tracking systems with fewer constraints on size specifications.
[0071] The RMS spot size on the photodiode can be directly related to the accuracy of the eye tracking system. If the spot size is larger than the pixel pitch of the photodiode, the spot will overlap multiple pixels. Therefore, the position may not be easy to know and the accuracy of the eye movement measurement is reduced. The RMS spot size is a function of the point source size (the size of the laser or LED; the LED can be masked with a pinhole) and the 'resolution' of the hologram (e.g., the point source size used when recording the hologram). The spot size can also increase as the eye turns and the glint moves away from the best focus of the HOE (acting as an ellipsoidal mirror). The spot size can also increase as the distance of the IR source to the photodiode increases, because again the spot will not be in the best focus. This means that although a larger photodiode can measure larger eye rotation angles, the spot size can increase towards the edge of the photodiode, so the accuracy may decrease. If the photodiode is tilted towards the center of the HOE (ellipsoidal mirror), that is, the photodiode surface is perpendicular to the chief ray, the spot size can be reduced due to minimizing the tilt of the image plane. However, due to the need to reduce the volume in the eyeglass size specifications, this may not be possible.
[0072] The spot size may also increase with the spectral bandwidth of the IR source, the size of the point source, and the off-axis angle of the hologram (due to the holographic blur function). Narrow bandwidth, small diameter laser sources (VCSELs or laser diodes) may be considered preferred over broadband LEDs for this reason. Although the image resolution of laser diodes is degraded due to speckle artifacts in normal camera-based imaging systems, according to embodiments of the present disclosure, speckle is insignificant. Alternatively, a broadband LED source may utilize a filter to reduce its spectral bandwidth, and assuming a bandwidth of 10 nm (FWHM), this will produce an acceptable spot size. The off-axis angle should be minimized while maintaining a compact size specification. Suitably configured LEDs may therefore be preferred over laser VCSELs because LEDs have a large spherical divergence compared to the lower (typically elliptical) divergence of laser diodes or VCSELs. This may allow for more uniform illumination of the entire eye from the distance of the IR source.
[0073] If the distance of the photodetector to the eye changes relative to the nominal design distance (e.g. due to the position of the glasses on the head of different users), the spot size may increase. This error may be reduced by allowing the eye relief or CCD position to be adjusted within the eyeglass frame. A lens over the CCD may also reduce this error, but would be undesirable as this would reduce the numerical aperture and thus the desired depth of field effect. However, a lens may be used as long as the lens diameter is large enough without constraining the NA (in image space). Preferably, an autofocus lens (e.g., a liquid lens or a movable lens stack) may be used, which will thus reduce the spot size if there is any variation in the nominal design distance.
[0074] There are generally three main application areas for eye tracking. The first area is academic research, where the user has a chin rest and everything is locked in place. For this case, the position accuracy is high, so the lens in the detector can be eliminated and low latency will have a significant benefit. The second area is advertising or user interface analysis. A pair of strong and / or rigid glasses (or other frames) can be used to limit the position tolerance. One or more of autofocus lenses, calibration, glasses adjustment, and a large detector area can also be used in this case. For AR or VR consumer applications, one or more of autofocus lenses, calibration, glasses adjustment, and a large detector area will most likely be required.
[0075] Due to the absence of machine vision, the configuration according to the present disclosure has low computational overhead (and therefore low power consumption and long battery life). In addition, another advantage in terms of low power consumption can be achieved compared to traditional eye trackers. The light usage is very efficient because a larger percentage of the emitted IR light is captured by the hologram and focused back onto the photodiode. Therefore, the power emission required to achieve a suitable signal-to-noise ratio is reduced. The new IR sensitive material can also achieve very high diffraction efficiency (at least 80%), which can help to maintain high system efficiency.
[0076] The eye tracking system is desirably calibrated to match the user's eye movements to the real world. This can be best achieved using a binocular eye tracking system and an outward-facing visible light camera. The user preferably focuses on known real-world objects (or virtual objects in the case of AR glasses) at different gaze directions and gaze focal planes to calibrate the system and achieve high accuracy gaze tracking.
[0077] In a broad sense, an optical system for eye tracking may be considered, the optical system comprising: a light source arranged to emit light; a light detector for detecting light and indicating position information of the light detected thereby; and a HOE arrangement configured to direct light from the light source to the eye, receive at least one glint of light from the eye and direct light from the received at least one glint to be focused at the light detector. The HOE arrangement is configured (e.g., sized) to have a sufficiently large numerical aperture so that ambient or stray light directed by the HOE arrangement is out of focus at the light detector. The numerical aperture may be at least (or preferably, greater than) 0.1, 0.2, 0.3, 0.4, 0.5 or 0.6, but preferably less than 0.7 or 0.8. The light source is preferably a point light source and / or is arranged to emit divergent light. The light is preferably infrared light. Advantageously, the HOE arrangement is further configured to converge (or focus) light from the light source onto the eye, and preferably to a position at the center of curvature of the sclera or cornea of the eye.
[0078] Also contemplated is an eye tracking method having steps corresponding to the operational features of any of the systems disclosed herein. For example, the method may include: directing (divergent and / or infrared) light to an HOE arrangement; directing light at the HOE to an eye (replaying a hologram); receiving at least one glint of light from the eye at the HOE arrangement; directing light from the received at least one glint to be focused at a light detector; and detecting light at the light detector to indicate position information of the light detected thereby.
[0079] In another aspect, an AR or VR system may be provided, comprising: an optical display system for displaying an AR or VR image; an optical eye tracking system as described herein, configured to provide eye measurement indications; and a processor configured to control the optical display system based on the eye measurement indications provided by the optical eye tracking system.
[0080] Additional optional, preferred and / or otherwise advantageous features applicable to any aspect may also be considered, as discussed below.
[0081] For example, the light detector may be further configured to generate an electrical signal based on the at least one glint detected. Then, the processor may be configured to determine the position information of the eye using the electrical signal and the position information of the detected glint indicated thereby. The light detector may detect an image (specifically, two-dimensional). Then, the processor may determine the position information of the eye based on the identified position of the at least one glint detected in the image.
[0082] A frame structure (e.g., an eyeglass frame, a helmet, or a shield) may be provided and configured to be worn by a user (e.g., head-mounted). The light source, light detector, and / or HOE arrangement may be mounted on the frame structure. The frame structure may position the eye so as to achieve any one or more positional relationships between the eye and other features of the optical system as disclosed herein.
[0083] Advantageously, the light source and the light detector are co-located.
[0084] The HOE arrangement advantageously comprises at least one point-to-point hologram recorded to act as an ellipsoidal mirror and / or to focus the received light to two finite focal points.In an embodiment, the HOE arrangement comprises a single HOE.
[0085] The light detector may include one or more of: a position sensitive detector; an array of photosensors (or photodiodes); and an optical flow sensor. Optionally, the light detector may include an array having a size of at least about 5 mm, about 7 mm, about 8 mm, about 9 mm, or 10 mm, and optionally not more than 10 mm (or both are two-dimensional), for example. Ideally, the detector is as large as possible. In reality, CCD devices and position detectors are typically 10 mm or less, and this would be a reasonable size for a pair of glasses, but larger sizes are possible.
[0086] The light source may include one or more of: a vertical cavity surface emitting laser (VCSEL); a laser diode; and a broadband LED having one or more filters to reduce the spectral bandwidth of the emitted light.
[0087] In some embodiments, there is no lens between the HOE arrangement and the photodetector. Alternatively, a large NA and / or autofocus lens may be provided between the HOE arrangement and the photodetector.
[0088] Further details of specific embodiments will now be discussed. The following will then return to the overall perception of the above detailed description.
[0089] The hologram of the HOE is designed as a point-to-point hologram and is recorded so that it acts as an ellipsoidal mirror. This is similar to the off-axis ellipsoidal mirror sold by Edmund Optics Ltd. using item number 12497. Figure 4, schematically depicting an ellipsoidal mirror 401. This type of mirror is a finite-finite conjugate mirror, with two foci F1 and F2. This is different from a spherical mirror (one focus) or a parabolic mirror (two foci, but one at infinity).
[0090] The IR sensor is preferably a position sensitive detector (PSD), such as the device manufactured by Thorlabs and sold under part number PDQ80A. This device has only two output lines, allowing very fast (low latency) readout. Alternatively, a low-cost silicon (square or rectangular) photodiode array (e.g., CCD or CMOS sensor) can be used, ranging in size from 10x 10 pixels to 100x 100 pixels, with a pixel pitch of about 3 to 50 microns and a total side dimension of about 1 to 10 mm. Such a low pixel count photodiode will allow per-pixel readout, thereby allowing extremely low latency readout. As described above, the rotation of the eye moves the back-reflected IR spot across the photodiode array. The IR spot will generate a higher photocurrent on the individual pixel it is hitting, so the xy position of the relevant photodiode that directly corresponds to the eye rotation can be determined. A low-cost optical mouse sensor (optical flow sensor, which can track at up to 6400fps) can also be used because it has a built-in algorithm to detect movement. Existing eye tracking systems typically use large CCDs with tens of thousands or millions of pixels, which read out much slower than smaller photodiode arrays. The CCD information must then also be interpreted via machine vision to provide eye movements, which adds latency. For example, due to the use of waveguides and pseudo-collimated IR light, the existing systems described above have a small numerical aperture (shallow depth of field), which limits the available angles of IR light that can be captured. Therefore (and as described above), some ambient and stray or undesirable light is also focused on the sensor, reducing the signal-to-noise ratio and increasing computation time.
[0091] The ellipsoidal mirror is optimized for high numerical aperture and low aberration. Eye tracking designs can be implemented using this (or similar) mirror. The center of curvature of the user's cornea is at position F1 and the CCD (or equivalent IR detector) is at position F2. Exhibits retro-reflective properties of two foci. Figure 4 The elliptical mirror shown in is not transparent, so its applications are limited. Holograms designed to act as elliptical mirrors while being transparent can be used in the present disclosure for a wider range of applications.
[0092] This design appears to provide the best performance in terms of numerical aperture and spot size (usually measured as root mean square RMS, with spot size in microns). The smaller the spot size, the lower the latency and the higher the accuracy of the eye tracking system. It should also be noted that the hologram is reflected towards the eye and also back to the camera. By their nature, holograms are reciprocal, i.e., a beam coming from either direction will interact with the hologram.
[0093] Increased accuracy can be achieved by reducing the separation between the photodiode and the point source. For example, a modulated IR LED can be used. The IR LED is turned on (forward biased), then quickly turned off, and a reverse voltage bias is applied to the LED (e.g., as discussed in US-2020 / 0089318A1). The LED acts as a photodetector (rather than a photoemitter) when reverse biased, and generates a small current when the back-reflected IR light is incident back on the LED. If an array of IR LEDs is used and sequentially modulated, the eye rotation angle can be determined based on which IR LED measures some photocurrent. However, LEDs are generally not optimized for this, so the results may be worse than other techniques. It is also known to have a photodiode with an IR sensor (camera) embedded behind a partially transparent photodiode array (or CCD). These photodiodes can be used, but are not currently widely commercially available.
[0094] Although the IR light source and IR detector are preferably integrated or at least as close together as possible, this is not absolutely necessary. Figure 5 , shows a schematic diagram of a second embodiment of an optical system according to the present disclosure using a beam splitter. Only a portion of the design is shown, as the remaining details can be understood with reference to other embodiments and information disclosed herein. The design includes: HOE 501; light source 504; beam splitter 504 (e.g., 50:50 IR beam splitter); and light detector 505 (e.g., CCD).
[0095] The chief ray 506 showing the illumination path is from the light source 503 via the beam splitter 504 and the HOE 501 to the eye 502 and then back reflected via the beam splitter back to the photodetector 505. The distance from the light source 503 to the beam splitter 504 is ideally the same as the distance from the photodetector 505 to the beam splitter 504. The beam splitter 504 allows for a minimum spot size on the photodetector 505. Although this design has a smaller footprint than Figure 1 Embodiments of the present invention have larger form factors, but their configuration will allow for a smaller spot size than the alternative of placing the light source as close as possible to the light detector.
[0096] Reference now Figure 6 , which schematically shows a system for recording a hologram (specifically, a HOE) for use in an optical system according to the present disclosure. In order to achieve a high numerical aperture of the HOE upon playback, high numerical aperture (i.e., rapidly converging and / or diverging) recording optics are used. The system includes: a coherent laser 601; a beam splitter 602; a first mirror 603; a second mirror 604; a spatial filter 605; an IR sensitive material 606; and a high NA lens 607. Only the chief ray of light is shown in this schematic for clarity.
[0097] 850nm coherent light from laser 601 is split into two paths by beam splitter 602. The first path of the split light (object beam) is reflected by first mirror 603 onto large lens 607. Lens 607 has a high NA and focuses the beam to a small spot where the HOE will focus the received light at playback (the eye will focus, for example, through Figure 1 The frame in the embodiment of is positioned there). As a result, the object beam converges.
[0098] The second path of the split light (reference beam) is directed via a second mirror 604 to a spatial filter (a small pinhole), thereby generating a divergent beam. The spatial filter is actually located in the position where the light detector will be during eye tracking using the generated HOE (the pinhole is actually located where the CCD will be). The reference beam is therefore divergent. The NA of the two recording beams should be as large as possible to achieve the maximum NA on playback. In particular, it should be emphasized that in practice the lens 607 is evenly illuminated from the mirror 603 (optimally with a large collimated beam), but as mentioned above, only the main ray is drawn for clarity. The resulting hologram is recorded as a point-to-point reflection hologram with the desired playback angle, and the off-axis elliptical mirror (as described above with reference to FIG. 1 ) is replicated. Figure 4 discussed above), but can also be transparent.
[0099] The hologram is preferably recorded at a desired IR playback wavelength that is not visible to the eye (e.g., 850 nm, as discussed above), where photosensitive materials, a coherent recording source, an IR playback source, and a camera are available. Recently it has become possible to obtain IR materials that are sensitive to 850 nm, which is the optimal wavelength considering the above factors. It is possible to record with holographic materials that are sensitive only to visible wavelengths, and change the recording angle to achieve the desired playback IR wavelength and angle. However, this may introduce aberrations and thus increase the final spot size. If collimated beam recording is used (rather than as Figure 6 ), this aberration can be minimized, but it may thus prevent the realization of a large numerical aperture system.
[0100] Although specific embodiments have now been described, the skilled person will appreciate that various modifications and alterations are possible. For example, although infrared light is preferably used, other wavelengths (such as visible light) may be employed. One such option may be to use some of the projected light from an AR projector (projecting an image for the user to see) to create glints. Divergent light is preferred, but collimated or converging light may be used in some embodiments.
[0101] It is of course understood that variations in design distances, dimensions, curvatures, angles, and component sizes are possible without changing the basic disclosure. Also, additional optical components may be incorporated to redirect and / or process light as desired. For example, filters are optionally used with eye tracking systems to block visible light and reduce noise. Additional methods in accordance with the present disclosure will now be discussed, recognizing that combinations of these methods and / or combinations with the configurations discussed above may be further contemplated.
[0102] One such possible variation involves the incorporation of waveguides. Figure 7 , a schematic diagram of a third embodiment of an optical system is shown accordingly. This optical system includes: a waveguide 704; an incoupling HOE 703; an outcoupling HOE 702; an IR light source 706 and an IR light detector 705 (e.g., a CCD). The light source 706 and the light detector 705 are shown as separate blocks, but they are co-located and / or integrated.
[0103] Divergent IR light from the light source 706 is incoupled into the waveguide by the incoupling HOE 703, propagates along the waveguide (due to total internal reflection), and is coupled out of the waveguide by the outcoupling HOE 702 and directed toward the eye 701 (and preferably converges to the center of curvature of the cornea or sclera of the eye). Glint from the eye 701 is incoupled into the waveguide by the outcoupling HOE 702, propagates along the waveguide, and is coupled out of the waveguide by the incoupling HOE 703 and directed toward the photodetector 705 (and focused to a spot on the photodetector).
[0104] This configuration can reduce the form factor, but may be undesirable due to limited numerical aperture (since the waveguide can limit the illumination angle) and possible aberrations. Existing approaches using waveguides are described in US-8,939,579 B2 and US-10,712,571 B2, for example using a set of compound lenses to focus the spot onto the sensor in an attempt to limit aberrations, which can be Figure 7 However, this would also limit the numerical aperture and is therefore not preferred. Another variation is also contemplated in which the light source 706 and the light detector 705 are spatially separated using a beam splitter to be directed along a common axis (e.g., as shown in FIG. Figure 5 described) redirecting light.
[0105] Another alternative approach (which can be combined with other approaches described herein) can use a multiplexed HOE design. One such approach can allow tracking of a wider range of eye rotation angles in a compact manner by recording multiple holograms into the HOE holographic mirror. Each of these holograms can act on the same IR light source, but converge the light at slightly different angles to generate multiple spots (flashes) on the eye (angle multiplexing). Each spot will then be reflected back and focused onto a photodiode, producing multiple spots. In fact, each of the multiplexed HOEs will be a point-to-point hologram recorded to act as an elliptical mirror, with one focus (detector point) that is the same for all HOEs, and the other focus will be slightly different for each HOE (eye point). As the eye rotates, some spots will move away from the photodiode active area, but some spots will move onto the active area. Therefore, there will always be at least one spot on the photodiode, increasing the measurable rotation angle of the eye.
[0106] View now Figure 8 A, schematically depicts a change in a portion of an optical system that uses angle-multiplexed HOEs to generate more than one glint according to this method. Shown are: a photodetector array (e.g., CCD) 802; a first HOE 803; and a second HOE 804. Example light rays 805 are depicted for illustration purposes. The light source is not shown, but its positioning can be, for example, from Figure 1 and 5 design inferences.
[0107] The first HOE 803 focuses the received light to the center of curvature of the eye 801. The second HOE 804 also focuses the light to the center of curvature of the eye 801, but at a slightly different angle, to a slightly different location on the eye 801. The angle of the lens through which the object beam is focused can be adjusted (e.g., at Figure 6 The lens 607 in the configuration of FIG. 804 may record the first 803 and second 805 HOE (or more) into the same holographic layer using sequential recording. Alternatively, multiple HOEs may be recorded separately and laminated superimposedly.
[0108] refer to Figure 8 B, showing the Figure 8Example imaging locations of two flashes in a pixelated photodetector array (CCD) 806 corresponding to the photodetector 802 of A. A first light spot 807 and a second light spot 808 are generated on the image plane of the detector array 806 from a single point source (not shown). This design can provide two identical (but spatially separated) light spots on the CCD that move relative to each other in a reasonably (but not completely) linear manner as the eye rotates. It will also be possible to manufacture one (or more) HOEs so that a diffuse image is generated and focused onto the photodetector array 806, for example a triangle, star or square shape (rather than a circular light spot) can be generated. This may allow easier distinction between 'light spots'.
[0109] Such designs will allow tracking of larger eye rotations, because as the eye rotates, there will always be at least one 'spot' on the CCD at any time and one spot moves away from the detector array active area, another spot created by a different hologram will move onto the detector array active area, thereby tracking over larger rotation angles. Large computational power may be required to track which spot corresponds to which glint, so there may be a tradeoff between latency and maximum measurable eye rotation angle. US-10,838,362 B2 describes the use of multiplexed holograms to generate multiple glints on the eye, but using a camera to view multiple spots on the eye to enhance eye tracking accuracy (rather than measuring the back-reflected glint directly on the CCD). Nevertheless, aspects of this operation may be incorporated into embodiments according to the present disclosure.
[0110] It is also possible to place multiple IR sources around a photodiode (which can be quickly switched on and off (modulated) and coupled to a single (or multiplexed) holographic Figure 1 It is further possible to use multiple wavelengths of IR light and a single hologram to generate multiple spatially separated spots. However, due to the need for multiple sources, this will not be feasible.
[0111] Another advantage of these methods may be to allow different light to converge to different points on the eye, such as some light (from one hologram) converging to the center of curvature of the cornea, and some light (from another hologram) converging to the center of curvature of the sclera (other points may be chosen alternatively or additionally). This may be achieved using multiple holograms, each with a correspondingly different converging power. This may allow tracking of the cornea and eyeball, which may help alleviate problems due to slippage of the glasses relative to the eye.
[0112] refer to Fig. 9, depicting a flow chart of a procedure according to the present disclosure. This method may be applied to any embodiment or configuration disclosed herein. In a first step 901, IR light from a point or divergent source is directed to a hologram (HOE or HOE arrangement) in front of the user's eye. The hologram then directs the light to the eye in a second step 902, and preferably converges the light to a point behind the surface of the eye, for example 8 mm behind the surface of the eye and most preferably at the center of curvature of the cornea and / or sclera. In a third step 903, the light is reflected back from the eye (retro-reflected) to the HOE arrangement.
[0113] In a fourth step 904, the HOE arrangement directs (preferably converges) the back-reflected light to a photodetector, which is preferably a two-dimensional (xy) array position detector. In a fifth step 905, the output of the photodetector is analyzed to determine the two-dimensional position with the maximum signal (peak). Finally, in a sixth step 906, the determined two-dimensional position is converted into an associated eye rotation. Thereby, the gaze direction can be established.
[0114] Reference now Fig.10 , which shows ray paths from a ray tracing model made with software from Zemax LLC with a nominal design according to the present disclosure. This shows the path from the eye 1001 via the HOE 1002 to the photodetector (CCD) 1003. In this model, an eye relief of 24 mm is used, providing a hologram active area with a diameter of 25 mm, a hologram to detector distance of 60 mm, and a detector size of 5 mm square. The object space NA is calculated to be 0.447, which is an advantageously high value that cannot be achieved with other common or known designs.
[0115] As the eye moves, the spot size RMS is approximately 0 when the eye is on axis (user looking straight ahead), 6 microns when the eye moves 0.1 degrees, and larger values as the eye moves further. As long as the spot size remains below the pixel pitch on the CCD (ideally less than 50 microns), accuracy is as high as possible and latency is as low as possible. If the spot size is increased beyond the pixel pitch, accuracy will decrease and the calculation of the gaze direction (and therefore latency) will increase. Even in this case, the system latency is still lower than existing systems. The spot size can be reduced by inserting an autofocus lens in front of the CCD, as long as the diameter of the lens is large enough not to limit the image space NA (0.14 in this configuration).
[0116] According to the above Figure 5 Embodiments of the present invention contemplate spatially separated light sources and light detectors, but the distance between the light source and the light detector need not be large. Fig.11, depicts a top-down schematic diagram of a fourth embodiment of an optical system using a clearly spatially separated light source and light detector. This design includes: eyeglass frame 1101; first HOE 1102; second HOE 1103; light detector 1104; and light source 1105.
[0117] The IR source 1105 and the IR detector (CCD) 1104 are in different positions, specifically on either side of the longitudinal axis of the first 1102 and second 1103 HOEs. The first 1102 and second 1103 HOEs are multiplexed to replay multiple holograms, for example, reference Figure 8 A and 8B. However, unlike Figure 8 In embodiments A and 8B, not all of the hologram is directed toward the detector.
[0118] The propagation direction of the IR light is shown in the diagram. The eyeglass frame 1101 positions the user's eye 1106 so that the light from the IR source 1105 directed to the first HOE 1102 replays a hologram that focuses the IR light to the center of curvature of the cornea of the eye 1106. The back-reflected light then replays the hologram recorded on the HOE 1103 and focuses it in a different direction to the light detector 1104. As long as the system is properly designed with the correct focus, this system will allow for a smaller spot size than if the IR detector and IR source were right next to each other (similar to Figure 5 with a beam splitter as shown in ).
[0119] Returning to the overall perception discussed above, additional details may be considered. Optionally, the light source and light detector are integrated. For example, the integrated light source and light detector may include one of: a plurality of light emitting diodes (LEDs) modulated to function as sensors when not emitting light; and a separate light source and light detector within a single device.
[0120] In some embodiments, the HOE arrangement is configured to use each of the received at least one glint to replay a corresponding hologram, each replayed hologram being focused onto the photodetector.
[0121] The system may be configured to receive multiple glints from the eye. For example, the HOE arrangement may be recorded with multiple holograms and configured to replay the multiple holograms using light from the light source. Specifically, this may be performed such that each of the multiple holograms converges at a slightly different angle to generate multiple glints from the eye. In some embodiments, the light source includes multiple distinct light sources such that the HOE arrangement is configured to direct light from the multiple distinct light sources to the eye, thereby generating multiple glints from the eye.
[0122] The HOE arrangement may comprise a plurality of HOEs, each of the plurality of HOEs being recorded with a different hologram. For example, the HOE arrangement may comprise: a first HOE configured to reflect light from a light source towards an eye; and a second HOE configured to receive the at least one glint of light from the eye and preferably focus light from the received at least one glint onto a light detector. The plurality of HOEs may be formed on the same substrate and / or multiplexed.
[0123] In some embodiments, a waveguide is provided. Thus, the HOE arrangement may include: a first HOE configured to couple light from the light source into the waveguide; and a second HOE configured to couple the incoupled light from the light source out of the waveguide to reflect the outcoupled light to the eye and couple the at least one glint of light from the eye into the waveguide. The first HOE may further be configured to couple the at least one glint of light out of the waveguide and focus light from the received at least one glint onto the light detector.
[0124] Although the light source and the light detector may be co-located, in other embodiments, the light source and the light detector are spatially separated. The beam splitter may then be arranged to receive light from the light source and direct the light to the HOE, and to receive light from the HOE and direct the light to the light detector. Alternatively, the first HOE may be configured to reflect light from the light source toward the eye on a first side of the HOE, and the second HOE may be configured to receive the at least one glint of light from the eye and focus light from the received at least one glint onto the light detector on a second side of the HOE opposite the first side.
Claims
1. An optical system for eye tracking, comprising: a light source arranged to emit light; a light detector for detecting light and indicating position information of the light detected thereby; A holographic optical element HOE arrangement comprising a finite-finite conjugate mirror configured to converge light from the light source to the center of curvature of the cornea of an eye at a first focus of the holographic optical element HOE, receive at least one glint of the light from the eye and direct light from the received at least one glint to be focused at the light detector at a second focus of the holographic optical element HOE, the holographic optical element HOE arrangement being configured with a sufficiently large numerical aperture to cause ambient or stray light directed by the holographic optical element HOE arrangement to be out of focus at the light detector.
2. The optical system of claim 1, wherein the light detector is further configured to generate an electrical signal based on at least one detected scintillation, the optical system further comprising: A processor is configured to determine position information of the eye using the electrical signal and position information of the detected glint indicated thereby.
3. The optical system of claim 2, wherein the light detector detects an image, and the processor is configured to determine position information of the eye based on an identified position of the detected at least one glint in the image. The optical system of claim 1 , wherein the numerical aperture is greater than 0.
2.
5. The optical system of claim 1, wherein the light source is arranged to emit divergent light, and the holographic optical element (HOE) arrangement is further configured to converge the light from the light source onto the eye.
6. The optical system according to any one of claims 1 to 5, further comprising: A frame structure is configured to be worn by a user, and the light source, light detector and holographic optical element HOE are arranged and mounted on the frame structure.
7. An optical system according to any one of claims 1 to 6, wherein the holographic optical element (HOE) arrangement is configured to use each of the received at least one scintillation to replay a corresponding hologram, each replayed hologram being focused onto the photodetector.
8. An optical system according to any one of claims 1 to 7, wherein the holographic optical element (HOE) is arranged to be recorded with multiple holograms and is configured to replay the multiple holograms using the light from the light source so that each of the multiple holograms converges at a slightly different angle to generate multiple glints from the eye.
9. The optical system of any one of claims 1 to 8, wherein the light source comprises a plurality of distinct light sources, such that the holographic optical element (HOE) arrangement is configured to direct light from the plurality of distinct light sources toward the eye, thereby generating a plurality of glints from the eye.
10. The optical system according to any one of claims 1 to 9, wherein the holographic optical element (HOE) arrangement comprises a single holographic optical element (HOE).
11. The optical system according to any one of claims 1 to 10, wherein the holographic optical element (HOE) arrangement comprises a plurality of holographic optical elements (HOE), each holographic optical element (HOE) of the plurality of holographic optical elements (HOE) being recorded with a different hologram.
12. The optical system of claim 11, wherein the holographic optical element (HOE) arrangement comprises: a first holographic optical element HOE configured to reflect light from the light source towards the eye; and a second holographic optical element, HOE, configured to receive said at least one glint of light from said eye and to focus said light from said received at least one glint onto said light detector.
13. The optical system of claim 11, further comprising a waveguide, and wherein the holographic optical element (HOE) arrangement comprises: a first holographic optical element (HOE) configured to couple said light from said light source into said waveguide; as well as a second holographic optical element HOE configured to couple incoupled light from the light source out of the waveguide to reflect the outcoupled light to the eye and couple the at least one glint of the light from the eye into the waveguide, the first holographic optical element HOE further configured to couple the at least one glint of the light out of the waveguide and focus the light from the received at least one glint onto the light detector.
14. The optical system of any one of claims 1 to 5, wherein the light source and the light detector are co-located.
15. The optical system of claim 14, wherein the light source and the light detector are integrated.
16. The optical system of claim 15, wherein the integrated light source and light detector comprise one of: a plurality of light emitting diodes (LEDs) modulated to function as sensors when not emitting light; and a separate light source and light detector within a single device.
17. The optical system of any one of claims 1 to 5, wherein the light source and light detector are spatially separated.
18. The optical system of claim 17, further comprising: A beam splitter is arranged to receive light from the light source and direct the light to the holographic optical element HOE, and to receive light from the holographic optical element HOE and direct the light to the light detector.
19. The optical system of claim 18, wherein the light detector comprises one or more of: a position sensitive detector; an array of photosensors; and an optical flow sensor.
20. The optical system of claim 18, wherein the light source comprises one or more of: a vertical cavity surface emitting laser (VCSEL); a laser diode; and a broadband LED having one or more filters to reduce the spectral bandwidth of the emitted light.
21. The optical system of claim 14, wherein there is no lens between the HOE arrangement and the light detector, or wherein an autofocus lens is disposed between the HOE arrangement and the light detector.
22. An augmented reality (AR) or virtual reality (VR) system, comprising: An optical display system for displaying AR or VR images; An optical eye tracking system comprising an optical system according to any preceding claim and configured to provide an eye measurement indication; as well as A processor is configured to control the optical display system based on the eye measurement indication provided by the optical eye tracking system.
