Eye movement tracking system and method of using same

Abstract

An eye movement tracking system for tracking an eye, the eye movement tracking system comprising a holographic optical element (HOE). The HOE has one or more specular reflection regions and one or more diffuse reflection regions provided as a predetermined pattern. The eye movement tracking system also includes a camera and a controller. The camera is configured to capture an image of the eye reflected by the one or more specular reflective regions of the holographic optical element such that a pattern defined by the one or more diffusely reflective regions of the holographic optical element is visible on the eye in the captured image and the one or more diffusely reflective regions. The controller is configured to: process a captured image of an eye in order to identify a pattern of one or more diffusely reflective regions in the image; and comparing i) the predetermined pattern with ii) the identified pattern in the captured image in order to determine the distance from the holographic optical element to the eye.

Description

Technical Field

[0001] This disclosure generally relates to the field of eye tracking. In particular, this disclosure relates to methods and systems for determining the distance between an eye and a holographic optical element in an eye tracking system. Background Technology

[0002] In eye-tracking applications, digital images of the user's eyes are acquired and analyzed to estimate the user's gaze direction. The gaze direction estimation can be based on computer-based image analysis of the features of the imaged eye. A known example of eye-tracking involves using infrared light and an image sensor. The infrared light is directed towards the user's eye, and the reflection of the light is captured by the image sensor.

[0003] Portable or wearable eye-tracking devices have been previously described. One such eye-tracking system is described in U.S. Patent No. 9,041,787 and PCT Patent Publication No. WO 2019 / 158709 (the entire contents of both patents are incorporated herein by reference). A wearable eye-tracking device is described that uses an illuminator and a camera to determine the direction of gaze. Summary of the Invention

[0004] According to a first aspect of this disclosure, an eye-tracking system for tracking eyes is provided, the eye-tracking system comprising: A holographic optical element comprising one or more specular reflection regions and one or more diffuse reflection regions, wherein the one or more specular reflection regions and one or more diffuse reflection regions of the holographic optical element are provided as a predetermined pattern; A camera configured to capture an image reflected by the eye through one or more specular reflection regions of the holographic optical element, such that a pattern defined by one or more diffuse reflection regions of the holographic optical element is visible on the eye in the captured image; and The controller is configured to: The captured image from the eye is processed to identify the pattern of the one or more diffuse regions in the image; and Compare: i) the predetermined pattern with ii) the identified pattern in the captured image in order to determine the distance from the holographic optical element to the eye.

[0005] The controller can be configured to determine multiple distances from the holographic optical element to different regions of the eye.

[0006] The controller can be configured to determine corneal topography based on multiple determined distances.

[0007] The predetermined pattern may include a two-dimensional pattern having a predetermined spacing between features in the pattern in two vertical dimensions.

[0008] The controller can be configured to: Compare: i) the predetermined spacing between features in the predetermined pattern and ii) the identified spacing between corresponding features in the identified pattern in the captured image, so that: Determine the distance from the holographic optical element to the eye region on which the corresponding feature in the identified pattern in the captured image is located.

[0009] The controller can be configured to: Compare: i) the predetermined pattern with ii) the identified pattern in the captured image, so as to: Detect any distortion in the identified pattern; and The distance from the holographic optical element to the eye is determined based on the detected distortion.

[0010] The camera can be configured to capture multiple images of the eye reflected by one or more specular reflection regions of the holographic optical element at corresponding multiple different times, such that the pattern defined by one or more diffuse reflection regions of the holographic optical element is visible on the eye in each captured image.

[0011] The controller can be configured to work for each of the multiple captured images: The captured image of the eye is processed to identify patterns in the image of the one or more diffuse regions; and Compare: i) the predetermined pattern with ii) the identified pattern in the captured image, in order to extract a portion of the captured image in which the identified pattern is present; The controller can be further configured to: The extracted portions of the captured images are combined to provide a composite image of the eye, including multiple identified patterns; and Compare each of the plurality of identified patterns in the combined image with the predetermined pattern to determine i) the distance between the area of ​​the eye represented by the identified pattern and ii) the distance between the holographic optical element.

[0012] The predetermined pattern may include a substantially one-dimensional pattern.

[0013] The controller can be further configured to: The eye-tracking model is updated based on the determined distance from the holographic optical element to the eye; and Use the updated eye-tracking model to perform eye tracking based on subsequent captured images of that eye.

[0014] During subsequent eye-tracking operations, the illuminator can be configured to: Background illumination for the eye is provided by illuminating one or more diffuse reflection areas of the holographic optical element; and One or more “flashes” are provided on the eye by illuminating one or more specular reflective areas of the holographic optical element.

[0015] The holographic optical element may include multiple specular reflective areas, such that the illuminator is configured to provide multiple “flashes” on the eye by illuminating the multiple specular reflective areas of the holographic optical element.

[0016] The eye-tracking system may include a head-mounted device.

[0017] The head-mounted device may include a pair of glasses having a lens area and a pair of arms, wherein: The camera can be mounted on one of the arms; and / or The holographic optical element can be located on at least one lens area.

[0018] The eye-tracking system may further include an illuminator configured to illuminate the eye via one or more specular reflection regions and one or more diffuse reflection regions of the holographic optics.

[0019] A computer-implemented method for operating an eye-tracking system is also disclosed, wherein the eye-tracking system includes: A holographic optical element comprising one or more specular reflection regions and one or more diffuse reflection regions, wherein the one or more specular reflection regions and one or more diffuse reflection regions of the holographic optical element are provided as a predetermined pattern; The method includes: Capture an image of the eye reflected by one or more specular reflection regions of the holographic optical element, such that a pattern defined by one or more diffuse reflection regions of the holographic optical element is visible in the captured image of the eye; and Process the captured image of the eye to identify the pattern of the one or more diffuse regions in the image; and Compare: i) the predetermined pattern with ii) the identified pattern in the captured image in order to determine the distance from the holographic optical element to the eye.

[0020] A computer program may be provided that, when executed on a computer, causes the computer to configure any device (including the controllers, apparatuses, or systems disclosed herein) or to perform any of the methods disclosed herein. The computer program may be a software implementation, and the computer may be considered any suitable hardware, including, as non-limiting examples, a digital signal processor, a microcontroller, and an implementation in read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM). The software may be an assembler.

[0021] A computer program may be provided on a computer-readable medium, which may be a physical computer-readable medium such as a disk or storage device, or may be embodied as a transient signal. Such a transient signal may be downloaded from a network, including the Internet. One or more non-transitory computer-readable storage media may be provided storing computer-executable instructions that, when executed by a computing system, cause the computing system to perform any of the methods disclosed herein. Attached Figure Description

[0022] One or more embodiments will now be described by way of example only with reference to the accompanying drawings, in which: Figure 1 A simplified view of a head-mounted eye-tracking system is shown; Figure 2 This demonstrates the use of eye-tracking systems (such as...) Figure 1 A simplified example of an image of a pair of eyes captured by a system; Figure 3 An example head-mounted eye-tracking system according to an embodiment of the present disclosure is shown; Figure 4a An example setup is shown that can be used to provide a holographic optical element (HOE) having a predetermined pattern of specular and diffuse reflection regions; Figure 4b An example of a patterned object is shown, which is used for Figure 4a Settings to create patterned HOEs; Figure 5 An example of an eye image is shown, which has been captured by a camera via a HOE with a concentric ring pattern having specular and diffuse reflection areas; Figure 6 Another example of an eye image is shown, which has been captured by a camera via a HOE (Homo Echo) pattern having concentric ring patterns of specular and diffuse reflection areas; and Figure 7 A flowchart is shown illustrating a method for operating an eye-tracking system according to this disclosure. Detailed Implementation

[0023] Figure 1 A simplified view of a head-mounted eye-tracking system 100 (which may also be referred to as a viewpoint tracking system) is shown, which takes the form of a virtual or augmented reality (VR or AR) device, VR or AR glasses, or anything related to it, such as an extended reality (XR) or mixed reality (MR) head-mounted device. System 100 includes an image sensor 120 (e.g., a camera) for capturing images of a user's eyes. The system may optionally include one or more illuminators 110-119 (also referred to herein as light sources) for illuminating the user's eyes; these illuminators may be, for example, light-emitting diodes (LEDs) emitting light in the infrared or near-infrared bands, and may be physically arranged in various configurations. Image sensor 120 may be, for example, any type of image sensor, such as a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor. The image sensor may be composed of an integrated circuit containing an array of pixel sensors, each pixel containing a photodetector and an active amplifier. The image sensor may be able to convert light into digital signals. In one or more examples, the image sensor may be an infrared image sensor or IR image sensor, an RGB sensor, an RGBW sensor, or an RGB or RGBW sensor with an IR filter.

[0024] The eye-tracking system 100 may include a circuitry or one or more controllers 125 for receiving and processing images captured by the image sensor 120. This circuitry or controllers may include, for example, a receiver 126 and a processing circuitry 127. The circuitry 125 may be connected to the image sensor 120 and optionally one or more illuminators 110-119, for example via a wired or wireless connection, and may be located in the same location as or at a distance from the image sensor 120 and the one or more illuminators 110-119 (e.g., in different devices). In another example, the circuitry 125 may be disposed in one or more stacked layers beneath the photosensitive surface of the light sensor 120.

[0025] The eye-tracking system 100 may include a display (not shown) for presenting information and / or visual stimuli to the user. The display may include a VR display that presents images and substantially obscures the user's real-world view, or an AR display that presents images perceived as superimposed on the user's real-world view.

[0026] In such a system 100, the image sensor 120 for one eye is typically positioned away from the user's line of sight so as not to obstruct the display for that eye. This configuration can be achieved, for example, by means of a so-called thermal mirror, which reflects a portion of the light while allowing the rest of the light to pass through; for example, infrared light is reflected while visible light is allowed to pass through.

[0027] While in the example above, the image of the user's eyes is captured by a head-mounted image sensor 120, in other examples, the image may be captured by a non-head-mounted image sensor. Such a non-head-mounted system may be referred to as a remote system.

[0028] Figure 2 This demonstrates the use of eye-tracking systems (such as...) Figure 1 This is a simplified example of an image 229 captured by a system (the system itself). Image 229 can be considered to include a right-eye image 228 of a person's right eye and a left-eye image 234 of a person's left eye. In this example, the right-eye image 228 and the left-eye image 234 are two parts of a larger image of a person's two eyes. In other examples, a single image sensor can be used to acquire the right-eye image 228 and the left-eye image 234. In yet another example, multiple image sensors can be used to acquire images capturing both eyes.

[0029] The system can employ image processing (such as digital image processing) to extract features from images. For example, the system can identify the location of the pupil 230 in one or more images captured by an image sensor. The system can use a pupil detection process to determine the location of the pupil 230. The system can also identify corneal reflections (also called flashes) 232 adjacent to the pupil 230. The system can estimate the corneal center and / or distance to the user's eye based on the corneal reflections 232. For example, the system can match each of the individual corneal reflections 232 of each eye with a corresponding illuminator and determine the corneal center and / or distance to the user's eye for each eye based on the match. In essence, the eye-tracking system can define the optical axis of the user's eye as a vector passing through the center of the pupil 230 and the center of the cornea. The gaze direction corresponds to an axis (visual axis) passing through the center of the cornea from the fovea of ​​the eye. The angle between the optical axis and the gaze direction is... Central concave offset This varies from user to user and is within a few degrees. Eye-tracking systems can perform a calibration procedure to determine foveal offset by instructing the user to gaze in a series of predetermined directions (e.g., via on-screen instructions). The determination of the optical axis described above is known to those skilled in the art and is commonly referred to as the central pupillary corneal reflex (PCCR). PCCR will not be discussed further in detail here.

[0030] Figure 3An example head-mounted eye-tracking system 335 according to an embodiment of this disclosure is shown. As will be discussed below, this head-mounted eye-tracking system 335 is a holographic eye-tracking system.

[0031] In this example, the eye-tracking system 335 includes a pair of glasses 336 having two lens areas 337 (one for each eye 342) and a pair of arms 340. Additionally, an illuminator 339 is mounted on each of the arms 340 so that each eye is individually illuminated. Also in this example, a camera 338 is mounted on each of the corresponding arms 340 so that an image of each eye 342 can be captured individually. Figure 3 As shown, each illuminator 339 and camera 338 is positioned close to each other on its respective arm 340. Although Figure 3 Not shown, but the eye-tracking system 335 also includes a controller for processing the captured images. The controller may be located on the glasses 336, or it may be positioned remotely from the glasses 336. In either case, it is in electronic communication with at least the camera 338.

[0032] Holographic optical elements (HOEs) 341 are located on each of the corresponding lens regions 337. HOEs 341 can be implemented by embedding a near-infrared (NIR) holographic film on the lens or by casting an NIR holographic film into the lens. Such NIR holographic films are invisible / transparent to visible light, and therefore do not obstruct the user's field of vision through the lens regions 337. However, as will be discussed below, because NIR holographic films are reflective of NIR light, they can be used for eye-tracking purposes.

[0033] In one example, at least a portion of the HOE 341 is provided as a specular reflection region. Furthermore, as is known in the art, they can be configured to function as one or more virtual mirrors oriented at an angle that is not necessarily coplanar with the lens region 337 on which the HOE 341 is embedded or cast. Such virtual mirrors are known in the art, for example, as described in the following literature: Tomoya Nakamura, Shinji Kimura, Kazuhiko Takahashi, Yuji Aburakawa, Shunsuke Takahashi, Shunsuke Igarashi, Shiho Torashima, and Masahiro Yamaguchi, “Off-axis virtual-image display and camera by holographic mirror and blur compensation,” Opt. Expression, Vol. 26, pp. 24864-24880 (2018); and Tomoya Nakamura, Shinji Kimura, Kazuhiko Takahashi, Yuji Aburakawa, Shunsuke Takahashi, Shunsuke Igarashi, et al., “Holographic Pepper's Ghost: Upright Virtual-Image Screen Realized by Holographic Mirror,” Holographic Materials and Applications. [and Applications], IntechOpen, 2019, Source: http: / / dx.doi.org / 10.5772 / intechopen.85600.

[0034] Eye tracking can be performed by Figure 3 The eye-tracking system 335 is performed via an illuminator 339 that emits light such that it is reflected from a specular reflection area in the HOE 341 toward the eye 342. As is known in the art, and as briefly noted above, this light can produce flashes on the eye 342. The camera 338 can then capture images of the eye 342 reflected by the specular reflection area of ​​the HOE 341, including those flashes, making eye tracking possible. In this way, illumination is provided via the same virtual mirror used by the camera 338 to capture the images. Typically, the illuminator 339 can be placed close to the camera 338 (e.g., within a few mm).

[0035] Alternatively, the illuminators used to generate the flash can be positioned differently. For example, the illuminators can be arranged around the HOE 341 so that they illuminate the eye directly. However, a potential drawback of placing the illuminators around the HOE is that only a lower percentage of light illuminates the eye, sometimes as low as 10%. Generally, it can produce a good flash at high focus angles, but it may be difficult to produce a flash when the eye is looking straight ahead. In cases of extremely low exit pupil distance and / or narrow eyelid opening, there may be too little or no flash.

[0036] For illuminators 339 that reflect light via virtual mirrors in HOE 341 (e.g.) Figure 3 The disadvantage of the illuminator shown is that if the reflectivity of the entire holographic film is insufficient, and therefore not all of the HOE 341 reflects light toward the eye 342, not all illumination can be utilized. Furthermore, if all the light on the eye 342 (including the light that produces the flash reflection) originates from the specular reflection area of ​​the HOE 341 (which can also be called the reflector portion), the contrast between the flash and the background illumination can be very high. This can lead to strong saturation of the flash in the image, making the flash larger and potentially blurring eye features such as the pupil or iris. Moreover, since the illumination is provided to the eye 342 via the path between the illuminator 339 and the HOE 341, the illumination is highly susceptible to obstruction by eyelids and eyelashes. Therefore, there is a possibility that the illumination of the pupil and iris may be poor due to light obstruction.

[0037] In the example disclosed herein, HOE 341 includes one or more specular reflection areas and one or more diffuse reflection areas. As discussed above, the specular reflection areas can provide the function of a virtual mirror, allowing camera 338 to capture an image of eye 341 via HOE 341. Camera 338 cannot obtain an accurate image of eye 342 via the diffuse reflection areas, which reflect light in a manner similar to that of white paper. However, Figure 3 The central illuminator 339 (which illuminates the entire HOE 341) can provide background illumination to the eye 342 at an intensity lower than the intensity of the light provided to the eye 342 via the specular reflection area to produce a flash.

[0038] By utilizing the HOE 341 region (i.e., the diffuse region), which does not reflect the image of the area where the eye may be located, different reflective areas can be created that do not function like normal mirrors. Ideally, these diffuse regions have a Lambertian reflectance distribution, making the illumination uniform. These diffuse regions help provide uniform illumination to the eye and also help illuminate previously unilluminated areas of the eye, such as areas blocked by eyelashes or eyelids. Since the direct path from the illuminator through the holographic mirror can be blocked, using large-area eye illumination allows light to illuminate every part of the eye from multiple directions, reducing the likelihood that a particular area of ​​the eye will be underiluted. This is crucial for achieving robust eye tracking in large crowds, especially for users with narrow eyelid openings or in situations where the layout of the camera and illuminator components is suboptimal.

[0039] As will now be discussed, the HOE of this disclosure has one or more specular reflection regions and one or more diffuse reflection regions. Furthermore, one or more specular reflection regions and one or more diffuse reflection regions of the holographic optical element are provided as a predetermined pattern. The use of this predetermined pattern can advantageously be used to determine the distance from the HOE to the eye, as will be discussed in detail below. Furthermore, the use of this predetermined pattern can be used to determine corresponding distances to different regions of the eye, enabling the generation of a corneal topography map. In this case, it should be understood that it is not necessary to calculate the absolute distance from the HOE to each region of the eye; the relative distances to different regions of the eye will be sufficient. The calculation of this determined distance / corneal map can be used to improve the accuracy of subsequent eye-tracking operations. For example, an eye-tracking model for eye-tracking as known in the art can be updated based on the determined distance / corneal map. The updated eye-tracking model can then be used to perform eye-tracking based on subsequently captured images of the eye, with higher accuracy than the case of an unupdated model.

[0040] In the following main examples, the predetermined patterns for the specular and diffuse reflection regions are concentric ring structures. However, we will also describe alternative predetermined patterns that can be used.

[0041] Figure 4a An example setup is shown that can be used to provide a HOE with a predetermined pattern of specular and diffuse reflection areas; Figure 4b An example of a patterned object is shown, which is used for Figure 4a The settings are used to create patterned HOEs.

[0042] Figure 4a This diagram illustrates a setup where a spherical beam interferes with two collimated beams. One of the collimated beams is adorned with a patterned object, such as... Figure 4b Object 444. For example... Figure 4b As shown, the patterned object 444 defines a concentric ring pattern by having concentric light-blocking portions, and the gaps between these concentric light-blocking portions define gaps through which light can pass. Figure 4a Example locations where the patterned object 444 can be located in the setup are identified by reference numerals 444a, 444b, and 444c. The ring defined by the patterned object 444 will be recorded simultaneously as an optical element that functions as both a mirror and a lens. When recording a hologram, the concentric ring pattern can be recorded by having a physical pattern in the reference beam or the object beam. In another interference beam, a plane mirror or a mirror with spherical wave refraction can be recorded.

[0043] In this way, by Figure 4a The HOE settings created have a pattern of specular and diffuse reflection regions, which in this example are concentric rings. The pattern is generated by positioning a physical shape (such as patterned object 444) within one of the interference beams, preferably within a collimated beam. Figure 4a In this setup, a spherical beam with a refractive lens effect is recorded in the hologram, which can be beneficial for telecentric imaging. There are two collimated beams in this setup, used to form two holograms side-by-side. This can provide a stereoscopic view of the eye. If the physical pattern is located in one of the beams, the concentric ring pattern may exist on only one hologram.

[0044] It should be understood that Figure 4a and Figure 4b The hologram creation technique is not limiting. Any other known technique can be used, such as using liquid crystals, replacing phase modulation with amplitude modulation, etc.

[0045] Return to Figure 3 The use of an eye-tracking system 335 with a patterned HOE 341 will now be described. That is, the HOE 341 has one or more specular reflection areas and one or more diffuse reflection areas provided as a predetermined pattern. The pattern is described as predetermined because when the controller begins processing an image of the eye 342 captured by the HOE 341, the controller already possesses details of the pattern expected to be identified in the image. A camera 338 captures an image of the eye 342 reflected by the patterned HOE 341, such that the pattern defined by the one or more diffuse reflection areas of the holographic optics is visible in the captured image of the eye.

[0046] The controller processes the captured image of eye 342 to identify patterns in one or more diffuse regions within the image. The controller then compares: i) a predetermined pattern (e.g., details of which can be retrieved by the controller from computer memory) with ii) the identified pattern in the captured image. Based on the result of this comparison, the controller determines the distance from HOE 341 to eye 342. Various example implementations for determining this distance are provided below.

[0047] A predetermined pattern is any pattern having an arrangement of specular and diffuse reflective regions, suitable for determining the distance from HOE 341 to eye 342 when applied to a three-dimensional surface of the eye, particularly the cornea. It will be understood from the following specific examples and from the common sense of those skilled in the art that a wide variety of predetermined patterns can be used, and their suitability can be readily verified directly and actively by testing or procedure without excessive experimentation. Such a predetermined pattern can be considered a pattern with a known structure.

[0048] In some examples, the predetermined pattern is a two-dimensional pattern, for example, by using... Figure 4b The pattern is created by creating concentric rings from a patterned object. A two-dimensional pattern can have predetermined spacing between its features. This predetermined spacing can be in one or both of the two vertical dimensions of the two-dimensional pattern. For the concentric ring example, the spacing between features can be the spacing between a pair of rings. In alternative examples, the pattern can include multiple sets of parallel lines (which can be straight or curved, horizontal, vertical, or at any angle). The predetermined pattern can include one or more of the following: rings (which can be concentric or not), arcs (which can be concentric or not), and crosses (where the intersections of the crosses are features with predetermined spacing between them). In a further example, the predetermined pattern can be a dot pattern. When deciding which predetermined pattern to use for a given application, there may be a trade-off between how easily the pattern can be found in the captured image and how well the distance to the eye can be determined (especially when determining multiple distances to different areas of the eye to map the corneal morphology, as discussed below). For example, a pattern including a cross can be easily found, but when covering a large area of ​​the cornea, this pattern provides far less morphological information than a dot pattern. However, such dot patterns may be more difficult to find in the captured image and more difficult to match with corresponding parts of the predetermined pattern. Therefore, the most suitable predetermined pattern may vary depending on the application scenario and what the calculated distance between the HOE and the eye is used for.

[0049] In an example of this disclosure, the controller of the eye-tracking system 335 can compare: i) a predetermined spacing between features in a predetermined pattern with ii) an identified spacing between corresponding features in an identified pattern in the captured image. Based on the comparison result, the controller can then determine the distance from the holographic optical element to the eye region on which the corresponding feature in the identified pattern in the captured image is located. In the concentric ring example, the spacing between adjacent rings of the same type (i.e., rings reflected by specular reflection areas or rings reflected by diffuse reflection areas) will represent the distance between the HOE 341 and the eye. That is, the greater the distance between the HOE 341 and the eye 342, the greater the spacing between adjacent rings of the same type in the captured image. It is worth noting that, as from... Figure 3 As understood in this example, camera 338 is at a fixed distance relative to HOE 341. Therefore, there is no possibility that changes in the distance between camera 338 and HOE 341 would affect the spacing between adjacent rings of the same type in the captured image.

[0050] Figure 5 An example of an eye image is shown, which was captured by a camera via a HOE with a concentric ring pattern having specular and diffuse reflection areas.

[0051] It should be understood that the controller in Figure 4 can determine the absolute distance to one or more regions on the eye where the identified pattern exists. For example, by applying an algorithm to the spacing between the rings (which can also be considered the thickness of the rings), the spacing is converted into an absolute distance (e.g., in millimeters) between the HOE and the eye. For example, the parameters of such an algorithm can be determined during a calibration operation.

[0052] In examples where multiple distances to different regions of the eye are calculated, these distances can be used to construct a corneal topography of the eye. As will be discussed below, this construction can be particularly useful for improving the accuracy of subsequent eye-tracking operations.

[0053] Alternatively or additionally, the controller of Figure 4 can determine multiple relative distances to different regions of the eye on which the identified pattern exists. For example, the relative variation in the thickness of the rings in the concentric pattern would represent the relative variation in the distance to different regions of the eye on which the identified pattern exists (assuming, of course, that the thickness of the concentric rings in the predetermined pattern in the HOE is uniform).

[0054] Return to Figure 3In some examples, the controller can compare: i) a predetermined pattern with ii) an identified pattern in the captured image to detect any distortions in the identified pattern. The controller can then determine the distance from HOE 341 to eye 342 based on the detected distortions. For example, the controller can determine relative distances from HOE 341 to eye 342 for multiple regions of eye 342 based on the detected distortions. This can be implemented, for example, by applying any suitable mathematical operations to the detected distortions to convert them into a 3D corneal topography profile.

[0055] In a specific example of this disclosure, the predetermined pattern is one that, when the camera captures an image of the eye via the specular reflection area of ​​the HOE 341, yields a captured image suitable for determining the corneal morphology of the eye.

[0056] The example of the predetermined pattern described above is a two-dimensional pattern. We will now describe an example of using a one-dimensional or two-dimensional pattern to determine the distance between a region of the HOE 341 and the eye 342. Such an example can advantageously utilize the HOE 341 to perform laser line triangulation to measure the 3D surface of the eye 342, as is known in the machine vision industry. In particular, we will describe below an example in which multiple images can be captured and processed over time to provide the controller with sufficient information to determine the distance between the HOE 341 and the eye 342.

[0057] In these examples, camera 339 captures multiple images of eye 342 reflected by one or more specular reflection areas of the holographic optics at corresponding different times. That is, camera 338 can capture a series of images of eye 342 over time. The controller can then perform the following operations for each of the multiple captured images: process the captured image of the eye to identify a pattern in the image; and compare: i) a predetermined pattern with ii) the identified pattern in the captured image to extract a portion of the captured image in which the identified pattern is present. For example, this portion of the captured image can be a subset of the captured image defined by a bounding box of predefined size including the identified pattern. The controller can then combine the extracted portions of the captured images to provide a composite image of the eye including multiple identified patterns. These extracted portions can be stitched together in any manner known in the art. The controller can then compare each of the multiple identified patterns in the composite image with the predetermined pattern to determine i) the distance between the region of the eye represented by the identified pattern and ii) the distance between the holographic optics. In one example, the predetermined pattern is a one-dimensional line to be imaged on a two-dimensional sensor. The offset on the two-dimensional sensor represents the distance measurement between the HOE and the eye, which can be calculated by triangulating the emission line and the detection line using a known baseline distance between the projector (illuminator) and the detector (camera).

[0058] In this example, the predetermined pattern can comprise a substantially one-dimensional pattern. That is, the predetermined pattern can be a very thin straight line, and therefore its width is negligible compared to its length. If the user moves their eye during image capture so that the identified pattern appears in different regions of the user's eye in different images, this predetermined pattern can provide sufficient information for the corneal topography mapping to be performed. In this way, the straight line (one-dimensional pattern) can be considered as a scan of the eye over time (i.e., a temporal scan), allowing distances to different regions of the eye to be determined over time. These distances can then be combined to generate a three-dimensional topography map of the eye. Of course, it should be understood that this process can also be performed using a two-dimensional predetermined pattern.

[0059] Return to Figure 5 In addition to the shape of the diffuse reflection area of ​​the HOE (which is reflected as an image on the cornea) (concentric rings) visible in the image, three flashes 550 are also visible on each side of the pupil. This is because the illuminators are located in the same position as the camera, and therefore they illuminate the eye via the HOE.

[0060] Figure 6Another example of an eye image is shown, which was captured by a camera via a HOE (House of Eyes) with a concentric ring pattern having specular and diffuse reflection areas. In this example, the eye is directly illuminated (not via the HOE), and therefore there is no flash in the image.

[0061] Return to Figure 5 As discussed above, the specular reflection region of a HOE has been divided into several discrete regions. In some examples, these different regions can have different properties. For example, they can be specularly reflective for different wavelengths of light and / or can have different polarizations. Additionally, they can be located side-by-side in the HOE, and / or they can overlap each other in different films / layers of the HOE. Such examples can be considered as multiple composite holograms because they can be used to obtain multiple images of the same eye from a single HOE and a single camera.

[0062] like Figure 5 As shown, discrete specular reflective regions can have blank spaces between them. In this way, as described above, diffuse regions (between the specular reflective regions) each reflect an "image" on the cornea. By having several regions, discrete shapes exist in the captured image of the eye, and these discrete shapes can be used to extract information about the eye. Such information can include: distances to the corneal surface (as detailed above); the center of the eyeball; information that can help with flash detection and matching; and information that can be used to measure the corneal surface topography.

[0063] The flash 550 can be used to locate the corneal center via known PCCR eye-tracking, while the concentric rings can be used to measure corneal morphology through their distortion and relative placement (as described above). From Figure 5 It can be noted that the brightness of the concentric rings is much lower than that of the flash. This is advantageous for flash detection because the patterned light is scattered across a much larger surface when reflected by the diffuse region of the HOE.

[0064] Continue to Figure 6 The image shows a comparison with Figure 5 The cornea reflects the concentric ring pattern in the same way. The center of the ring can be used for PCCR eye tracking, and the ring can be used to map the corneal topography. Advantageously, the center of the ring pattern can be found even if only a portion of the ring pattern is visible. A standard circular Hough transform can be used. The advantage of using a concentric ring pattern, unlike flash patterns, is that only one center exists instead of multiple flash centers. It also eliminates the need to track individual flashes. The corneal topography can be mapped during routine individual calibration. The corneal topography can later be used for more accurate corneal centering and pupil mapping as part of subsequent eye tracking operations. Therefore, it is advantageous to account for variability between the corneas of different users.

[0065] Eye-tracking operations can utilize eye-tracking models that typically assume the cornea to be spherical. However, the real cornea is elliptical, with a radius of curvature that is larger at the periphery and smaller at the center. For example, a typical radius of curvature at the center of the cornea can be 7.8 mm, while at the periphery it can be 10 to 11 mm. Errors in the radius of curvature can lead to errors in the calculation of the distance from the camera to the eye. For more details, see “General Theory of Remote Gaze Estimation Using the Pupil Center and Corneal Reflections”, Guestring and Eisenman, IEEE Transactions on Biomedical Engineering, Vol. 53, No. 6, June 2006, pp. 1124–1133.

[0066] Any controller described herein can use the determined distance from the holographic optics to the eye (or multiple determined distances, which can be represented as a corneal topography) as part of subsequent eye-tracking operations. Advantageously, using the determined distance can improve the accuracy, reliability, and / or robustness of subsequent eye-tracking operations. In one example, the controller can update the eye-tracking model based on the determined distance from the holographic optics to the eye and / or the determined corneal topography. In this way, the eye-tracking model can represent the determined corneal topography, rather than assuming the cornea has a spherical profile. The controller can then use the updated eye-tracking model to perform eye tracking based on subsequently captured images of the eye.

[0067] For any hologram-based eye-tracking system described herein, the illuminator can be provided as a vertical cavity surface-emitting laser (VCSEL). Due to the bandwidth limitations of holograms, VCSELs are well-suited for these applications. In some examples, one or two VCSELs can be used to provide two or four flashes, depending on whether a single hologram or a double hologram (i.e., a HOE including one or two virtual mirrors) is used. The flashes may disappear at large gaze angles. For LED-based eye-tracking systems, eight to ten LEDs can be integrated into the head-mounted device. To reduce costs, using only one or two VCSELs per eye may be beneficial. This can be achieved by using concentric ring patterns covering the entire eye (e.g., ...). Figure 5 and Figure 6As shown, the pattern can remain in place even at large angles. If concentric rings are used, the center can serve as a reference point for the viewing angle. Furthermore, concentric rings can be used to compose corneal topography using the principle of keratometry.

[0068] Figure 7 A computer-implemented method for operating an eye-tracking system according to this disclosure is schematically illustrated. As discussed in detail above, the eye-tracking system includes at least a holographic optical element comprising one or more specular reflection regions and one or more diffuse reflection regions. The one or more specular reflection regions and one or more diffuse reflection regions of the holographic optical element are provided as a predetermined pattern.

[0069] At step 770, the method includes capturing an image of the eye reflected by one or more specular reflection regions of a holographic optical element, such that a pattern defined by one or more diffuse reflection regions of the holographic optical element is visible in the captured image of the eye.

[0070] At step 771, the method includes processing the captured image of the eye to identify a pattern in one or more diffuse regions of the image. Then, at step 772, the method includes comparing: i) the predetermined pattern with ii) the identified pattern in the captured image to determine the distance from the holographic optics to the eye. Various specific embodiments of how this distance can be determined have been described above.

[0071] Regarding the above description, it should be understood that the various examples disclosed herein include illuminators that can provide background illumination to the eye by illuminating one or more diffuse regions of the HOE, and can also provide one or more “flashes” to the eye by illuminating one or more specular regions of the HOE. Furthermore, the HOE may include multiple specular regions, such that the illuminator provides multiple “flashes” to the eye by illuminating multiple specular regions of the HOE. As will be understood from this specification, multiple flashes can be achieved, for example, by using multiple different specular regions (which define virtual mirrors at different angles), by using multiple different specular regions for different wavelengths, or by using multiple different polarizations. Additionally, the illuminator can illuminate the eye via one or more specular regions and one or more diffuse regions of the holographic optics. The diffuse region of the holographic optics is not necessarily within the field of view of the camera of the eye-tracking system.

Claims

1. An eye-tracking system for tracking eyes, the eye-tracking system comprising: A holographic optical element comprising one or more specular reflection regions and one or more diffuse reflection regions, wherein the one or more specular reflection regions and one or more diffuse reflection regions of the holographic optical element are provided as a predetermined pattern; A camera, configured to capture an image of the eye reflected by one or more specular reflection regions of the holographic optical element, such that a pattern defined by one or more diffuse reflection regions of the holographic optical element is visible on the eye in the captured image; and The controller is configured to: The captured image from the eye is processed to identify patterns in the one or more diffuse regions within the image; and Compare i) the predetermined pattern with ii) the identified pattern in the captured image to determine the distance from the holographic optical element to the eye.

2. The eye-tracking system as described in claim 1, wherein, The controller is configured to determine multiple distances from the holographic optical element to different regions of the eye.

3. The eye-tracking system as described in claim 2, wherein, The controller is configured to determine a corneal topography based on a plurality of determined distances.

4. The eye-tracking system as claimed in any of the preceding claims, wherein, The predetermined pattern includes a two-dimensional pattern having a predetermined spacing between features in the pattern in two vertical dimensions.

5. The eye-tracking system as described in claim 4, wherein, The controller is configured to: Compare: i) the predetermined spacing between features in the predetermined pattern and ii) the identified spacing between corresponding features in the identified pattern in the captured image, so that: Determine the distance from the holographic optical element to the eye region on which the corresponding feature in the identified pattern in the captured image is located.

6. The eye-tracking system as claimed in any of the preceding claims, wherein, The controller is configured to: Compare: i) the predetermined pattern with ii) the identified pattern in the captured image, so as to: Detect any distortions in the identified pattern; as well as The distance from the holographic optical element to the eye is determined based on the detected distortion.

7. The eye-tracking system as claimed in any of the preceding claims, wherein, The camera is configured to capture multiple images of the eye reflected by one or more specular reflection regions of the holographic optical element at corresponding multiple different times, such that a pattern defined by one or more diffuse reflection regions of the holographic optical element is visible on the eyepiece in each captured image; and The controller is configured to: For each of the plurality of captured images: The captured image from the eye is processed to identify patterns in the image of the one or more diffuse regions; and Compare: i) the predetermined pattern with ii) the identified pattern in the captured image, in order to extract a portion of the captured image in which the identified pattern is present; Combine the extracted portions of the captured images to provide a combined image of the eye including multiple identified patterns; and Compare each of the plurality of identified patterns in the combined image with the predetermined pattern to determine i) the distance between the region of the eye represented by the identified pattern and ii) the distance between the holographic optical element.

8. The eye-tracking system as described in claim 7, wherein, The predetermined pattern includes essentially a one-dimensional pattern.

9. The eye-tracking system as claimed in any of the preceding claims, wherein, The controller is further configured to: The eye-tracking model is updated based on the determined distance from the holographic optical element to the eye; and The updated eye-tracking model is used to perform eye tracking based on subsequently captured images of the eye.

10. The eye-tracking system of claim 9, wherein, During subsequent eye-tracking operations, the illuminator is configured to: Background illumination of the eye is provided by illuminating one or more diffuse reflection areas of the holographic optical element; and One or more "flashes" are provided on the eye by illuminating one or more specular reflective areas of the holographic optical element.

11. The eye-tracking system of claim 10, wherein, The holographic optical element includes multiple specular reflective areas, such that the illuminator is configured to provide multiple "flashes" on the eye by illuminating the multiple specular reflective areas of the holographic optical element.

12. The eye-tracking system as claimed in any of the preceding claims, wherein, The eye-tracking system includes a head-mounted device.

13. The eye-tracking system of claim 12, wherein, The head-mounted device includes a pair of glasses having a lens area and a pair of arms, wherein: The camera is mounted on one of the arms; and The holographic optical element is located on at least one lens area.

14. The eye-tracking system of claim 13, further comprising: An illuminator configured to illuminate the eye via one or more specular reflection regions and one or more diffuse reflection regions of the holographic optical element.

15. A computer-implemented method for operating an eye-tracking system, wherein, The eye-tracking system includes: A holographic optical element comprising one or more specular reflection regions and one or more diffuse reflection regions, wherein the one or more specular reflection regions and one or more diffuse reflection regions of the holographic optical element are provided as a predetermined pattern; The method includes: Capture an image of an eye reflected by one or more specular reflection regions of the holographic optical element, such that a pattern defined by one or more diffuse reflection regions of the holographic optical element is visible on the eye in the captured image; and Process the captured image from the eye to identify patterns in the one or more diffuse regions of the image; and Compare i) the predetermined pattern with ii) the identified pattern in the captured image to determine the distance from the holographic optical element to the eye.

Images