Eye-tracking devices and methods

Abstract

To provide an eye-tracking device and method for measuring an eye position and / or a gaze direction.SOLUTION: The method includes the steps of: receiving image data showing at least two images of the user's eyes; identifying an area associated with the corneal limbus in each image; determining a geometric representation of the limbal structure; and determining the three-dimensional position and gaze direction (i.e., the complete six degrees of freedom of the eye) of the user's eye by triangulation of the geometric representations of the limbal structures in at least two images.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an eye tracking device and method for measuring the position of an eye and / or the direction of a line of sight.

Background Art

[0002] The following shows documents considered relevant as the background art of the subject matter of the present disclosure. 1. US Patent Application No. 2013 / 120712 2. International Publication No. 9418883 3. US Patent Application No. 2014 / 180162 4. US Patent Application No. 2001 / 035938 5. Constable PA, Bach M, Frishman LJ, Jeffrey BG, Robson AG, International Society for Clinical Electrophysiology of Vision. ISCEV Standard for clinical electro-oculography (2017 update). Doc Ophthalmol. 2017;134(1):1-9 6. McCamy, Michael & Collins, Niamh & Otero-Millan, Jorge & Al-Kalbani, Mohammed & Macknik, Stephen & Coakley, Davis & Troncoso, Xoana & Boyle, Gerard & Narayanan, Vinodh & R Wolf, Thomas & Martinez-Conde, Susana. (2013). Simultaneous recordings of ocular microtremor and microsaccades with a piezoelectric sensor and a video-oculography system. Peer J. 1.e14. 10.7717 / peerj.14

[0003] The references to the above-mentioned documents in this specification should not be inferred to mean that they are in any way related to the patentability of the subject matter of this disclosure.

[0004] Various eye-tracking devices and methods, such as remote eye trackers and head-mounted eye trackers, have been known for some time. The first eye trackers were created at the end of the 19th century. However, they were difficult to manufacture and uncomfortable for subjects. Specially designed rings, contact lenses, and suction cups were attached to the eye to help measure eye movements. The first photograph-based eye trackers, which examined light reflected from various parts of the eye, were introduced at the beginning of the 20th century. These were far less invasive and marked a new era in eye research. For most of the 20th century, researchers created their own eye trackers, but they were expensive and difficult to obtain. Commercially available eye trackers did not appear until the 1970s. Since the 1950s, researchers have developed many different technologies that are still in use today, such as contact lenses with mirrors (more accurately, suction cups), contact lenses with coils, electrooculography (EOG)[5], and piezoelectric sensors[6]. However, by their very nature, these methods can only measure angular eye movements and cannot measure lateral eye movements (mirrors, coils, and EOGs can only measure angles). Furthermore, these systems are stationary and require the user's head to be stabilized, making them unsuitable for studying eye movements in more natural environments.

[0005] In the latter half of the 20th century, far less invasive illumination and light sensor-based approaches became dominant. However, the main limitation of this approach is that the camera is relatively slow due to the exposure and processing power required to extract eye movement data. Position-sensing photodetectors (PSDs) and quad-based approaches were developed in the mid-20th century. Dual Purkinje imaging eye-tracking systems were developed in the 1970s and are still in use today. However, dual Purkinje imaging eye-tracking systems are very expensive, stationary, heavy, cameraless (PSD-based) systems with relatively low recording angles. Since the 1980s, advances in camera sensors and computer technology have led to the dominance of so-called videooculography (VOG) systems in the eye-tracking market. VOG systems typically capture images of the user's eyes and identify specific features of the eyes based on the captured images. These systems are non-invasive and are usually based on irradiating the eyes with infrared light in a way that does not cause injury or discomfort to the user. Some of these systems rely on wearable, small, lightweight cameras, while more sophisticated systems, like most of the systems mentioned above, are stationary. The frame rates of these systems are limited due to the need for good lighting (but below safe levels) and the exposure requirements of the camera sensors.

[0006] In eye-tracking approaches combining pupil / corneal reflection (first Purkinje), both high-end stationary and low-end wearable systems, the eye is illuminated with multiple infrared diodes, the surface of the eye is imaged with one or more cameras (usually to increase the frame rate), and the pupil and diode first Purkinje images are segmented (as the darkest part of the eye). Changes in the position of the pupil relative to the first Purkinje image of the IR diodes indicate eye movement. User calibration is required to calculate the actual angle. Typically, the user needs to focus on several targets moving along a known path to calibrate the system.

[0007] The accuracy and precision of this method are relatively low due to the distortion of the pupil's position by the cornea, further reduction in measurement accuracy due to pupil dilation, and interference of the image processing algorithm by ambient reflection from the cornea. Additional factors such as different eye colors, long eyelashes, and contact lenses further complicate the image processing system. Therefore, these systems are typically noisy, with angular motion accuracy of less than 1 degree and no information on lateral eye movements. Furthermore, recalibration is required if the system moves relative to the head. There are also several less common approaches, including retinal imaging, approaches utilizing bright pupils, and approaches using MRI to examine eye movements. These approaches have limitations and are not widely used. Eye trackers and eye-tracking methods generally always require accuracy, precision, low latency, and miniaturization. Furthermore, as eye trackers are integrated into various devices such as computers, automobiles, and virtual reality glasses, there is a need to provide highly flexible and extremely compact eye-tracking devices.

[0008] Eye-tracking systems sometimes track the limbus (the boundary between the sclera and iris) or the boundary between the pupil and iris to measure relative eye rotation. Tracking the limbus yields accurate results because it is unaffected by corneal refractive power. Furthermore, the limbus forms a linear plane. Because the limbus defines the connection area between iris muscles, there is a direct correlation between the three-dimensional parameters of the limbus and the three-dimensional parameters of the eye. Unfortunately, the limbus is not a sharp boundary but a transition zone between the cornea and sclera. Because the limbus is insufficient as a candidate for methods relying on so-called feature point or edge detection, previous limbus trackers have been unsuccessful. Therefore, it becomes difficult to extract signals from noise due to poorly defined edges and shapes that appear to deform with rotation. As a result, methods that rely on limbal edge detection or feature point detection may not achieve the required precision and accuracy. While a suitable tracking response might be possible using a known "pinpoint" limbal tracker with a photodetector precisely positioned and aligned along the edge of the iris / sclera boundary, providing and / or maintaining such alignment adds additional system components and complexity, especially considering the variability in the geometric shape of the eye between different patients. Some limbal tracking techniques rely on a set of two quadrant detectors as a position-sensing photodetector (PSD). From the simple fact that the iris, enclosed by the limbus, is darker than the sclera, a system consisting of one or more quads can be constructed to examine the boundary of the iris / sclera region, i.e., the limbus. As the eye moves, the image of the iris also moves on the PSD, allowing for the estimation of the eye movement causing it. Despite the fact that the laser can be stabilized on the cornea, it cannot directly indicate where the eye is looking, and user calibration is required to obtain this data. Also, many factors that can be controlled during eye surgery, such as eyelid movements, eyelashes, and slight changes in lighting, affect the accuracy of this approach, making it unsuitable for eye research. PSD-based limbal trackers measure the centroid of the light spot's position. This is because there is contrast between the white sclera and the dark iris.When the eye moves relative to the system, the light point on the PSD moves accordingly. In such a system, the only thing that is controlled is that the iris image is in the same location on the PSD; the actual position of the eye is neither measured nor estimated. [Overview of the Initiative]

[0009] Eye tracking refers to the process of tracking eye movements to determine where a user is looking. The various technologies mentioned above can roughly estimate the magnitude of eye movements and indicate that movement has occurred. However, they do not measure the six degrees of freedom of eye movement. Furthermore, these systems do not track the visual features of the eye. One of the problems in tracking the visual features of the eye is that the most distinctive part of the eye is the iris. Both the pupil and the iris are strongly distorted by the cornea and lens. Because the shape of this lens varies from person to person, it is difficult to accurately determine where these features are located. Astigmatism (corneal asymmetry) further complicates this problem. The limbus (the boundary between the white sclera and the dark iris) is not distorted by the cornea, but it appears very uniform, slightly blurred, and lacks the prominent features that can be sought by classical image processing algorithms. Therefore, there is a need in this field to provide eye tracking technology based on limbal tracking. The inventors have found that eye tracking of the ring can be provided by triangulating identified stable regions that directly correspond to existing, i.e., directly visible anatomical features or virtual features (i.e., calculated features).

[0010] According to a broad aspect of the present invention, an eye-tracking method is provided, comprising the steps of: receiving image data showing at least two images of a user's eye; identifying a region related to the limbus in each image; identifying a geometric representation of the limbal structure; and determining the three-dimensional position and gaze direction (i.e., the full six degrees of freedom of the eye) of the user's eye by triangulation of the geometric representations of the limbal structure in at least two images. Because the limbus maintains a certain relationship and near-perfect roundness with respect to the lens of the cornea, the three-dimensional position and gaze direction of the user's eye can be obtained by triangulation of the geometric representations of the limbal structure in at least two images (i.e., two-dimensional limbal parameters). The shapes of at least two images are identified and matched. The three-dimensional distance to the matching region can be determined using triangulation. By applying epipolar geometric methods commonly used in this art, the corresponding region can be matched and optionally the three-dimensional limbal parameters can be determined. Based on the three-dimensional limbal parameters, the three-dimensional position and gaze direction of the user's eye are determined.

[0011] In this regard, it should be understood that stereo vision algorithms generally rely on matching corresponding points or edges between two images. There are two basic approaches to this. One is an intensity-based method, where information from the intensity value of each line is matched at each pixel, and the difference in intensity is minimized. The other is a feature-based method, where information is extracted using groups of pixel values ​​and their spatial distribution. Commonly used features are simple features, such as edge and corner detectors (Canny, Harris, LoG filters). Generally, eye-tracking systems that rely on the features described above have several drawbacks, such as different noises occurring in each of the multiple photodetectors as a result of parasitic reflections on the cornea, and distortion of iris and pupil features as a result of the curvature of the corneal lens. Therefore, eye-tracking based on feature-based methods does not yield sufficiently accurate results. Also, methods based on matching corneal reflections with eye features require user calibration. Regarding limbal tracking, as mentioned above, the limbus does not have clearly defined features, so classical approaches cannot provide three-dimensional limbal parameters. It should be noted that the technology of the present invention provides accurate eye tracking with precision and accuracy ranging from approximately 1 / 6 to 1 / 60 of a degree, without requiring user calibration of the eye-tracking device.

[0012] In some embodiments, the method includes the step of capturing at least two images of the user's eyes at different angles.

[0013] In some embodiments, identifying a geometric representation of the limbal structure includes digital image preprocessing. The digital image preprocessing includes calculating an image intensity gradient map of the limbus based on pupillary orientation; identifying, in each image, at least one region of the limbal structure where the local direction of the gradient is substantially uniform; processing data representing the limbal structure by weighting pixels in such regions; and generating a geometric representation of the limbal structure based on matching pixels associated with the corneal limbus.

[0014] In some embodiments, the method includes the step of determining three-dimensional ring parameters, which include at least one of the coordinates of the ring's center, the direction of the normal to the ring plane, the position of the ring plane, and the size of the ring. The three-dimensional position and gaze direction of the user's eye can be determined based on the three-dimensional ring parameters.

[0015] In some embodiments, the method further includes the step of determining the radius and / or torsional rotation of the ring portion.

[0016] In some embodiments, the method further includes the steps of identifying image data representing the initial limbal region in each image using an iterative pixel filtration process, and generating data representing the initial limbal region.

[0017] In some embodiments, identifying image data representing the initial limbal region in each image includes at least one of the following: identifying image data representing eye features such as the pupil, eyelids, sclera, iris, and eyelashes; and identifying the initial limbal region based on anatomical parameters. Identification of image data representing eye features can be performed using machine learning.

[0018] In some embodiments, the identification of image data representing eye features includes segmenting each image to identify pixels related to the pupil region.

[0019] In some embodiments, the method further includes the steps of: determining three-dimensional pupil parameters by performing triangulation between at least two geometric representations of the pupil; estimating the position of the initial limbus region based on the three-dimensional pupil parameters; and generating data indicating the initial limbus region position in each image. The three-dimensional pupil parameters include at least one of the following in each image: the direction of the normal to the pupillary plane, the coordinates of the center of the pupil, and the diameter of the pupil.

[0020] In some embodiments, the method further includes the steps of processing a triangulated three-dimensional ring, reprojecting the triangulated ring region onto an image plane, and refining the ring region in each image.

[0021] According to another broad aspect of the present invention, an eye-tracking device is provided comprising a processing unit, which is configured and operable to receive at least two images representing the user's eye, identify a region related to the limbus in each image, identify a geometric representation of the limbal structure, and determine the three-dimensional position and gaze direction of the user's eye by triangulation of the geometric representations of the limbal structure in at least two images. The processing unit is configured and operable to determine the three-dimensional position and gaze direction of the user's eye based on three-dimensional limbal parameters. The eye-tracking device of the present invention may be a head-mounted eye-tracking device such as eye-tracking glasses, or an eye-tracker integrated into a helmet, or it may be integrated into a head-mounted display device, virtual reality glasses, augmented reality glasses, or other head-mounted device. The eye-tracking device may also be a remote eye-tracker optionally integrated or coupled with other devices such as a computer, display, or monitor.

[0022] In some embodiments, the eye-tracking device further comprises at least two imagers, each imager configured to capture images of at least one of the user's eyes at different angles.

[0023] The imager may be a capture unit and may include one or more cameras (i.e., digital cameras or video cameras), optical sensors, and image sensors such as CCD sensors or CMOS sensors. A single imager can be used and moved at a known speed along a known trajectory. Alternatively, it is also possible to use optical images with some kind of mirror. In some embodiments, the present invention uses a stereo imager and uses two or more imagers with a precisely known relative offset. The two or more imagers can capture images simultaneously or with a known time delay (e.g., in interlaced mode).

[0024] Generally, the processing unit may be a processor, controller, microcontroller, or any type of integrated circuit. The eye-tracking device may be associated with an optical system that includes one or more optical elements such as lenses, prisms, beam splitters, mirrors, reflectors, optical guides, collimators, etc. The eye-tracking device may also be associated with an illumination source configured to produce illumination that is as dispersed as possible, preferably not creating or minimizing patterns on the surface of the eye. The eye-tracking device includes a processing unit that communicates data with each of at least two imagers to receive at least two images.

[0025] The novel technology of the present invention uses two or more captured images of the eye, and in each captured image, detects three-dimensional limbal region parameters by an iterative pixel filtering process, thereby determining the orientation of the eyeball by triangulation of the two-dimensional limbal region parameters obtained from the images. This novel technology determines the orientation of the eye based on the identification of the user's limbus in the image data. Two or more imagers can be used to capture two or more images of each of the user's eyes. Each imager is configured to capture the user's eye from a different angle. Each image visualizes the user's iris and adjacent and peripheral regions such as the pupil, sclera, and eyelids. For triangulation of the limbal region, multiple images of the eye are captured from two or more different angles, preferably at close time intervals (e.g., simultaneously).

[0026] In some embodiments, the processing unit includes an area detector that receives each of at least two images indicating the user's eyes, and in each image, uses an iterative pixel filtering process to identify image data indicating an initial rim area, preferably an area / section without initial distortion (i.e., not distorted by the eye optical system itself such as the cornea), or an area with known distortion corresponding to the same physical area of the eye, and is configured to generate data indicating the initial rim area. The inventors have found that such a stable area identified in this way can be used for triangulation calculations. In the present invention, such stable areas corresponding to the same physical area of the eye can be generated and identified on a stereo image. The area detector is configured to identify image data indicating the initial rim area by at least one of identifying image data indicating eye features such as the pupil, eyelid, sclera, iris, eyelashes, etc., and identifying the initial rim area based on anatomical parameters. These areas are identified by projecting the geometric representation of an eye feature (e.g., the pupil) onto each image, and then these areas are collated between images taken at different angles. The larger the size of such an area on the image, the more accurately the position of the corresponding physical area is triangulated, and as a result, the position of the eye itself is accurately determined. The area detector uses an iterative pixel filtering process that enables the execution of triangulation to generate a function that defines a two-dimensional virtual object on the eyeball. By this triangulation, the direction of the rim corresponding to the line of sight is defined. The rim parameters are defined as a mathematical formula where the variables represent the six degrees of freedom of a rigid body in three-dimensional space (x, y, z, α, β, δ). Specifically, x, y, z define the change in position due to forward / backward (surge), up / down (heave), left / right (sway) movement along three perpendicular axes, combined with the change in orientation due to rotation about three perpendicular axes, yaw (vertical axis), pitch (horizontal axis), roll (vertical axis).

[0027] In this regard, it should be understood that because the dimensions of the limbus are large and virtually undistorted by the cornea, by identifying the three-dimensional limbus parameters, it is possible to find regions without such distortion, or regions with known distortion corresponding to the same physical region of the eye. However, as mentioned above, the limbus itself has a slightly blurred and indistinct boundary, appears differently in each image, and does not have identifiable features that can function as anchor points for such matching. Therefore, it is difficult to match the limbus region itself across two or more different images. We have found that, instead of matching the features of the limbus region itself, it is possible to perform digital image preprocessing to compute several virtual regions based on the three-dimensional parameters of the limbus in each image that well correspond to the same physical region of the eye. Digital image preprocessing involves the use of complex algorithms such as classification, feature extraction, multiscale signal analysis, pattern recognition, or projection mathematical transformation. The virtual regions must correspond to the same physical region of the eye, regardless of the angle at which the image was taken. For this reason, the region detector is configured and operable to identify image data representing the limbus region in each image and generate data representing the limbus region. This can be done by estimating the limbal region using the anatomical relationship between an identified virtual region corresponding to specific eye features and the limbus. For example, the pupil's position and center can be calculated by triangulation of the pupil region (a defined circular region with contrast) between two images, and then the limbal region can be defined as a circular region with a certain radius range (e.g., approximately 5.5 mm to 6.5 mm) centered on the pupil's center. This radius range may be based on statistical data defining a Gaussian distribution of the distance between the limbus and the pupil's center.

[0028] In some embodiments, a pattern recognition tool is used to identify the pupil pixels and eyelid pixels within each image. The image pixels of the pupil can be identified by a specific contrast within the image. The identified image regions corresponding to the pupil are discarded, and the image regions of the eyelid can be used to determine calibration data (such as the distance / angle of each imager, the center of the pupil / limbus, the overall direction, etc.). More specifically, since the data image related to the possible intersection between the eyelid and the presumed limbus region is discarded, the number of potential limbus pixels can be further reduced. Next, based on the calibration data and the three-dimensional model of the eye, two-dimensional corneal limbus parameters are identified. Thereafter, in each of two or more captured images, an iterative filtering process is used to discard the pixels located at the boundary of the limbus. The geometric representation of the limbus structure includes a ring-shaped or elliptical structure. Assuming that the limbus is circular, using the two-dimensional parameters of the limbus obtained from each image, for each image, a complete limbus ring / ellipse structure function (reconstructed by discarding the eyelid region hiding the limbus region) is defined, and the orientation of the eye is specified by the triangulation of the defined limbus ring / ellipse structure obtained from each image. By filtering the image data corresponding to specific eye features, specific pixels can be discarded and the resolution can be increased. If the orientation of the eye cannot be specified, this process is repeated while increasing the region of the limbus and potentially retaining the pixels related to the limbus.

[0029] In some embodiments, the method first performs a mathematical transformation, such as separating the limbus region and finding a geometric representation of the limbus structure (e.g., by projecting a set of elliptical functions passing through the center of the limbus ring) in each of the images, and then triangulates the corresponding circles in six degrees of freedom (6DOF) that are accurately aligned with the physical eye.

[0030] In some embodiments, the processing unit is configured and operable to include a limbal detector which receives data indicating limbal regions and determines the geometric representation of the limbal structure by digital image preprocessing, such as performing limbal recognition processing for each image and / or performing mathematical transformations, including the use of intensity gradient maps. More specifically, the digital image preprocessing may include calculating an image intensity gradient map of the limbal region, identifying at least one region of the limbal structure where the local direction of the gradient is substantially uniform, processing data indicating the limbal structure by weighting pixels in such region, and determining the geometric representation of the limbal structure based on matching pixels associated with the corneal limbus.

[0031] More specifically, after estimating the limbal region, a ring-shaped area is projected onto the surface of the eye within the estimated limbal region. Subsequently, most pixels of the image, including the pupil, most of the iris, most of the sclera, and most of the eyelid, are discarded, leaving only the ring around the potential limbal position based on the estimated anatomical characteristics of a normal eye and, optionally, the calculated three-dimensional pupil position. However, corneal distortion in the pupil image can introduce further estimation errors. This distortion leads to an increase in the initial estimation of the limbal region.

[0032] In some embodiments, the region detector is configured to identify image data exhibiting eye features using machine learning. An iterative pixel filtering process is used to train one or more neural networks to identify the limbal region in each acquired image. The region detector may be configured and operable to identify image data exhibiting eye features by segmenting each image to identify pixels associated with the pupil region. Thus, one or more neural networks can be used to directly segment the approximate limbal region. More specifically, the pupil can be tracked to estimate the limbal region. Additionally or alternatively, the neural network can also be used to directly estimate the location of other eye features, such as the eyelids or sclera, without tracking the pupil. The neural network can be applied to the image itself or to a mathematically transformed image, i.e., a gradient map. The neural network can be trained with conventional machine learning algorithms based, for example, on the results of other eye-tracking techniques where the characteristic features of the eye are predefined.

[0033] In some embodiments, the processing unit is configured and operable to determine the size of the ring portion.

[0034] In some embodiments, pixels in the identified eyelid region are triangulated to improve the estimation of eye position.

[0035] In some embodiments, the eye-tracking device further comprises a ring triangulation instrument configured and operable to receive data representing the geometric representation of the ring structure for each image and to perform triangulation of the geometric representations of at least two images to determine three-dimensional ring parameters. The three-dimensional ring parameters include at least one of the following: three coordinates of the ring's center, the direction of the normal to the ring plane, the position of the ring plane, and the ring's size. The ring triangulation instrument may also be configured and operable to determine the ring's radius and / or its torsional rotation.

[0036] In some embodiments, the limbal triangulation instrument is configured to perform triangulation between at least two geometric representations of the pupil to determine three-dimensional pupillary parameters, the three-dimensional pupillary parameters including at least one of the direction of the normal to the pupillary plane, the coordinates of the center of the pupil, and the diameter of the pupil, and the limbal triangulation instrument estimates the position of the initial limbal region based on the three-dimensional pupillary parameters and generates data indicating the position of the initial region of the limbus in each image.

[0037] In some embodiments, the ring triangulation instrument is configured to process the triangulated three-dimensional ring, reproject the triangulated ring region onto an image plane, and refine the ring region in each image. [Brief explanation of the drawing]

[0038] Embodiments will be described, with reference to the accompanying drawings, as merely non-limiting examples, in order to better understand the subject matter disclosed herein and to illustrate how it may be put into practice. [Figure 1] Figure 1 is a schematic block diagram showing the main functional parts of the eye-tracking device of the present invention. [Figure 2] Figure 2 is a schematic flowchart showing the main steps of the eye-tracking method of the present invention. [Figure 3] Figure 3 shows two stereo images of an eye captured at different angles. [Figure 4] Figures 4A to 4D show examples of pupil detection steps according to several embodiments of the present invention. [Figure 5] Figure 5 shows an example of the eye feature detection step illustrated on the image in Figure 3, according to some embodiments of the present invention. [Figure 6] Figure 6 shows an example of the projection of the ring circle illustrated on the image in Figure 3, according to some embodiments of the present invention. [Figure 7]Figure 7 is a schematic block diagram illustrating possible functional parts of an eye-tracking device according to several embodiments of the present invention. [Figure 8] Figure 8 shows a two-dimensional eye-tracking trajectory obtained using the teaching method of the present invention. [Figure 9] Figure 9 shows the two angular degrees of freedom (i.e., angle measurements) of the eye-tracking directional trajectory with respect to azimuth and pitch as functions of time. [Figure 10] Figures 10A and 10B show the five degrees of freedom of the eye-tracking trajectory obtained using the teaching of the present invention. [Modes for carrying out the invention]

[0039] Referring to Figure 1, an example block diagram illustrating the main functional components of the eye-tracking device of the present invention is shown. The eye-tracking device 100 comprises an operable processing unit 106 configured to receive at least two images representing the user's eye, identify a region related to the limbus in each image, identify the geometric representation of the limbal structure, and measure the three-dimensional position and gaze direction of the user's eye by triangulation of the geometric representations of the limbal structure in at least two images. In this regard, as described above, it should be noted that the present invention triangulates the shape (geometric representation), but does not triangulate individual corresponding points or edges.

[0040] The processing unit 106 is typically configured as a computing / electronic utility, specifically including utilities such as data input / output modules / utilities 106A and 106B, memory 106D (i.e., a non-volatile computer-readable medium), and analyzer / data processing utility 106C. Therefore, the utilities of the processing unit 106 can be executed by appropriate circuitry and / or by software and / or hardware components containing computer-readable code configured to perform the operation of method 200 shown later in Figure 2.

[0041] Features of the present invention may include general-purpose or dedicated computer systems, including various computer hardware components, which are described in more detail below. Features within the scope of the present invention also include computer-readable media for executing, or having, computer-executable instructions, computer-readable instructions, or data structures stored therein. Such computer-readable media may be any available medium accessible by a general-purpose or dedicated computer system. In non-limiting examples, such computer-readable media may include physical storage media such as RAM, ROM, EPROM, flash disks, CD-ROMs or other optical disk storage, magnetic disk storage or other magnetic storage devices, or other media that can be used to hold or store desired program code means in the form of computer-executable instructions, computer-readable instructions, or data structures, and that are accessible by a general-purpose or dedicated computer system. Computer-readable media may include computer programs or computer applications that can be downloaded to a computer system via a wide area network (WAN), such as the Internet.

[0042] In this specification and the following claims, “processing unit” is defined as one or more software modules, one or more hardware modules, or a combination thereof, which work together to perform operations on electronic data. For example, the definition of a processing utility includes not only the hardware components of a personal computer but also software modules such as the operating system of a personal computer. The physical layout of the modules is irrelevant. A computer system may include one or more computers connected via a computer network. Similarly, a computer system may include a single physical device in which internal modules (such as memory and a processor) work together to perform operations on electronic data. Any computer system may be mobile, but as used herein, the terms “mobile computer system” or “mobile computer device” include, in particular, laptop computers, netbooks, mobile phones, smartphones, wireless phones, personal digital assistants, and portable computers with touch-sensitive screens. Processing unit 106 has a processor built in or fitted with a processor that runs a computer program. A computer program product may be embedded in one or more computer-readable media and have computer-readable program code embedded therein. The computer-readable media may be a computer-readable signaling medium or a computer-readable storage medium. Computer program code for performing operations according to aspects of the present invention may be written in any combination of one or more programming languages. The program code may run entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or to an external computer (for example, via the Internet using an Internet service provider). These computer program instructions are provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device, and the instructions executed via the processor of the computer or other programmable data processing device may result in a machine that creates means for performing a function / operation specified in one or more blocks of a flowchart and / or block diagram. The specified function of the processor may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0043] The eye-tracking device 100 may include at least two imagers 110, each imager configured to capture an image of at least one of the user's eyes at different angles. Each imager can focus on the user's iris. In this regard, it should be noted that the limbal tracking of the present invention is performed independently for each eye.

[0044] In some embodiments, the processing unit 106 includes a ring detector 102 configured and operable to receive data indicating ring regions and to identify the geometric representation of ring structures (e.g., ring-shaped or elliptical structures) through digital image preprocessing. In specific non-limiting examples, the ring detector 102 is configured and operable to perform ring recognition processing on each image and / or perform mathematical transformations of the image, including the use of intensity gradient maps, and then perform ring region recognition processing on the transformed image. In other words, the ring detector 102 can search for ring regions on the transformed image after transforming the image into an intensity gradient map. Alternatively, an entropy map may be used instead of, or in addition to, a gradient map, or processing may be performed directly on the image. When an intensity gradient map is used, the limbal detector 102 can process data indicating the limbal structure by calculating an image intensity gradient map of the limbal region, identifying at least one region of the limbal structure where the local direction of the gradient is substantially uniform, and weighting the pixels in such regions to identify a geometric representation of the limbal structure based on matching pixels associated with the corneal limbus. In addition to collinearity, knowledge of the anatomical morphology of the limbus (i.e., that the collinearity vectors radiate from the center of the limbus and that the entire limbus is continuous) can be used. More specifically, regions of the limbal structure are identified by identifying local uniformity, i.e., that the direction of the gradient at each point is collinear only with neighboring points.

[0045] Each module of the processing unit 106, such as the ring detector 102, is not limited to a specific number of modules and may be configured to process multiple images independently, simultaneously, or in other ways. The ring triangulator 108 is configured and operable to perform triangulation of the geometric representation of the ring structure in at least two images in order to determine the three-dimensional ring parameters. The ring triangulator 108 may also be configured and operable to determine the radius and / or torsional rotation of the ring. The three-dimensional ring parameters define at least five degrees of freedom of the position and orientation of the ring. The three-dimensional ring parameters include any one of the following: the position of the plane of the ring and / or the direction of the normal to the plane of the ring and / or the size of the ring and / or the center of the ring or a combination thereof.

[0046] In some embodiments, the processing unit 106 comprises a region detector 104, which is configured and operable to receive each image of at least two images representing each user's eye, identify image data representing the initial limbal region within each image by using an iterative pixel filtering process, and generate data representing the initial limbal region. This can be done by identifying image data representing eye features such as the pupil, eyelid, sclera, iris, and eyelashes, and / or by identifying the initial limbal region based on anatomical parameters. More specifically, the identification of the initial limbal region may include pupil segmentation and pupil triangulation, and / or estimation of the limbal region based on anatomical structure, and / or estimation of the eyelid region, and / or estimation of the limbal region excluding the eyelid region. This may be performed by a limbal triangulator 108 or by any other processing module. However, the present invention is not limited by a particular modular configuration of the processing unit 106. The processing unit 106 may include a pupil detector, a pupil triangulator, and other modules. For example, the identification of image data representing eye features may include pupil segmentation. Therefore, the region detector 104 may be a pupil detector intended to provide an initial limbal ring. These regions, both the pupil and the limbus, are assumed to be concentric and coplanar. Pupil segmentation can be performed by segmenting each image to identify pixels related to the pupil region, performing triangulation between at least two images to determine three-dimensional pupil parameters, estimating the position of the initial limbal region based on the three-dimensional pupil parameters, and generating data indicating the initial region position of the limbus in each image. The three-dimensional pupil parameters include the direction of the normal to the pupillary plane, and / or the coordinates of the center of the pupil, and / or the diameter of the pupil. The three-dimensional pupil parameters are used to estimate the three-dimensional limbal parameters, thereby defining an initial limbal region that can be further refined.Furthermore, the identification of image data representing eye features may include eyelid identification and filtering of this image data, as image data related to the eyelids may obstruct a portion of the limbal ring. The present invention is not limited to a specific method for identifying image data representing the initial limbus. A neural network-based approach can be used to identify image data representing the initial limbus. For example, limbal region estimation can be performed using a neural network, while the eyelid region can be estimated using a classical algorithm with a pupil detector. Alternatively, both limbal region estimation and eyelid region estimation can be performed using a neural network-based approach. Alternatively, the neural network can be used only for pupil segmentation.

[0047] In some embodiments, the region detector 104 is configured to identify image data exhibiting eye features using machine learning. Machine learning can self-classify / learn the characteristics of input data related to eye features. The region detector 104 can predict three-dimensional limbal region parameters using a data recognition model based on a neural network. The network can be trained based on segmentation results from a classical approach or using an existing system. Additionally or alternatively, the three-dimensional limbal region parameters obtained by the eye-tracking device of the present invention can be used to train the neural network. A deep learning network (DLN), such as an artificial neural network (ANN), run by the region detector 104 can generate a representation of the limbus based on a series of images. For example, the representation generated by the DLN may include probabilities regarding the placement of the limbus. This representation is used to generate a model of the limbus for adapting the recognition classifier to humans.

[0048] After the ring detector 102 obtains the three-dimensional ring parameters, the region detector 104 can receive the three-dimensional ring parameters and generate more accurate data indicating the ring region. For example, the ring triangulation instrument 108 is configured to process the triangulated three-dimensional ring, reproject the triangulated ring region onto the image plane, and refine the ring region in each image.

[0049] Referring to Figure 2, an illustrative flowchart shows the main steps of the eye-tracking method of the present invention. The eye-tracking method 200 includes, in 204, receiving image data showing at least two images of the user's eye; in 206, identifying a region related to the limbus in each image; in 208, identifying a geometric representation of the limbal structure; and in 210, performing triangulation of the geometric representations of the limbal structure in at least two images, thereby determining, in 212, the three-dimensional position and gaze direction of the user's eye.

[0050] In some embodiments, step 210 includes performing triangulation of the geometric representation of the ring structure in at least two images in order to determine the three-dimensional ring parameters as described above. Determining the three-dimensional ring parameters may include determining the radius and / or torsional rotation of the ring.

[0051] In some embodiments, after triangulation of the geometric representation of the ring structure in at least two images in 210, the method 200 may include the step of identifying the geometric representation of the ring structure by processing three-dimensional ring parameters to generate more accurate data indicating the ring region.

[0052] In some embodiments, the method 200 may include the step of capturing at least two images of each user's eyes at different angles in 202.

[0053] In some embodiments, identifying the geometric representation of the limbal structure in 208 may include digital image preprocessing, such as performing a limbal recognition process on each image or performing a mathematical transformation of the image, including using an intensity gradient map, and then performing a limbal region recognition process on the transformed image, in 218. The digital image preprocessing step in 218 may include processing data indicating the limbal structure by calculating an image intensity gradient map of the limbal region, identifying at least one region of the limbal structure where the local direction of the gradient is substantially uniform, weighting the pixels in such region to identify the geometric representation of the limbal structure based on matching pixels associated with the corneal limbus. More specifically, the digital image preprocessing may include identifying regions of the limbal structure by identifying local uniformity, i.e., the direction of the gradient at each point is collinear only with its adjacent points. As described above, an entropy map can be used as an intensity gradient map (instead of or in addition to a gradient map), or the processing can be performed directly on the image.

[0054] In some embodiments, in 206, identifying regions related to the limbus in each image may include identifying image data representing eye features such as the pupil, eyelids, sclera, iris, and eyelashes, and / or identifying the initial limbal region based on anatomical parameters using iterative pixel filtering, and generating data representing the initial limbal region. For example, the eyelid region can be estimated in parallel with the pupil and limbal regions. The present invention is not limited to the order in which the various regions are estimated. The various eye feature regions may be identified based on any geometric shape model (e.g., ellipses or circles).

[0055] In some embodiments, identifying the initial limbal region may include pupil segmentation and pupil triangulation, and / or estimation of the limbal region based on anatomical structure, and / or estimation of the eyelid region, and / or estimation of the limbal region excluding the eyelid region. The method may include the steps of segmenting each image to identify pixels related to the pupil region, performing triangulation between at least two images to determine three-dimensional pupillary parameters, estimating the location of the initial corneal limbal region based on the three-dimensional pupillary parameters, and generating data indicating the location of the initial region of the limbus in each image. Identifying image data representing eye features may also include identifying the eyelid and filtering this image data, since image data related to the eyelid may obstruct part of the limbal ring. As described above, the present invention is not limited to any particular method for identifying image data representing the initial limbus. A neural network-based approach can be used to identify image data representing the initial limbus. Limbal region estimation and eyelid region estimation can also be performed using a neural network-based approach. Neural networks may also be used only for pupil segmentation.

[0056] In some embodiments, machine learning can be used to perform the identification of image data that exhibits eye features. Determining (i.e., predicting) three-dimensional limbal parameters may involve using a data recognition model based on a neural network. This method may involve training the network based on segmentation results from classical approaches or using existing systems for training. Generating a limbal representation based on a series of images can be achieved using a deep learning network (DLN), such as an artificial neural network (ANN). For example, generating a limbal representation may involve calculating probabilities regarding the placement of the limbus and / or generating a model of the limbus, adapting the recognition classifier to humans.

[0057] Referring to Figure 3, two stereo images of the user's eyes, taken at different angles, are shown.

[0058] Referring to Figures 4A to 4D, examples of data processing of two stereo images from Figure 3 for pupil region detection according to some embodiments of the present invention are shown. Figure 4A shows a segmentation step in which each image is segmented to identify pixels associated with the pupil region. As shown in Figure 4B, pixels not associated with the pupil region are discarded to increase the resolution. Figure 4C shows segmentation of the pupil boundary based on the intensity map of each image. Figure 4D shows a refined pupil boundary based on the projection of the geometric representation of the pupil (e.g., a ring shape) in each image. Subsequently, the region in Figure 4D is triangulated between the images to determine the pupil diameter, 3D position, and orientation.

[0059] Referring to Figure 5, an example of data identification of eye features in the two stereo images of Figure 3 is shown according to some embodiments of the present invention. Marked regions represent areas where the local direction of the image intensity gradient is substantially uniform.

[0060] Referring to Figure 6, an example of a ring recognition process based on the projection of an elliptic function onto each of the two stereo images in Figure 3, according to some embodiments of the present invention, is shown.

[0061] Referring to Figure 7, an illustrative block diagram shows possible functional components of the eye-tracking device of the present invention. In this specific non-limiting example, the eye-tracking device 300 comprises two imagers, referred to herein as Cam 1 and Cam 2, configured to capture images of at least one of each user's eyes at different angles. Each imager focuses on the user's iris. The eye-tracking device 300 comprises a processing unit 310 including a pupil detector 302 and a limbal detector 304. The pupil detector 302 is a specific example of the region detector defined with respect to Figure 1 above. The pupil detector 302 comprises a plurality of modules and is configured and operable to receive each image of at least two images representing each user's eye, identify image data representing the limbal region within each image, and generate data representing the limbal region. This may be done using eye feature detection such as pupil region detection, or it may be done using a neural network. Each image is segmented by a segmentation module to identify pixels related to the pupil region. Pixels not related to the pupil region are discarded in each image to increase the resolution. Subsequently, each elliptic module Ell calculates the elliptic curve corresponding to the pupil region on each image and finds a matching elliptic curve. For each image, a three-dimensional equation representing the three-dimensional pupil region is generated. Then, given that the distance between two imagers capturing two stereo images is known, an initial triangulation is performed to match the three-dimensional equations of the two stereo images and determine the direction of the pupil. Following this, using the average size of the limbus and assuming that the pupil and limbus are concentric and coplanar, the algorithm yields a three-dimensional limbus parameter estimate. This three-dimensional limbus region is projected onto each image, referred to herein as the mask equation, as shown in Figure 3. For example, the limbus region may be enclosed by two elliptic curves. The mask equation is used by the collinearity map module to give the initial estimate of the limbus projection on each image. The mask equation is also used by the weighting module.

[0062] To determine the three-dimensional limbal parameters, a collinearity module, referred to herein as a collinearity map, performs a mathematical transformation. In specific non-limiting examples, such a mathematical transformation may include calculating an image intensity gradient map of the limbus based on pupillary orientation, identifying at least one region of the limbal structure where the gradient direction is substantially uniform, processing data representing the limbal structure by weighting pixels in such regions, and determining the three-dimensional limbal parameters based on matching pixels associated with the corneal limbus. Alternatively, such a mathematical transformation may include using the entropy of the image. In specific non-limiting examples, the collinearity map module generates a gradient collinearity map from the original image masked by an estimated limbal region (calculated from the pupillary 3D parameters and mean limbal size). The limbal detector 304 receives a three-dimensional equation indicating the three-dimensional limbal region and the orientation of the limbus. The ring-shaped region is then projected onto the image of each eye to generate the estimated limbal region. Most pixels in the image, including the pupil, a large portion of the iris, a large portion of the sclera, and a large portion of the eyelid, are discarded using a mask equation, leaving only a ring around the potential limbal region based on the estimated anatomical characteristics of a normal eye.

[0063] The weighting module assigns a weight to each pixel, based on the collinearity map module, representing the probability that the pixel belongs to the ring. The weighting module is then used to narrow down the potential ring region, providing input to the Ell module, which finds an estimate of the matching ellipse of the ring.

[0064] To perform a second triangulation, several techniques can be employed, such as cone triangulation or quadratic estimation (linear or nonlinear). The processing unit 310 is configured and operable to determine the three-dimensional position and gaze direction of the user's eye by triangulating the geometric representations of the limbal structure of at least two images using linear or nonlinear quadratic estimation. In this specific non-limiting example, cone triangulation can be performed as follows: Each ellipse module Ell generates a mathematical function / equation of the geometric representation of the limbal structure that defines the anatomical relationships between identified virtual regions corresponding to specific eye features (e.g., ellipses with centers corresponding to the pupil centers), and the limbal structure is calculated for each image. The elliptical structures of each image are compared, and the intersection of the cones defined by the elliptical structures defines an ellipse in space, whose projection coincides with the elliptical structures of the two images. This defines three-dimensional limbal parameters, referred to herein as limbusR, Cx, Cy, Cz, azimuth, and pitch. In this regard, it should be noted that in this invention, the three-dimensional ring parameters are obtained by ellipse triangulation, not pixel triangulation. As mentioned above, pixel triangulation of the ring is virtually impossible because the correspondence between pixels in different images is unknown (most pixels along the epipolar line of the second image are very similar). Classical computer vision can characterize the matching between two pixels along the epipolar line of different images using special features of adjacent pixel regions, such as different colors, corners, line intersections, or texture patches. Such features are missing in the ring region due to the rough boundary of the ring, which is generally defined as a large area, and therefore classical natural feature tracking algorithms cannot be used. Subsequently, a mask is used to discard regions related to eye features. Then, the same procedure as above is repeated using an iterative pixel filtering process to more accurately define the ring region and reduce its size.

[0065] Alternatively or additionally, the iterative pixel filtering process can be performed using linear quadratic estimation, such as the Kalman filter, or nonlinear quadratic estimation, such as the extended Kalman filter and unscented Kalman filter. The Kalman filter generates estimates of the ring equation for five degrees of freedom (e.g., an elliptic structure) as a weighted average of the predicted eye state and the new measured state. The weights are calculated from the covariance, which is a measure of the estimated uncertainty of the predicted eye state. As a result of the weighted average, the estimate of the new state lies between the predicted and measured states and has better estimated uncertainty than either state taken individually. This process is repeated at each time step, and the new estimate and its covariance inform the prediction to be used in the next iteration.

[0066] At the end of the process, the processing unit 310 determines the three-dimensional position and gaze direction of the user's eye based on the three-dimensional ring parameters.

[0067] Referring to Figure 8, an example of a two-dimensional eye-tracking position trajectory obtained using the teaching of the present invention is shown, where different colors distinguish different parts of the trajectory, i.e., the color changes over time. Axes Xc and Yc are measured in millimeters.

[0068] Referring to Figure 9, the two angular degrees of freedom (i.e., angular measurements measured in degrees) of the eye-tracking trajectory in Figure 8 are illustrated with respect to azimuth and pitch as a function of time (measured in milliseconds). Figures 8, 9, and 10 all show data from the same recording.

[0069] Referring to Figures 10A and 10B, the five degrees of freedom of the eye-tracking trajectory in Figure 8 are illustrated. More specifically, Figure 10A shows the two angular degrees of freedom of the eye-tracking directional trajectory (i.e., angular measurements measured in degrees) with respect to azimuth and pitch as a function of time. Figure 10A is an enlarged portion of the graph in Figure 9. Figure 10B shows the three-dimensional degrees of freedom of the eye-tracking position trajectory in Cartesian coordinate space as a function of time (measured in milliseconds).

Claims

1. In an eye-tracking device equipped with a processing unit, The processing unit includes an ring detector that is configured and operable to receive and process image data representing at least one image of the user's eye, Processing the image data of at least one of the aforementioned images In each of the at least one images, at least one region related to the limbus, which is defined as the transitional region between the cornea and sclera of the user's eye, An eye-tracking device characterized by applying digital image preprocessing to data representing at least one region related to the limbus of the cornea to identify a geometric representation of the limbal structure.

2. In the eye-tracking device according to claim 1, Applying digital image preprocessing to data showing at least one region related to the limbus of the cornea is The process involves performing ring region recognition processing to generate data indicating the ring region, and An eye-tracking device characterized by including the following: analyzing data representing the ring region to identify the geometric representation of the ring structure.

3. In the eye-tracking device according to claim 2, An eye-tracking device characterized in that the ring region recognition process includes performing a mathematical transformation of the image using at least one of an intensity gradient map and an entropy map, and searching for the ring region in the transformed image.

4. In the eye-tracking device according to any one of claims 1 to 3, The ring-shaped detector is configured and operable to perform the digital image preprocessing, An eye-tracking device characterized in that the digital image preprocessing includes: calculating an image intensity gradient map of at least one region related to the limbus; identifying a region in the image intensity gradient map where the local direction of the image intensity gradient is locally uniform as the limbus region; and weighting the pixels of the at least one region to identify a geometric representation of the limbus structure based on matching pixels related to the limbus structure.

5. The eye-tracking device according to any one of claims 1 to 4 further comprises a region detector, wherein the region detector An eye-tracking device characterized by being configured and operable to receive at least one image representing a user's eye, identify image data representing an initial ring region in the at least one image by using an iterative pixel filtering process, and provide data representing the initial ring region to the ring detector.

6. In the eye-tracking device according to claim 5, The aforementioned region detector, Identifying image data showing eye features including pupil, eyelid, sclera, iris, and eyelashes, and An eye-tracking device characterized by being configured to identify image data representing the initial limbal region by at least one of the following: identifying the initial limbal region based on anatomical parameters.

7. In the eye-tracking device according to claim 6, An eye-tracking device characterized in that the region detector is configured to identify image data that shows features of the eye using machine learning.

8. In the eye-tracking device according to claim 6, An eye-tracking device characterized in that the region detector is configured and operable to identify image data representing eye features by segmenting each image to identify pixels related to the pupil region.

9. In the eye-tracking device according to any one of claims 1 to 8, An eye-tracking device characterized in that the geometric representation of the ring-shaped or elliptical structure includes a ring-shaped or elliptical structure.

10. In eye-tracking methods, The steps include receiving image data showing at least one image of the user's eye, The steps include identifying at least one region in the image data that is related to the limbus, which is defined as the transitional region between the user's cornea and sclera, A method characterized by comprising the step of performing digital image preprocessing on data showing at least one region related to the limbus of the cornea to identify a geometric representation of the limbal structure.

11. In the method according to claim 10, Performing digital image preprocessing on data representing at least one region related to the limbus of the cornea is, The process involves performing ring region recognition processing to generate data indicating the ring region, and A method characterized by comprising: analyzing data representing the ring region to identify the geometric representation of the ring structure.

12. In the method according to claim 11, A method characterized in that the ring region recognition process includes performing a mathematical transformation of the image using at least one of an intensity gradient map and an entropy map, and searching for the ring region in the transformed image.

13. In the method according to any one of claims 10 to 12, A method characterized in that performing the digital image preprocessing includes: calculating an image intensity gradient map of at least one region related to the limbus; identifying a region in the image intensity gradient map where the local direction of the image intensity gradient is locally uniform as the limbus region; and weighting the pixels of the at least one region to identify a geometric representation of the limbus structure based on matching pixels related to the limbus structure.

14. The method according to any one of claims 10 to 13 is further, The steps include applying an iterative pixel filtering process to at least one image of the user's eye to identify image data representing the initial limbal region, A method characterized by comprising the step of generating data indicating the initial ring region.

15. In the method according to any one of claims 10 to 14, A method characterized in that the geometric representation of the ring structure includes a ring-shaped or elliptical structure.

16. In the method according to any one of claims 10 to 15, The step of identifying image data showing at least one region related to the corneal region in the above-mentioned image of at least one image is: Identifying image data showing eye features including pupil, eyelid, sclera, iris, and eyelashes, and A method characterized by comprising at least one of the following: identifying at least one region related to the limbus of the cornea based on anatomical parameters.

17. In the method according to claim 16, A method characterized in that identifying the image data that shows the characteristics of an eye includes performing machine learning.

18. In the method according to claim 17, A method for identifying the image data representing the features of an eye, comprising: segmenting the image to identify pixels related to the pupil region; and identifying at least one limbal region based on the pupil region.

Images