Eye tracking device and method thereof

Abstract

This invention relates to an eye-tracking method and apparatus. The method includes receiving image data of at least two images indicating a user's eyes, identifying a region associated with the limbus of the eye in each image; determining a geometric representation of the limbal structure; and determining the three-dimensional position and gaze direction of the user's eyes (i.e., all six degrees of freedom of the eye) by triangulation of the geometric representation of the limbal structure in the at least two images.

Description

Technical Field

[0001] The present invention relates to eye tracking devices and methods for determining eye position and / or viewing direction. Background Art

[0002] Listed below are references believed to be relevant as background to the presently disclosed subject matter:

[0003] 1. U.S. Patent Application No. 2013 / 120712;

[0004] 2. International patent application number WO9418883;

[0005] 3. U.S. Patent Application No. 2014 / 180162;

[0006] 4. U.S. Patent Application No. 2001 / 035938.

[0007] 5. Constable PA, Bach M, Frishman LJ, Jeffrey BG, Robson AG, International Society for Clinical Electrophysiology of Vision. ISCEV Standard for clinical electro-oculography (2017 update). Doc Ophthalmol. 2017; 134(1): 1-9.

[0008] 6. McCamy, Michael & Collins, Niamh & Otero-Millan, Jorge & Al-Kalbani, Mohammed & Macknik, Stephen & Coakley, Davis & Troncoso, Xoana & Boyle, Gerard & Narayanan, Vinodh & R Wolf, Thomas & Martinez-Conde, Susana. (2013). Simultaneous recordings of ocular microtremor and microsaccades with a piezoelectric sensor and a video-oculography system. Peer J.1.e14.10.7717 / peerj.14.

[0009] Acknowledgment of the above references herein should not be inferred as an intention that these are in any way relevant to the patentability of the presently disclosed subject matter.

[0010] background

[0011] Different eye tracking devices such as remote eye trackers or head mounted eye trackers and different eye tracking methods are known from the prior art. The first eye trackers were built in the late 19th century. They were difficult to build and caused discomfort to the participants. Specially designed rings, contact lenses and suction cups were attached to the eyes to facilitate the measurement of eye movements. The first photography-based eye trackers, which examined the light reflected from different parts of the eye, were introduced in the early 20th century. They were much less invasive and ushered in a new era of eye research. For most of the 20th century, researchers built their own eye trackers, which were expensive and had limited availability. The first commercially available eye trackers only appeared in the 1970s. Since about the 1950s, many different technologies have been developed by researchers that are still in use today, such as contact lenses with mirrors (more precisely, suction cups), contact lenses with coils, electrooculography (EOG) [5] and piezoelectric sensors [6]. However, these methods are only able to have angular eye movements and cannot measure any lateral eye movements due to their nature (mirrors, coils and EOG can only measure angles). Furthermore, these systems are stationary and require the user's head to be stabilized, making them unsuitable for studying eye movements in a more natural environment.

[0012] The second part of the 20th century was dominated by much less invasive illumination and light sensor based methods. The main limitation of this approach is that the cameras are relatively slow due to the exposure and processing power required to extract the eye movement data. Position sensing photodetector (PSD) or quad-wire based methods were developed in the mid-20th century. Dual Purkinje imaging eye tracker systems were developed in the 70s and are still in use today. However, dual Purkinje imaging eye tracker systems are very expensive, fixed, heavy camera-less (PSD based) systems with relatively low recording angles. Starting in the 80s, due to improvements in camera sensors and computer technology, the eye tracking market has been dominated by so-called video eye graph (VOG) systems. VOG systems typically capture an image of the user's eye and determine certain features of the eye based on the captured image. These systems are non-invasive and typically rely on infrared illumination of the eye that does not cause interference or discomfort to the user. Some of them rely on small lightweight cameras that are wearable, but more accurate systems are fixed, like most of the systems mentioned above. The frame rates of these systems are limited because good lighting (but below safe levels) is required and given the exposure requirements of the camera sensor.

[0013] Combined pupil / corneal reflection (first Purkinje) eye tracker methods (not only fixed high end but also wearable low end) illuminate the eye with multiple IR diodes and image the surface of the eye with one or more (usually to increase frame rate) cameras, segmenting the pupil (as the darkest part of the eye) and the first Purkinje image of the diodes. Changes in pupil position relative to the first Purkinje image of the IR diodes indicate movement of the eye. User calibration must be used to calculate the true angle. Typically, the user needs to focus on some target and move on a known path to calibrate the system.

[0014] The precision and accuracy of this method are relatively low because the pupil position is distorted by the cornea, pupil dilation further reduces the measurement accuracy, and environmental reflections from the cornea confuse the image processing algorithm. Different eye colors, long eyelashes, and contact lenses are additional factors that further complicate the image processing system. Therefore, these systems are usually noisy and provide less than 1 degree of accuracy for angular motion without information about lateral eye movement. In addition, recalibration is required after the system moves relative to the head. There are many other less common methods, such as imaging the retina, bright pupil method, and even using MRI machines to check eye movement. These methods have their limitations and are not very common. Generally, with respect to eye trackers and eye tracking methods, there is always a need to improve accuracy, precision, low latency, and compact size. In addition, as eye trackers become more integrated in many devices such as computers, cars, virtual reality glasses, etc., there is also a need to provide very compact eye tracking devices (providing great flexibility).

[0015] Eye tracking systems can track the limbus (border between the sclera and iris area) or pupil-iris border in particular to measure relative eye rotation. It should be noted that the limbus is not affected by the optical power of the cornea, and therefore tracking the limbus provides accurate results. In addition, the limbus creates a straight plane. The limbus defines the area of ​​connection between the muscles of the iris, and therefore there is a direct correlation between the three-dimensional parameters of the limbus and the three-dimensional parameters of the eye. Unfortunately, the limbus is more of a transition zone between the cornea and the sclera, rather than a sharp boundary. Existing limbus trackers are not successful because the limbus is a poor candidate for any technology that relies on so-called feature points or edge detection. Therefore, unclear edges and shapes that appear to be deformed due to rotation lead to difficulties in extracting signals from noise. As a result, technologies that rely on edge detection of the limbus or detection of edge feature points may lack the desired accuracy and precision. Although adequate tracking response may also be possible using known "pinpoint" limbal trackers with accurately aligned photodetectors precisely arranged along the edge of the iris / sclera interface, providing and / or maintaining such alignment adds additional system components and complexity, particularly in terms of the variability of eye geometry between different patients. Some techniques for tracking the limbus rely on a set of two-quadrant detectors as position sensing photodetectors (PSDs). The simple fact that the iris surrounded by the limbus is darker than the sclera allows the construction of a system consisting of one or more four-core wires that inspect the boundary of the iris / sclera area (i.e., the limbus). When the eye moves, the image of the iris also moves through the PSD, thus allowing estimation of the eye movement that causes this. Despite the fact that it allows the laser on the corneal area to be stabilized, it does not provide a direct indication of where the eye is looking, and user calibration is required to obtain the data. In addition, many factors that can be controlled during eye surgery, such as eyelid movement, eyelashes, and even minimal lighting changes, affect the accuracy of this method and make it unsuitable for eye research. PSD-based limbal trackers measure the centroid of the spot position. This is because there is a contrast between the white sclera and the darker iris. If the eye moves relative to the system, the spot on the PSD moves accordingly. Such a system only controls the image of the iris to be located on the same spot on the PSD, but does not measure and is not able to estimate the actual eye position.

[0016] Overview

[0017] Eye tracking refers to the process of tracking the movement of the eye and determining where the user is looking. The different techniques described above roughly estimate the scale of eye movement and can indicate that movement has occurred. However, they do not measure the six degrees of freedom of eye movement. In addition, these systems do not track the visual features of the eye. One of the problems associated with tracking the visual features of the eye is that the most obvious feature of the eye is located on the iris. Both the pupil and the iris are strongly distorted by the corneal lens. The shape of this lens varies from person to person, so it brings challenges to understand where exactly these features are located. Astigmatism (corneal asymmetry) further complicates this problem. The limbus (i.e., the boundary between the white sclera and the darker iris) is not distorted by the cornea, but it looks very uniform, slightly blurred, and does not have any prominent features sought by classical image processing algorithms. Therefore, there is a need in the art for providing a technique for eye tracking based on limbus tracking. The inventors have found that limbus eye tracking can be provided by triangulating the identified stable area, which is real, i.e., directly corresponds to some directly visible anatomical or virtual features, i.e., the calculated features.

[0018] According to a broad aspect of the invention, an eye tracking method is provided, which includes receiving image data of at least two images indicative of a user's eye, identifying an area associated with the limbus of the eye in each image; determining a geometric representation of the limbus structure, and determining a three-dimensional position and gaze direction of the user's eye (i.e., all six degrees of freedom of the eye) by triangulating the geometric representation of the limbus structure of the at least two images. Because the limbus maintains a fixed relationship and closed circularity relative to the lens of the cornea, the three-dimensional position and gaze direction of the user's eye can be obtained by triangulating the geometric representation of the limbus structure of the at least two images (i.e., two-dimensional eye limbus parameters). Shapes in the at least two images are identified and matched. Triangulation can be used to determine the three-dimensional distance to the matching area. The kernel surface geometry techniques commonly used in this field are applied to match the corresponding areas, and the three-dimensional eye limbus parameters are optionally determined. The three-dimensional position and gaze direction of the user's eye are determined based on the three-dimensional limbus parameters.

[0019] In this regard, it should be understood that stereo vision algorithms typically rely on matching corresponding points or edges between two images. There are two basic categories of methods: intensity-based techniques, in which the information in the intensity values ​​of each line is matched at each pixel and the difference in intensity is minimized; and feature-based techniques, in which a set of pixel values ​​and their spatial distribution are used to extract information. Commonly used features are simple features, such as edge and corner detectors (Canny, Harris, LoG filters). Typically, eye tracking systems that rely on the features mentioned above suffer from several disadvantages, such as different noise generated by each of the multiple photodetectors as a result of parasitic reflections on the cornea and distortion of iris and pupil features as a result of the curvature of the corneal lens. Therefore, eye tracking based on feature-based techniques does not provide sufficiently accurate results. In addition, methods based on matching corneal reflections with eye features require user calibration. With regard to limbus tracking, as mentioned above, the limbus has no clear features, and therefore classical methods do not provide three-dimensional eye limbus parameters. It should be noted that the technology of the present invention does not require user calibration of the eye tracking device and provides accurate eye tracking with accuracy and precision in the range of approximately 1 / 6 degree to 1 / 60 degree.

[0020] In some embodiments, the method includes capturing at least two images of the user's eyes at different angles.

[0021] In some embodiments, determining the geometric representation of the limbal structure includes digital image preprocessing. The digital image preprocessing includes computing an image intensity gradient map of the limbus based on the pupil direction, identifying at least one region of the limbus structure in each image where the local direction of the gradient is substantially consistent, processing data indicative of the limbus structure by weighting pixels in such a region, and generating the geometric representation of the limbus structure based on matching pixels associated with the limbus of the eye.

[0022] In some embodiments, the method includes determining three-dimensional limbus parameters, which include three coordinates of the center of the limbus and at least one of the normal direction of the limbus plane, the position of the limbus plane, and the size of the limbus. Determining the three-dimensional position and gaze direction of the user's eye can be based on the three-dimensional limbus parameters.

[0023] In some embodiments, the method further comprises determining a radius of the limbus and / or a torsional rotation of the limbus.

[0024] In some embodiments, the method further comprises identifying image data indicative of an initial limbal region in each image by using an iterative pixel filtering process and generating data indicative of an initial limbal region.

[0025] In some embodiments, identifying image data indicative of an initial limbal region in each image includes at least one of the following operations: identifying image data indicative of eye features such as pupils, eyelids, sclera, irises, and eyelashes, and identifying an initial limbal region based on anatomical parameters. Identifying image data indicative of eye features may be performed using machine learning.

[0026] In some embodiments, identifying image data indicative of eye features includes segmenting each image to identify pixels associated with a pupil region.

[0027] In some embodiments, the method further comprises performing triangulation between at least two pupil geometry representations to determine three-dimensional pupil parameters, estimating a location of an initial eye limbus region based on the three-dimensional pupil parameters, and generating data indicating an initial region location of the limbus in each image. The three-dimensional pupil parameters comprise at least one of a direction of a normal to a pupil plane in each image, a coordinate of a center of the pupil, and a pupil diameter.

[0028] In some embodiments, the method further comprises processing the three-dimensional limbus determined by triangulation, reprojecting the limbus region determined by triangulation back to the image plane, and refining the limbus region at each image.

[0029] According to another broad aspect of the present invention, there is provided an eye tracking device comprising a processing unit, the processing unit being configured and operable to receive at least two images indicative of a user's eyes, identifying in each image an area associated with the limbus of the eye; determining a geometric representation of the limbus structure, and determining a three-dimensional position and gaze direction of the user's eyes by triangulating the geometric representation of the limbus structure of the at least two images. The processing unit is configured and operable to determine the three-dimensional position and gaze direction of the user's eyes based on three-dimensional limbus parameters. The eye tracking device of the present invention may be a head-mounted eye tracking device, such as an eye tracking eyewear device or an eye tracker integrated in a helmet or also integrated in a head-mounted display device, in virtual reality glasses, in augmented reality glasses or other head-mounted devices. The eye tracking device may also be a remote eye tracker optionally integrated or coupled with other devices such as computers, displays, monitors, etc.

[0030] In some embodiments, the eye tracking device further comprises at least two imagers, each imager configured to capture at least one image of the user's eye at a different angle.

[0031] The imager can be a capture unit and can include one or more cameras (i.e., digital cameras or video cameras), light sensors such as image sensors, such as CCD sensors or CMOS sensors. One imager can be used and moved quickly on a known trajectory at a known speed. Alternatively, an optical image with a certain type of reflector can also be used. In some embodiments, the present invention uses a stereo imager to use two or more imagers with a precisely known relative offset. Two or more imagers can capture images simultaneously or with a known time delay (e.g., in an interlaced mode).

[0032] Typically, the processing unit may be a processor, a controller, a microcontroller or any kind of integrated circuit. The eye tracking device may be associated with an optical system, which may include any optical elements, such as one or more lenses, prisms, beam splitters, mirrors, reflectors, light guides, collimators, etc. The eye tracking device is also associated with an illumination source configured to produce as much dispersed illumination as possible, preferably without producing any pattern on the surface of the eye or minimizing the number of patterns. The eye tracking device includes a processing unit in data communication with each of the at least two imagers to receive the at least two images.

[0033] The new technology of the present invention uses two or more captured images of the eye, and detects three-dimensional eye limbus parameters in each captured image through an iterative pixel filtering process, thereby determining the orientation of the eyeball by triangulation of the two-dimensional parameters of the eye limbus obtained from the image. The new technology determines the eye orientation based on the identification of the limbus in the user's eye in the image data. Two or more imagers can be used to capture two or more corresponding images of the user's eye. Each imager is configured to capture an image of the user's eye from a different angle. Each imager images the user's iris and adjacent and surrounding areas (including pupil, sclera, eyelids, etc.). For triangulation of the limbus area, multiple images of the eye are preferably captured from two or more different angles close in time (e.g., simultaneously).

[0034] In some embodiments, the processing unit includes a region detector configured to receive each of at least two images indicating an eye of a user and identify in each image image data indicating an initial limbal region, i.e., some region / zone (i.e., real or virtual calculated region) that is preferably initially undistorted (i.e., not distorted by the eye optics itself, such as the cornea) or a region with known distortion corresponding to the same physical region of the eye by using an iterative pixel filtering process and generating data indicating an initial limbal region. The inventors have discovered that these identified stable regions can be used for triangulation calculations. The present invention allows such stable regions to be generated corresponding to the same physical region of the eye and to be identified on stereo images. The region detector is configured to identify image data indicating an initial limbal region by at least one of the following operations: identifying image data indicating eye features such as pupils, eyelids, sclera, irises, and eyelashes and identifying an initial limbal region based on anatomical parameters. These regions can be identified by projecting a geometric representation of eye features (e.g., pupils) on each image and then photographing the regions that match between the images at different angles. The larger the size of such an area on the image, the more accurately the position of the corresponding physical area is determined by triangulation, and therefore the more accurately the position of the eye itself is. The area detector generates a function that defines a two-dimensional virtual object on the eye by using an iterative pixel filtering process that can perform triangulation. This triangulation defines the direction of the limbus corresponding to the gaze direction. The limbus parameters are defined as mathematical equations, in which the variables (x, y, z, α, β and δ) represent the six degrees of freedom of motion of a rigid body in three-dimensional space. In particular, x, y, z define the change in position as forward / backward (fluctuation), upward / downward (undulation), left / right (sway) translation on three vertical axes, combined with changes in orientation through rotation around the three vertical axes, yaw (normal axis), pitch (lateral axis) and roll (longitudinal axis).

[0035] In this regard, it should be understood that the determination of three-dimensional limbal parameters can locate such undistorted areas or areas with known distortions corresponding to the same physical area of ​​the eye, because the size of the limbus is large and is not actually distorted by the cornea. However, as mentioned above, the limbus itself has a boundary that looks slightly fuzzy and not well defined, it looks different on each image, and it has no distinguishable features that can be used as anchor points for such matching. Therefore, matching the limbus region itself on two or more different images brings challenges. The inventors have found that instead of trying to match the features of the limbus region itself, digital image preprocessing can be performed to allow some virtual regions to be calculated based on the limbus three-dimensional parameters of each image that does correspond well to the same physical area of ​​the eye. Digital image preprocessing includes the use of complex algorithms such as classification, feature extraction, multi-scale signal analysis, pattern recognition or projection mathematical transformation. The virtual region should correspond to the same physical area of ​​the eye, independent of the angle at which the image is taken. The region detector is therefore configured and operable to identify image data indicating the limbus region in each image, and generate data indicating the limbus region. This can be achieved by estimating the limbus region by using the anatomical relationship between the identified virtual region corresponding to certain eye features and the limbus. For example, the position and center of the pupil can be calculated by triangulating the pupil region between the two images (i.e., the region with the defined circular contrast), and then the limbus region can be defined as a circular region with a certain radius range (e.g., approximately 5.5 mm to 6.5 mm) around the center of the pupil. The radius range can be based on statistics defining a Gaussian distribution of the distance between the limbus and the center of the pupil.

[0036] In some embodiments, a pattern recognition tool is used to identify pupil pixels and eyelid pixels in each image. Due to a certain contrast in the image, the image pixels of the pupil can be identified. The identified image area corresponding to the pupil is discarded, and the image area of ​​the eyelid can be used to determine calibration data (e.g., the distance / angle of each imager, the center of the pupil / corneal limbus and the general direction). More specifically, the data image associated with the possible intersection between the eyelid and the estimated corneal limbus area is discarded, thus further reducing the number of potential corneal limbus pixels. Then two-dimensional eye corneal limbus parameters are identified based on the calibration data and the three-dimensional model of the eye. Then, an iterative filtering process is used to discard pixels located at the boundary of the corneal limbus in each of two or more captured images. The geometric representation of the corneal limbus structure includes annular or elliptical structures. When it is assumed that the corneal limbus is circular, the corneal limbus two-dimensional parameters obtained from each image are then used to define a complete corneal limbus ring / ellipse structure function for each image (reconstructed after discarding the eyelid area that hides the corneal limbus area), and the eye orientation is determined by triangulation of the defined corneal limbus ring / ellipse structure obtained from each image. Filtering out image data corresponding to certain eye features can discard certain pixels and improve resolution. If the technique does not successfully determine eye orientation, the process repeats again with the limbus area being increased to potentially accommodate limbus-related pixels.

[0037] In some embodiments, the method includes the following steps: first, isolating the corneal limbal region, and then performing a mathematical transformation, such as finding a geometric representation of the corneal limbal structure on each of the images (e.g., by projecting a set of elliptic functions that go through the middle of the corneal limbal ring), and triangulating the corresponding circle in six degrees of freedom (6DOF) that is strongly aligned with the physical eye.

[0038] In some embodiments, the processing unit includes a limbus detector configured and operable to receive data indicative of a limbus region and determine a geometric representation of limbus structure by digital image preprocessing, such as performing a limbus identification process and / or performing a mathematical transformation on each image, including using an intensity gradient map. More particularly, the digital image preprocessing may include calculating an image intensity gradient map of the limbus region, identifying at least one region of limbus structure in which the local direction of the gradient is substantially uniform, processing the data indicative of limbus structure by weighting pixels of these regions and determining a geometric representation of the limbus structure based on matching pixels associated with the limbus of the eye.

[0039] More specifically, after the limbal region has been estimated, an annular region is projected onto the eye surface at the estimated limbal region. Most of the pixels of the image, such as the pupil, most of the iris, most of the sclera, and most of the eyelid are then discarded based on the estimated anatomical properties of a normal eye and optionally the calculated three-dimensional pupil position, leaving only a ring around the potential limbal position. However, due to corneal distortion of the pupil image, additional estimation errors are possible. This distortion results in an increase in the initial estimate of the limbal region.

[0040] In some embodiments, the region detector is configured to identify image data indicating eye features by using machine learning. An iterative pixel filtering process is used to train a neural network to identify the limbal region in each acquired image. The region detector can be configured and operable to identify image data indicating eye features by segmenting each image for identifying pixels associated with the pupil region. Therefore, the neural network can be used to directly segment the approximate limbal region. More particularly, the pupil can be tracked in order to estimate the limbal region. In addition or alternatively, the neural network can also be used to directly estimate the position of other eye features (such as eyelids and sclera) without going through pupil tracking. The neural network can be applied to the image itself or to a mathematically transformed image, i.e., a gradient map. The neural network can be trained on a traditional machine learning algorithm, for example, based on the results of other eye tracking techniques (wherein the eye feature characteristics have been previously defined).

[0041] In some embodiments, the processing unit is configured and operable to determine the size of the limbus.

[0042] In some embodiments, pixels of the identified eyelid region are measured using triangulation to improve the estimate of the position of the eye.

[0043] In some embodiments, the eye tracking device also includes a limbal triangulator configured and operable to receive data indicating a geometric representation of a limbal structure for each image and perform triangulation of the geometric representation of at least two images to determine three-dimensional limbal parameters. The three-dimensional limbal parameters include three coordinates of a center of the limbus and at least one of a direction of a normal to a limbal plane, a position of a limbal plane, and a size of the limbus. The limbal triangulator can be configured and operable to determine a radius of the limbus and / or a torsional rotation of the limbus.

[0044] In some embodiments, the limbus triangulator is configured to perform triangulation between at least two pupil geometry representations to determine three-dimensional pupil parameters, the three-dimensional pupil parameters including the direction of a normal to a pupil plane, coordinates of a center of the pupil, and at least one of a pupil diameter; estimate a location of an initial eye limbus region based on the three-dimensional pupil parameters; and generate data indicating an initial region location of the limbus in each image.

[0045] In some embodiments, the limbus triangulator is configured to process the triangulated three-dimensional limbus, reproject the triangulated limbus region back to the image plane, and refine the limbus region at each image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to better understand the subject matter disclosed herein and to illustrate how it may be carried out in practice, embodiments will now be described, by way of non-limiting examples only, with reference to the accompanying drawings, in which:

[0048] Figure 1 is a schematic block diagram showing the main functional parts of the eye tracking device of the present invention;

[0049] Figure 2 is a schematic flow chart showing the main steps of the eye tracking method of the present invention;

[0050] Figure 3 Two stereo images of the eye captured at different angles are shown;

[0051] Figure 4A-4D is an example of a pupil detection stage according to some embodiments of the present invention;

[0052] Figure 5 According to some embodiments of the present invention Figure 3 An example of the eye feature detection stage illustrated on an image of;

[0053] Figure 6 According to some embodiments of the present invention Figure 3 Example of projection of the limbal circle illustrated on the image;

[0054] Figure 7 is a schematic block diagram illustrating possible functional parts of an eye tracking device according to some embodiments of the present invention;

[0055] Figure 8 represents a two-dimensional eye tracking trajectory obtained by using the teachings of the present invention;

[0056] Fig. 9 Two angular degrees of freedom (i.e., angular measurements) representing the eye tracking orientation trajectory in azimuth and pitch as a function of time; and

[0057] Figure 10A-10B The five degrees of freedom of the eye tracking trajectory obtained by using the teachings of the present invention are shown.

[0058] Detailed description of embodiments

[0059] refer to Figure 1 , which illustrates a block diagram of the main functional parts of the eye tracking device of the present invention. The eye tracking device 100 includes a processing unit 106, which is configured and operable to receive at least two images indicating a user's eye, identify a region associated with the limbus of the eye in each image, determine a geometric representation of the limbus structure, and determine the three-dimensional position and gaze direction of the user's eye by triangulating the geometric representation of the limbus structure of the at least two images. In this regard, it should be noted that, as detailed above, the present invention triangulates shapes (geometric representations) rather than individual corresponding points or edges.

[0060] The processing unit 106 is generally configured as a computing / electronic utility that includes, among other things, utilities such as data input and output modules / utilities 106A and 106B, a memory 106D (i.e., a non-volatile computer-readable medium), and an analyzer / data processing utility 106C. Thus, the utilities of the processing unit 106 may be implemented by suitable circuits and / or by software and / or hardware components, including software and / or hardware components configured to implement Figure 2 1 and 2. Computer readable code for the operations of method 200 shown in and described below.

[0061] Features of the present invention may include general or special computer systems, which include various computer hardware components discussed in more detail below. Features within the scope of the present invention also include computer-readable media, which are used to perform or have computer-executable instructions, computer-readable instructions or data structures stored thereon. Such computer-readable media can be any available medium accessible by a general or special computer system. As an example and not limitation, such computer-readable media may include physical storage media, such as RAM, ROM, EPROM, flash disk, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices or may be used to carry or store desired program code modes in the form of computer-executable instructions, computer-readable instructions or data structures and may be accessible by a general or special computer system. Computer-readable media may include computer programs or computer applications that may be downloaded to a computer system such as a wide area network (WAN), such as the Internet, through a network.

[0062] In this specification and in the claims that follow, a "processing unit" is defined as one or more software modules, one or more hardware modules, or a combination thereof that work together to perform operations on electronic data. For example, the definition of a processing utility includes hardware components of a personal computer and software modules, such as an operating system for a personal computer. The physical layout of the modules is not relevant. A computer system may include one or more computers coupled via a computer network. Likewise, a computer system may include a single physical device in which internal modules (e.g., memory and processor) work together to perform operations on electronic data. Although any computer system may be mobile, the term "mobile computer system" or the term "mobile computer device" as used herein particularly includes laptop computers, netbook computers, cellular phones, smart phones, wireless phones, personal digital assistants, portable computers with touch-sensitive screens, and the like. The processing unit 106 may consist of a processor embedded therein or attached thereto running a computer program. The computer program product may be embodied in one or more computer-readable media having computer-readable program code embodied thereon. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer program code for performing the operations of aspects of the present invention may be written in any combination of one or more programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scheme, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that instructions executed via a processor of a computer or other programmable data processing device create a means (means) for implementing the functions / actions specified in one or more blocks of a flow chart and / or block diagram. The specified functions of the processor may be implemented by a system based on special-purpose hardware that performs the specified functions or actions, or a combination of special-purpose hardware and computer instructions.

[0063] The eye tracking device 100 may include at least two imagers 110, each imager being configured to capture at least one image of a user's eye at a different angle. Each imager may be focused on the user's iris. In this regard, it should be noted that the limbus tracking of the present invention is performed independently for each eye.

[0064] In some embodiments, the processing unit 106 includes a limbal detector 102 that is configured and operable to receive data indicating a limbal region and determine a geometric representation of a limbal structure (e.g., a ring-shaped or elliptical structure) by digital image preprocessing. In a specific and non-limiting example, the limbal detector 102 is configured and operable to perform a limbal identification process on each image, and / or perform a mathematical transformation of the image, including using an intensity gradient map and then running a limbal region identification process on the transformed image. In other words, the limbal detector 102 can transform the image into an intensity gradient map, and then the limbal region is searched on the transformed image. Instead of or in addition to the gradient map, an entropy map can also be used, or the processing can be performed directly on the image. If an intensity gradient map is used, the limbus detector 102 can calculate an image intensity gradient map of the limbus region, identify at least one region of limbus structure where the local direction of the gradient is substantially consistent, process data indicative of limbus structure by weighting the pixels of such region, and determine a geometric representation of the limbus structure based on matching pixels associated with the limbus of the eye. In addition to collinearity, knowledge of the anatomical form of the limbus can be used, i.e., collinear vectors should extend radially from the center of the limbus and the entire limbus region is continuous. More specifically, the region of limbus structure is determined by identifying local consistency, i.e., the gradient direction of each point is only collinear with its neighboring points.

[0065] Each module of the processing unit 106, such as the limbus detector 102, is not limited to a specific number of modules and can be configured to process multiple images independently, simultaneously, or not in this manner. The limbus triangulator 108 is configured and operable to perform triangulation of geometric representations of limbus structures in at least two images to determine three-dimensional limbus parameters. The limbus triangulator 108 can also be configured and operable to determine the radius of the limbus and / or the torsional rotation of the limbus. The three-dimensional limbus parameters define at least five degrees of freedom of the position and orientation of the limbus. The three-dimensional limbus parameters include the position of the plane of the limbus and / or the direction of the normal to the limbus plane and / or the size of the limbus and / or the center of the limbus or any combination thereof.

[0066] In some embodiments, the processing unit 106 includes a region detector 104 that is configured and operable to receive each of at least two images indicating each user's eye, and to identify image data indicating an initial limbal region in each image by using an iterative pixel filtering process and generating data indicating an initial limbal region. This can be achieved by identifying image data indicating eye features such as pupils, eyelids, sclera, irises, and eyelashes and / or identifying an initial limbal region based on anatomical parameters. More specifically, identifying an initial limbal region can include pupil segmentation and pupil triangulation and / or limbal region estimation based on anatomy and / or eyelid region estimation and / or limbal region estimation without an eyelid region. This can be achieved by a limbal triangulator 108 or any other processing module. However, the present invention is not limited to any particular module configuration of the processing unit 106. The processing unit 106 may include a pupil detector, a pupil triangulator, and other modules. For example, the identification of image data indicating eye features may include pupil segmentation. Therefore, the region detector 104 may be a pupil detector intended to provide an initial limbal ring. Assume that these two regions - pupil and limbus - are concentric and in the same plane. Pupil segmentation can be achieved by segmenting each image for identifying pixels associated with the pupil region, performing triangulation between at least two images to determine three-dimensional pupil parameters, estimating the position of the initial eye limbus region based on the three-dimensional pupil parameters, and generating data indicating the initial region position of the limbus in each image. The three-dimensional pupil parameters include the direction of the normal to the pupil plane and / or the coordinates of the center of the pupil and / or the pupil diameter. The three-dimensional pupil parameters are used to estimate the three-dimensional limbus parameters, thereby defining an initial limbus region that can be further refined. The identification of image data indicating eye features can also include the identification of eyelids and filtering out the image data because the image data associated with the eyelids blocks a portion of the limbus ring. The present invention is not limited to a specific way of identifying image data indicating the initial limbus. A neural network-based method can be used to identify image data indicating the initial limbus. For example, limbus region estimation can be performed using a neural network, but a pupil detector can be used to estimate the eyelid region using a classical algorithm. Optionally, limbus region estimation and eyelid region estimation may be performed with a neural network based approach. Optionally, the neural network may be used only for pupil segmentation.

[0067] In some embodiments, the region detector 104 is configured to identify image data indicating eye features by using machine learning. Machine learning is capable of self-classifying / learning characteristics of input data related to eye features. The region detector 104 can use a data recognition model based on a neural network to predict three-dimensional eye limbus parameters. The network can be trained based on the segmentation results of classical methods or by using an existing system for training. In addition or alternatively, the three-dimensional eye limbus parameters obtained by the eye tracking device of the present invention can be used to train the neural network. A deep learning network (DLN) such as an artificial neural network (ANN) implemented by the region detector 104 can generate a representation of the limbus based on a series of images. For example, the representation generated by the DLN can include a probability of placement of the limbus. This representation is used to generate a model that adapts the recognition classifier to the limbus of a person.

[0068] After the limbus detector 102 has determined the three-dimensional limbus parameters, the region detector 104 may receive the three-dimensional limbus parameters to further generate more accurate data indicating the limbus region. For example, the limbus triangulator 108 is configured to process the three-dimensional limbus determined by triangulation, reproject the limbus region determined by triangulation back to the image plane, and refine the limbus region at each image.

[0069] refer to Figure 2 , which illustrates a flow chart of the main steps of the eye tracking method of the present invention. The method 200 for eye tracking includes receiving image data of at least two images indicating an eye of a user in 204; identifying a region associated with the limbus of the eye in each image in 206; determining a geometric representation of the limbus structure in 208; and performing triangulation of the geometric representation of the limbus structure of the at least two images in 210, thereby determining a three-dimensional position and gaze direction of the eye of the user in 212.

[0070] In some embodiments, step 210 includes performing triangulation of geometric representations of limbal structures in at least two images to determine three-dimensional limbal parameters as described above.Determining three-dimensional limbal parameters may include determining a radius of the limbus and / or a torsional rotation of the limbus.

[0071] In some embodiments, after triangulation of the geometric representation of the limbal structure of the at least two images in 210, method 200 may include determining the geometric representation of the limbal structure by processing three-dimensional limbal parameters to generate more accurate data indicative of the limbal region.

[0072] In some embodiments, method 200 may further include capturing at least two images of each user's eyes at different angles at 202 .

[0073] In some embodiments, determining the geometric representation of the limbal structure in 208 may include digital image preprocessing in 218, such as performing a limbal identification process on each image or performing a mathematical transformation of the image, including using an intensity gradient map, and then running a limbal region identification process on the transformed image. The digital image preprocessing step in 218 may include calculating an image intensity gradient map of the limbal region, identifying at least one region of limbal structure (where the local direction of the gradient is substantially consistent), processing data indicative of limbal structure by weighting pixels in such a region, and determining the geometric representation of the limbal structure based on matching pixels associated with the limbus of the eye. More specifically, digital image preprocessing may include determining regions of limbal structure by identifying local consistency (i.e., the direction of the gradient at each point is only collinear with its neighboring points). As described above, the entropy map may also be used as an intensity gradient map (instead of or in addition to the gradient map), or the processing may be performed directly on the image.

[0074] In some embodiments, identifying in each image at 206 an area associated with the limbus of the eye may include identifying image data indicating eye features such as pupils, eyelids, sclera, irises, and eyelashes, and / or identifying an initial limbus region based on anatomical parameters using an iterative pixel filtering process and generating data indicating an initial limbus region. For example, eyelid region estimation may be implemented in parallel with pupil and limbus regions. The present invention is not limited to the order of estimation of different regions. Different eye feature regions may be determined based on any geometric model (e.g., an ellipse or a circle).

[0075] In some embodiments, identifying the initial limbal region may include pupil segmentation and pupil triangulation and / or limbal region estimation based on anatomy and / or eyelid region estimation and / or limbal region estimation without eyelid region. The method may include segmenting each image for identifying pixels associated with the pupil region, performing triangulation between at least two images to determine three-dimensional pupil parameters, estimating the position of the initial eye limbal region based on the three-dimensional pupil parameters, and generating data indicating the initial region position of the limbus in each image. Identifying image data indicating eye features may also include identifying eyelids and filtering out the image data because the image data associated with the eyelids blocks a portion of the limbal ring. As described above, the present invention is not limited to any particular manner of identifying image data indicating the initial limbus. A neural network-based method may be used to identify image data indicating the initial limbus. A neural network-based method may also be used to perform limbal region estimation and eyelid region estimation. A neural network may also be used only for pupil segmentation.

[0076] In some embodiments, identifying image data indicative of eye features can be achieved by using machine learning. Determining (i.e., predicting) three-dimensional eye limbus parameters can include using a data recognition model based on a neural network. The method can include training the network based on the segmentation results of classical methods or by using an existing system for training. A representation of the limbus generated based on the series of images can be obtained by using a deep learning network (DLN) such as an artificial neural network (ANN). For example, generating a representation of the limbus can include calculating a probability of placement of the limbus and / or generating a model of the limbus so that the recognition classifier is adapted to the person.

[0077] refer to Figure 3 , which shows two stereo images of a user's eyes taken at different angles.

[0078] refer to Figure 4A-4D , showing a method for pupil region detection according to some embodiments of the present invention Figure 3 Example of data processing of two stereo images. Figure 4A The segmentation stage is shown, where each image is segmented to identify pixels associated with the pupil region. Pixels not associated with the pupil region are discarded, and the resolution is enhanced, as shown in Figure 4B shown. Figure 4C Segmentation of the pupil boundary based on the intensity map of each image is shown. Figure 4D The pupil boundary refinement is shown based on the projection of a geometric representation of the pupil (e.g., a ring shape) onto each image. The pupil diameter is then determined using triangulation between the image and Figure 4D area and determine the 3D position and orientation.

[0079] refer to Figure 5 , which shows some embodiments according to the present invention Figure 3 Example of data identification of eye features in two stereo images. The marked areas indicate where the local directions of the image intensity gradients are substantially consistent.

[0080] refer to Figure 6 , which shows the elliptic function based on some embodiments of the present invention. Figure 3 Example of the limbus identification process with projections of the two stereo images onto each image.

[0081] refer to Figure 7, which illustrates a block diagram of possible functional parts of an eye tracking device of the present invention. In this specific and non-limiting example, eye tracking device 300 includes two imagers, referred to herein as cam 1 and cam 2, which are configured to capture at least one image of each eye of a user at different angles. Each imager is focused on the iris of the user. Eye tracking device 300 includes processing unit 310, which includes pupil detector 302 and limbus detector 304. Pupil detector 302 is the same as described above with respect to Figure 1 A specific example of a defined region detector. The pupil detector 302 includes a plurality of modules and is configured and operable to receive each of at least two images indicating each eye of a user and identify image data indicating a limbal region in each image and generate data indicating the limbal region. This can be achieved by using eye feature detection such as pupil region detection or by using a neural network. Each image is segmented by a segmentation module to identify pixels associated with the pupil region. Pixels not associated with the pupil region are discarded in each image and the resolution is enhanced. Each ellipse module Ell then calculates an elliptic curve corresponding to the pupil region on each image to find a matching elliptic curve. A three-dimensional equation indicating a three-dimensional pupil region is generated for each image. Then, when the distance between the two imagers capturing the two stereo images is known, a first triangulation of the three-dimensional equations matching the two stereo images is performed to provide the direction of the pupil. Thereafter, using the average size of the limbus and assuming that the pupil and limbus are concentric and on the same plane, the algorithm produces a three-dimensional limbus parameter estimate. This three-dimensional limbus region is projected onto each image and is referred to herein as Figure 3 The mask equation shown in . For example, the limbus region can be bounded by two elliptic curves. The mask equation is used by the collinearity mapping module which gives a first estimate of the limbus projection on each image. The mask equation is also used by the weight module.

[0082] In order to determine the three-dimensional limbus parameters, the collinearity module referred to herein as collinearity mapping performs a mathematical transformation. In a specific and non-limiting example, such a mathematical transformation may include calculating an image intensity gradient map of the limbus based on the pupil direction, identifying at least one region of the limbus structure (wherein the direction of the gradient is substantially consistent), processing data indicating the limbus structure by weighting the pixels of such a region, and determining the three-dimensional limbus parameters based on matching pixels related to the limbus of the eye. Optionally, such a mathematical transformation may include using the entropy of the image. In a specific and non-limiting example, the collinearity mapping module generates a gradient collinearity mapping from the original image masked by the estimated limbus region (calculated from pupil 3D parameters and average limbus size). The limbus detector 304 receives a three-dimensional equation indicating the direction of the three-dimensional limbus region and the limbus. The annular region is then projected onto each eye image to produce the estimated limbus region. Then, most pixels of the image are discarded by using mask equations based on the estimated anatomical properties of a normal eye, such as the pupil, most of the iris, most of the sclera, and most of the eyelid, leaving only a ring around the potential limbus location.

[0083] The module weights of the collinearity mapping module are used to assign a weight to each pixel based on the probability that the pixel belongs to the limbus region. The module weights are then used to refine the potential limbus region and provide input to the Ell module that finds a matching ellipse estimate of the limbus.

[0084] In order to perform the second triangulation, several techniques can be implemented, such as the triangulation of cones or quadratic estimation (linear or nonlinear). The processing unit 310 is configured and operable to determine the three-dimensional position and gaze direction of the user's eyes by triangulating the geometric representation of the limbus structure of at least two images by means of using linear or nonlinear quadratic estimation. In this specific and non-limiting example, the triangulation of the cone can be performed as follows: each ellipse module Ell generates a mathematical function / equation of the geometric representation of the limbus of the anatomical relationship between the identified virtual areas corresponding to certain eye features (e.g., an ellipse with a center corresponding to the center of the pupil), and the limbus is calculated for each image. The ellipse structure of each image is compared, and the intersection between the cones defined by the ellipse structure defines an ellipse in space, and its projection matches the ellipse structure of the two images. This definition is referred to as the limbus R, Cx, Cy, Cz, azimuth and pitch three-dimensional limbus parameters herein. In this regard, it should be noted that the three-dimensional limbus parameters are obtained by triangulation of the ellipse rather than by pixel triangulation in the present invention. As described above, pixel triangulation of the limbus is virtually impossible because the correspondence between pixels of different images is unknown (because most pixels along the epipolar line of the second image look too similar). In classical computer vision, special near-pixel region features are used, such as different colors, corners, line intersections, or texture patches, which allow characterization of the match between two pixels along the epipolar line from different images. Due to the rough border of the limbus, which is usually defined as a wide area, such features are missing in the limbus region, thus not allowing the use of classical natural feature tracking algorithms. A mask is then used to discard areas associated with eye features. The same process as described above is then repeated using an iterative pixel filtering process to more accurately define the limbus region and reduce the limbus region size.

[0085] Alternatively or in addition, the iterative pixel filtering process can be implemented using a linear quadratic estimate such as a Kalman filter or a nonlinear quadratic estimate such as an extended Kalman filter and an unscented Kalman filter. The Kalman filter produces an estimate of the limbal equation with five degrees of freedom (e.g., an elliptical structure) as an average of the predicted state of the eye and the average of new measurements using a weighted average. The weights are calculated from the covariance, a measure of the estimated uncertainty in the prediction of the state of the eye. The result of the weighted average is a new state estimate that is between the predicted state and the measured state and has a better estimated uncertainty than either state taken individually. This process is repeated at each time step, with the new estimate and its covariance providing information for the prediction used in the next iteration.

[0086] At the end of the process, processing unit 310 determines the three-dimensional position and gaze direction of the user's eyes based on the three-dimensional limbus parameters.

[0087] refer to Figure 8 , which illustrates a two-dimensional eye tracking position trajectory obtained by using the teachings of the present invention, where different colors distinguish different parts of the trajectory, ie, the color changes over time. The axes Xc and Yc are measured in millimeters.

[0088] refer to Fig. 9 , which illustrates the azimuth and pitch as a function of time (in milliseconds) Figure 8 The eye tracking orientation trajectories have two angular degrees of freedom (i.e., angular measurements measured in degrees). Figure 8 , 9 10 and 11 show data from the same record.

[0089] refer to Figures 10A-10B , which exemplifies Figure 8 The five degrees of freedom of the eye tracking trajectory. More specifically, Fig. 10A The two angular degrees of freedom (ie, angular measurements measured in degrees) of the eye tracking orientation trajectory in azimuth and pitch are shown as a function of time. Fig. 10A It's above Fig. 9 An enlarged view of a portion of a graph of FIG. Fig. 10B The three-dimensional degrees of freedom of the eye tracking position trajectory in Cartesian coordinate space are shown as a function of time (measured in milliseconds).

Claims

1. An eye tracking device, the eye tracking device comprising: a processing unit configured and operable to receive and process at least two images indicative of an eye of a user by performing the following operations: identifying in each image an area associated with a limbus of the eye, the limbus of the eye being defined as a transition zone between a cornea and a sclera of the user; applying digital image pre-processing to the area associated with the limbus of the eye and determining a geometric representation of the limbus structure; and determining the three-dimensional position and orientation of the user's eye by triangulation of the geometric representation of the limbal structure of at least two images.

2. The eye tracking device according to claim 1, wherein: The geometric representation of the limbal structure comprises a ring-shaped or elliptical structure.

3. The eye tracking device according to claim 1 or 2, further comprising at least two imagers, each imager being configured to capture at least one image of the user's eye at a different angle.

4. The eye tracking device according to claim 1 or 2, wherein: The processing unit includes a limbus detector configured and operable to receive data indicative of an area associated with the limbus of the eye and to perform the digital image pre-processing to determine the geometric representation of the limbus structure, the digital image pre-processing being configured to calculate some virtual areas corresponding to physical areas of the eye independent of the angle at which the at least two images are taken.

5. The eye tracking device according to claim 4, wherein: For each image, the limbus detector is configured and operable for digital image preprocessing, which includes: calculating an image intensity gradient map of the limbus region; identifying at least one region of the limbus structure in which the local direction of the gradient is substantially consistent; processing data indicative of the limbus structure by weighting pixels of such regions; and generating a geometric representation of the limbus structure based on matching pixels associated with the limbus of the eye.

6. The eye tracking device of claim 4, further comprising a limbal triangulator configured and operable to receive data indicating a geometric representation of the limbal structure for each image, and to perform triangulation of the geometric representation of at least two images to determine three-dimensional limbal parameters.

7. The eye tracking device according to claim 6, wherein: The processing unit is configured and operable to determine a three-dimensional position and orientation of the user's eye based on the three-dimensional limbus parameters.

8. The eye tracking device according to claim 6, wherein: The three-dimensional limbus parameters include at least one of three coordinates of the center of the limbus, the direction of the normal to the limbus plane, the position of the limbus plane, and the size of the limbus.

9. The eye tracking device according to claim 6, wherein: The limbal triangulator is configured and operable to determine a radius of the limbus and / or a torsional rotation of the limbus.

10. The eye tracking device according to claim 1 or 2, wherein: The processing unit includes a region detector configured and operable to receive each of the at least two images indicating an eye of the user, and to identify image data indicating an initial limbal region in each image by using an iterative pixel filtering process and generating data indicating the initial limbal region.

11. The eye tracking device according to claim 10, wherein: The region detector is configured to identify image data indicating an initial limbal region by at least one of: identifying image data indicating eye features including at least one of a pupil, an eyelid, a sclera, an iris, and an eyelash, and identifying the initial limbal region based on anatomical parameters.

12. The eye tracking device according to claim 11, wherein: The region detector is configured to identify image data indicative of eye features by using machine learning.

13. The eye tracking device according to claim 11, wherein: The region detector is configured and operable to identify the image data indicative of eye features by segmenting each image to identify pixels associated with a pupil region.

14. An eye tracking device according to claim 13, comprising a limbal triangulation device, the limbal triangulation device being configured and operable to receive data indicative of the geometric representation of the limbal structure for each image and to perform triangulation of the geometric representation of at least two images to determine three-dimensional limbal parameters, wherein The limbal triangulator is configured to perform triangulation between at least two pupil geometry representations to determine three-dimensional pupil parameters, the three-dimensional pupil parameters comprising at least one of a direction of a normal to a pupil plane, a coordinate of a center of the pupil, and a pupil diameter; estimating a position of the initial limbal region based on the three-dimensional pupil parameters; and generating data indicating the location of the initial limbal region in each image.

15. The eye tracking device of claim 6, wherein: The limbus triangulator is configured to process the triangulated three-dimensional limbus, reproject the triangulated limbus region back to the image plane, and refine the limbus region at each image.

16. An eye tracking method, the method comprising: receiving image data indicative of at least two images of eyes of a user; identifying in each image an area associated with a limbus of the eye, the limbus of the eye being defined as a transition area between a cornea and a sclera of the user; determining a geometric representation of limbal structure by applying digital image pre-processing to data indicative of an area associated with a limbus of said eye; A triangulation of the geometric representation of the limbal structure of at least two images is performed to determine a three-dimensional position and orientation of the user's eye.

17. The method of claim 16, comprising capturing at least two images of the user's eyes at different angles.

18. The method according to claim 16 or 17, wherein: The digital image pre-processing is configured to calculate some virtual areas corresponding to physical areas of the eye independently of the angle at which the at least two images are taken.

19. The method according to claim 18, wherein: The digital image preprocessing includes: calculating an image intensity gradient map of the corneal limbus based on pupil direction; identifying at least one region of the corneal limbus structure in each image, in which the local direction of the gradient is substantially consistent; processing data indicative of the corneal limbus structure by weighting pixels of such region; and generating the geometric representation of the corneal limbus structure based on matching pixels associated with the corneal limbus of the eye.

20. The method of claim 19, comprising determining three-dimensional limbus parameters including three coordinates of a center of the limbus and at least one of a direction of the limbus, and determining a three-dimensional position and orientation of a user's eye based on the three-dimensional limbus parameters.

21. The method according to claim 20, wherein: The determination of the three-dimensional position and orientation of the user's eye includes processing of the three-dimensional limbal parameters.

22. The method of claim 16 or 17, further comprising determining a radius of the limbus and / or a torsional rotation of the limbus.

23. The method according to claim 16 or 17, wherein: The geometric representation of the limbal structure comprises a ring-shaped or elliptical structure.

24. The method of claim 16 or 17, further comprising identifying image data indicative of an initial limbal region in each image by using an iterative pixel filtering process and generating data indicative of the initial limbal region.

25. The method according to claim 24, wherein: The identifying image data indicating the initial limbal region in each image includes at least one of the following operations: identifying image data indicating eye features including at least one of a pupil, an eyelid, a sclera, an iris, and eyelashes, and identifying the initial limbal region based on anatomical parameters.

26. The method according to claim 25, wherein: The identifying of image data indicative of eye features is performed by using machine learning.

27. The method according to claim 25, wherein: Identifying image data indicative of eye features includes segmenting each image to identify pixels associated with a pupil region.

28. The method of claim 27, further comprising performing triangulation between at least two pupil geometry representations to determine three-dimensional pupil parameters, the three-dimensional pupil parameters comprising at least one of a direction of a normal to a pupil plane in each image, a coordinate of a center of the pupil, and a pupil diameter; estimating a position of the initial limbal region based on the three-dimensional pupil parameters; And generate data indicating the initial area position of the corneal limbus in each image.

29. The method of claim 16 or 17, further comprising processing the three-dimensional corneal limbus determined by triangulation, reprojecting the corneal limbus region determined by triangulation back to the image plane, and refining the corneal limbus region at each image.

Images