User eye model match detection

Abstract

Methods and apparatus for providing eye model matching in a device are disclosed. When a user activates a device and a presence of an eye of the user is detected, an image of the eye of the user is captured. An eye model matching process is then implemented to determine a stored eye model that best matches the eye in the captured image (e.g., a stored eye model after registration of the eye on the device). The determination of the best matching eye model may be based on a match between characteristics of the user's eye, such as corneal and pupil characteristics, in the captured image and characteristics of the user's eye as determined by the eye model. A best-match eye model may then be implemented, for example, in an eye gaze tracking process. In some cases, the best matching eye model satisfies a threshold of the match prior to implementation in a downstream process.

Description

Background Art Technical Field

[0001] Embodiments described herein relate to eye modeling systems. More specifically, embodiments described herein relate to systems and methods for matching eye models based on captured images of a user's eyes.

[0002] Description of the Prior Art

[0003] Virtual reality (VR) allows users to experience and / or interact with an immersive artificial environment so that users feel as if they are in the environment. For example, a virtual reality system may display a stereoscopic scene to a user to create an illusion of depth, and a computer may adjust the scene content in real time to provide the illusion that the user is moving within the scene. When a user views an image through a virtual reality system, the user may therefore feel as if they are moving within the scene from a first-person perspective. Similarly, mixed reality (MR) combines computer-generated information (referred to as virtual content) with real-world images or real-world views to enhance or add content to the user's world view. Therefore, the simulated environment of VR and / or the mixed environment of MR can be used to provide an interactive user experience for multiple applications, such as an application that adds virtual content to a real-time view of the viewer's environment, an application that interacts with a virtual training environment, a game application, an application that remotely controls a drone or other mechanical system, an application that views digital media content, an application that interacts with the Internet, and the like.

[0004] An eye or gaze tracker is a device used to estimate eye positioning and eye movement. Eye or gaze tracking processes and systems have been used in the study of visual systems, psychology, psycholinguistics, marketing, and as an input device for human-computer interaction. In the latter application, the intersection of a person's gaze point and a desktop monitor is typically considered. An eye model is a mathematical representation of the human eye that can be used in processes and systems including, but not limited to, gaze tracking processes and systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a flowchart of the workflow for implementing the eye model matching process in a device according to some embodiments.

[0006] Figure 2 is a flowchart of an eye model matching process in a device according to some embodiments.

[0007] Figure 3 is a flowchart of a process for determining a best matching eye model according to some embodiments.

[0008] Figure 4 Example graphs depicting matching scores (y-axis) versus time (x-axis) for various eye models.

[0009] FIG. 5A to FIG. 5F An example eye camera system is illustrated according to some embodiments.

[0010] Figure 6 is a high-level flow chart of a registration method according to some embodiments.

[0011] Fig. 7A and Figure 7B is an example of a method that may include, according to some embodiments, Figures 1 to 6 Block diagrams of the illustrated components and apparatus implementing the methods as illustrated in these Figures.

[0012] Figure 8 An example head mounted device (HMD) according to some embodiments is illustrated, which example HMD may include Figures 1 to 6 The components illustrated in the figures and implement the methods illustrated in these figures.

[0013] Fig. 9 is a block diagram illustrating an example system according to some embodiments, which example system may include Figures 1 to 6 The components illustrated in the figures and implement the methods illustrated in these figures.

[0014] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0015] The term "comprising" is open ended. As used in the claims, the term does not exclude additional structures or steps. Consider the following cited claim: "an apparatus comprising one or more processor units..." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, a graphics circuit, etc.).

[0016] "Configured to", various units, circuits or other components may be described or described as "configured to" perform one or more tasks. In such contexts, "configured to" is used to imply a structure (e.g., a circuit) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes the structure. In this way, the unit / circuit / component is said to be configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not turned on). The units / circuits / components used with the "configured to" language include hardware-such as circuits, memories storing executable program instructions to implement operations, etc. Reference to a unit / circuit / component "configured to" perform one or more tasks is explicitly intended not to invoke the sixth paragraph of 35 U.S.C. §112 for the unit / circuit / component. In addition, "configured to" may include a general structure (e.g., a general circuit) manipulated by software or firmware (e.g., an FPGA or a general processor executing software) to operate in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0017] "First," "second," etc. As used herein, these terms act as labels for the nouns that precede them, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation of a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0018] "Based on" or "depending on", as used herein, these terms are used to describe one or more factors that influence a determination. These terms do not exclude additional factors that may influence the determination. That is, the determination may be based solely on these factors or at least in part on these factors. Consider the phrase "A is determined based on B." In this case, B is a factor that influences the determination of A, and such phrases do not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B.

[0019] "Or", when used in the claims, the term "or" is used as an inclusive or rather than an exclusive or. For example, the phrase "at least one of x, y or z" means any one of x, y and z and any combination thereof. DETAILED DESCRIPTION

[0020] Various embodiments of methods and devices for matching a user's eyes with an eye model stored in a device are described. In some embodiments, an eye model is generated during registration of a user's eyes on a device such as a head mounted device (HMD). The imaging system described herein may include: two or more illumination sources (e.g., point light sources, such as light emitting diodes (LEDs)), which illuminate a person's eyes or eye regions; and at least one camera, which is configured to capture images of light reflected from the illumination sources when the eyes are illuminated. During the registration process, the eye images captured when the eyes are in two or more different orientations and with two or more different levels of display brightness may be processed by a controller to generate an eye model. The eye model may include, for example, models of corneal and pupil features of the eye. In some embodiments, the registration process may be performed as an initial process when the user uses the device (e.g., when the user places the HMD on their head). Alternatively, the registration process may be a continuous or iterative process that builds and improves the eye model while the user is using the device.

[0021] Embodiments of methods and apparatus for user eye registration and user eye model matching as described herein may be used, for example, in a head mounted device (HMD), such as an HMD of a computer generated reality (XR) system, such as a mixed or augmented reality (MR) system or a virtual reality (VR) system. In conventional systems, a general human eye model is typically used as a basis. However, in various disclosed embodiments, a user-specific eye model (which may be referred to as a user-perceived eye model) is constructed from images of the user's eyes captured during the registration process and is used in place of the general human eye model. Thus, after registration, when the user activates the device (e.g., when the user's eyes are located and detected in the device, such as when the HMD is placed on the user's head), eye model matching may be achieved.

[0022] In various embodiments, eye model matching may be implemented for a single user using a device in different situations (e.g., with or without glasses, with different glasses, with or without contact lenses, with or without clip-on glasses, etc.). In some embodiments, a user may register for each of the different situations, and eye model matching may be implemented to automatically select the eye model that best fits a particular situation when the user activates the device. In some cases, a user may attempt to operate the device in a new, unregistered situation. In this case, eye model matching may determine whether the registered eye model is appropriate for use of the device or whether the user needs to register to adapt to the new situation. Additional embodiments are envisioned where eye model matching is used to distinguish different users of a single device when there are detectable differences between the various user eye models.

[0023] In some embodiments, eye model matching is implemented using images of the user's eyes captured under conditions similar to those implemented during eye registration. For example, a user may wear a device (e.g., an HMD) and stimulation or guidance may be displayed on a display to cause the user to look in different directions. The brightness or other display characteristics of the display may be adjusted to stimulate different pupil responses. Images of the user's eyes may be captured by one or more eye cameras in two or more different orientations (postures) and at two or more brightness levels. Based on these images, the system determines features of the user's eyes. Examples of features of the user's eyes that can be determined include, but are not limited to, corneal contour features and pupil features. User eye model matching may be performed for one eye or for both eyes.

[0024] In various cases, eye model matching is implemented in combined gaze tracking. A gaze tracking system can, for example, be used to calculate a gaze direction and visual axis using glint and eye features based on a three-dimensional (3D) geometric model of the eye. Eye model matching can be implemented before gaze tracking to select an eye model to be used in the gaze tracking process. In some cases, when the device is activated by the user, eye model matching selects the eye model that best matches the user's eye for the gaze tracking process. Therefore, implementing eye model matching to select the eye model for gaze tracking when the user activates the device can ensure that the quality and accuracy of the gaze tracking process is maintained between uses of the device. In addition, implementing eye model matching reduces the need for the user to register his / her eyes before each use. Therefore, the user can obtain a more satisfactory experience using the device. The disclosed embodiments of the eye model matching process can be used in any additional application or system that uses a model of the user's eye other than those described herein.

[0025] Figure 1 1 is a flowchart of a workflow for implementing an eye model matching process in a device according to some embodiments. Workflow 100 can be implemented for any of the various embodiments of a device (e.g., an HMD) that captures an image of a user's eye as described herein, but it should be understood that the eye model matching process 100 is not limited to the disclosed embodiments. In 102, the presence of a user's eye is detected in a device (such as device 3000, HMD device 4000, or HMD device 500 described herein) having at least one camera pointed at the user's eye. For example, the presence of the user's eye can be detected when the user activates the device and then the camera captures at least one image of the user's eye.

[0026] In response to detecting the presence of an eye in 102, an eye model matching process is implemented in 104. One embodiment of the eye model matching process implemented is process 200, as shown in FIG. Figure 2As shown and described below. In various embodiments, the eye model matching process is implemented without the user noticing that the process is implemented. For example, the eye model matching process can be implemented during the initial startup of the device when the user activates the device and the user's eyes are detected.

[0027] The eye model matching process implemented in 104 may determine whether a good match for the user's eyes is found, and in 106, a decision may be made based on the match determination. If a good match is found in 106, the eye model determined by the eye model matching process in 104 may be implemented in a downstream process on the device in 108. For example, matching the eye model may be implemented in a gaze tracking process as described herein, or in any other process that implements an eye model. When a good match is not found in 106, then in 110 the device may request or otherwise trigger the user to re-register the user's eyes (e.g., using a registration process as described herein). In some embodiments, a good match is determined based on an evaluation of the eye model against a threshold, as described below.

[0028] Figure 2 2 is a flow chart of an eye model matching process in a device according to some embodiments. In some embodiments, the eye model matching process 200 is implemented in response to detecting the presence of a user's eye in the device (e.g., Figure 1 Additional embodiments are contemplated in which process 200 is implemented in response to other actions. For example, process 200 may be implemented in response to a user prompt on a device or when a downstream process (e.g., a gaze tracking process) is not operating correctly.

[0029] In various embodiments, process 200 begins by capturing an image of the user's eye in 202. As described herein, the image may be captured by a camera positioned to capture an image of the user's eye when the user wears or operates the device. In some embodiments, multiple images of the user's eye may be captured. For example, multiple images may be captured, and image data from the images may be combined to increase the signal-to-noise ratio. In some embodiments, multiple images are acquired by multiple cameras. For example, two cameras may capture images of the user's eye at substantially the same time. When two cameras are used, the images may have different perspectives (e.g., angles) of the eye, and therefore, the combined image may provide a stereoscopic view of the eye and provide a 3-dimensional assessment of the eye.

[0030] After the image is captured in 202, data in the captured image may be analyzed in 204 to determine eye characteristics of the captured image. For example, the captured image may be analyzed by a controller or other processor on the device to detect characteristics of the user's eye in the captured image. In some embodiments, the eye characteristics determined from the captured image include features of the eye in the captured image. The eye characteristics may include, for example, corneal features (such as corneal contours) and / or pupil features. The eye characteristics may also include, but are not limited to, other characteristics, such as positioning of the eye, movement of the eye (as detected between multiple images), or pupil dilation. In various embodiments, the eye characteristics determined from the captured image are features found in 3-dimensional (3D) space of the user's eye.

[0031] At 206, the eye characteristics determined from the captured image are compared with the eye characteristics determined from the eye models in the eye model set 208 to determine the eye model that provides the best match to the captured image. In some embodiments, the eye model set 208 includes eye models determined by the user eye model registration method described herein. For example, the eye model set 208 may include multiple eye models generated during registration of the user's eyes on the device.

[0032] At 206, eye characteristics from the captured image, such as corneal and pupil features, are compared to eye characteristics determined from a particular eye model to determine a difference between the eye characteristics in the captured image and the eye characteristics determined from the eye model. The comparison may be performed on each individual eye model in the eye model set 208 or on a selected number of individual eye models from the set. In some embodiments, the eye model that provides the best match is the eye model that provides the smallest difference to the eye characteristics in the captured image.

[0033] Figure 3 is a flow chart of a process for determining a best matching eye model according to some embodiments. In some embodiments, process 300 Figure 2 The process 200 is implemented in 206 of the illustrated process 200. At 310, the eye characteristics of each eye model are compared with the eye characteristics in the captured image to determine a match score for each eye model. The match score may then be provided to a match score storage 320 (which may be in a memory of the device). The determination of the match score is repeated for each of the eye models in the set 208 until the match score for each eye model is stored in the match score storage 320.

[0034] In some embodiments, the match score of the eye model is determined based on the residual error determined for the eye model. The residual error is a measure of the difference between the eye characteristics determined from the eye model and the eye characteristics in the captured image. For example, the residual error can be a measure of the difference between the position of the eye feature predicted by the eye model and the position of the eye feature actually in the captured image. In some embodiments, the residual error is the total amount of error for each eye characteristic in the eye characteristics evaluated in the comparison (e.g., the total error of the cornea and pupil features). The larger the residual error, the greater the difference between the eye model and the captured image. The residual error is also a measure of the confidence that the eye model matches the captured image, where a smaller residual error provides a higher confidence. Therefore, when there is a higher confidence in a match, the smaller the residual error is and the higher the match score is.

[0035] In some embodiments, the best matching eye model 210 is the eye model having the highest matching score in the matching score storage 320 after all individual eye models have been evaluated. Accordingly, the best matching eye model 210 determined in the process 300 is the eye model having the smallest amount of residual error between the eye characteristics determined by the eye model and the eye characteristics in the captured image. Figure 4 An example graph of matching scores (y-axis) versus time (x-axis) for various eye models is depicted. The example includes a first eye model 410, a second eye model 420, and a third eye model 430. Figure 4 , the lower the residual error, the higher the match score (e.g., the higher the confidence that the eye model matches the eye in the captured image). As shown, the first eye model 410 provides the highest match score, the second eye model 420 provides the second highest match score, and the third eye model 430 has the lowest match score. Therefore, the first eye model 410 can then be determined to be the best matching eye model 210, as shown in FIG. Figure 2 and Figure 3 shown.

[0036] Back to Figure 2 In some embodiments, the best matching eye model 210 determined in 206 is compared to a threshold to determine whether the best matching eye model 210 is a good match. For example, as described above, the residual error is a measure of the confidence that the eye model matches the captured image. Therefore, the threshold can be set to a specified error level corresponding to a specified confidence that the eye model matches the captured image. Alternatively, the threshold can be set to a specified match score corresponding to a specified confidence. An example of an achievable threshold (threshold 440) is Figure 4Determining a specified confidence level for a threshold may, for example, provide suitable user satisfaction in the performance of the device (e.g., suitable performance in eye gaze tracking on the device). The threshold may be determined from statistical studies or other analysis of device operation.

[0037] like Figure 2 As shown, when the best matching eye model 210 meets the threshold, then the best matching eye model 210 can be selected at 108 ( Figure 1 In contrast, when the best matching eye model 210 does not meet the threshold, the user may be asked to re-register the eye in 110 (e.g., Figure 1 As described above, the best matching eye model 210 provides the least amount of residual error between the eye model and the captured image. Therefore, when the best matching eye model 210 does not meet the threshold, none of the eye models in the eye model set 208 meets the threshold. As another example, turning again to Figure 4 , the first eye model 410 satisfies the threshold 440 (e.g., the matching score of the first eye model is higher than the threshold). Figure 4 In the example of , the third eye model 430 is the only eye model that does not meet the threshold 440. Therefore, if the third eye model 430 is the only eye model or the "best" matching eye model, the best eye matching model will not meet the threshold.

[0038] As can be appreciated, Figures 1 to 4 The above-described embodiments are implemented for captured images to provide eye model matching for one eye of a user. As described herein, the registration and generation of eye models are typically implemented independently (e.g., the left eye and right eye are registered independently, with each eye having a different eye model). Therefore, in various embodiments, the eye model matching described herein is implemented independently for each eye of the user. For example, capturing an image of the left eye and matching the eye model of the left eye with the captured image of the left eye is independent of capturing an image of the right eye and matching the eye model of the right eye with the captured image of the right eye. The eye matching process for the left eye and the right eye can be implemented substantially simultaneously (e.g., when the user activates the device and the device detects the presence of the eye).

[0039] Example Device

[0040] A non-limiting example application of the method and apparatus for registration is in a device including an imaging system having at least one eye camera (e.g., an infrared (IR) camera) located on each side of a user's face, and an illumination source (e.g., an array or ring of point light sources such as IR light emitting diodes (LEDs)) that emits light to the user's eyes. The imaging system may, for example, be a component of a head mounted device (HMD), for example, an HMD of a computer generated reality (XR) system such as a mixed or augmented reality (MR) system or a virtual reality (VR) system. The HMD may, for example, be implemented as a pair of glasses, goggles, or a helmet. Other example applications of the imaging system include mobile devices such as smart phones, tablets or tablet devices, desktop computers and notebook computers, and stand-alone systems mounted on a wall or otherwise located in a room or on a building. In any of these example systems, the imaging system may be used for gaze tracking, biometric authentication, both, or for other applications.

[0041] FIG. 5A to FIG. 5F An example imaging system according to some embodiments is illustrated. The imaging system may include, but is not limited to, one or more cameras 540, an illumination source 530, and a controller 560. Figure 5A 5 shows an imaging system in which the eye camera 540 directly images the eye 592. However, in some embodiments, the eye camera 540 may instead image the reflection of the eye 592 off the hot mirror 550, such as Figure 5B In addition, in some embodiments, the eye camera 540 can image the eye through the lens 520 of the imaging system, for example, Figure 5C shown.

[0042] In some embodiments, a device (e.g., a head mounted device (HMD)) may include an imaging system comprising at least one eye camera 540 (e.g., a visible light and / or infrared (IR) camera) positioned on one or each side of the user's face, and an illumination source 530 (e.g., a point light source such as an array or ring of IR light emitting diodes (LEDs)) that emits light toward the user's eye 592 or periorbital area. Figure 5D An example illumination source 530 is shown that includes a plurality of LEDs 532. In this example, there are eight LEDs 532 arranged in a ring. However, it should be noted that the number and arrangement of LEDs 532 in the illumination source 530 can be different. Furthermore, in some embodiments, other light emitting elements besides LEDs can be used. In some embodiments, the LEDs 532 can be configured to emit light in the IR or NIR range, for example at 740, 750, 840, 850, 940, or 950 nanometers.

[0043] In an embodiment, the eye camera 540 may be directed toward the eye 592 to receive light reflected from the eye 592 from the illumination source 530, such as Figure 5A However, in some embodiments, the eye camera 540 may instead image the reflection of the eye 592 off the thermal mirror 550, as shown. Figure 5B In addition, in some embodiments, the eye camera 540 can image the eye 592 through the lens 520 or other optical elements of the device, such as Figure 5C Some embodiments may include a single camera 540 that captures an image of a user's eye 592, such as Figure 5E Some embodiments may include two or more cameras 540 that capture images of a user's eyes 592, such as Fig. 5F shown.

[0044] include FIG. 5A to FIG. 5F The device of the imaging system shown may include a controller 560 that includes one or more processors and memory. The controller 560 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), and / or other components for processing and rendering video and / or images. In some embodiments, the controller 560 may be integrated into the device. In some embodiments, at least some of the functions of the controller 560 may be implemented by an external device coupled to the device via a wired or wireless connection. Although in FIG. 5A to FIG. 5F Not shown, but in some embodiments, controller 560 may be coupled to an external memory for storing and reading data and / or software.

[0045] The controller 560 may transmit control signals to the illumination source 530 and the camera 540 to control the illumination of the eye 592 and to capture images of the eye 592. The controller 560 may use the image 542 of the eye 592 captured by the eye camera 540 to build or adjust a 3D model of the eye 592 during the registration process. The eye model may then be used for one or more purposes. For example, the controller 560 may implement a gaze tracking algorithm that estimates the user's gaze direction based on the additional image 542 captured by the camera and the eye model generated during the registration process. The gaze tracking algorithm may, for example, process the image 542 captured by the camera 540 to identify the flash (reflection of the LED 530), pupil location and diameter, or other features of the eye obtained from the eye camera 540, and apply this information to the eye model to determine and track the direction the user is currently looking at (gaze direction).

[0046] Example Registration Process

[0047] An implementation scheme of a user eye model registration method and a system for implementing the registration method is described. Figure 6 6 is a high-level flow chart of a registration method according to some embodiments. In an embodiment, during the registration process, one or more cameras capture images of the user's eyes when the eyes are in two or more different orientations and at two or more different levels of display brightness, as indicated at 600. As indicated at 610, the captured images are processed by a controller to generate a 3-dimensional user-perceived eye model, such as a model of at least the cornea and pupil features of the eye. As indicated at 620, the generated user-perceived eye model can be used in other processes, such as in a gaze tracking process. The embodiment can be used to generate an eye model for one eye or for two eyes. As indicated by the arrow returning from 610 to 600, in some embodiments, the registration process can be an iterative process or a continuous process performed when the user is using the system (e.g., HMD). It should be noted that the generated eye model can be stored and used in the future, such as when a particular user wears or uses the device again. In addition, separate user-perceived eye models can be generated for two or more different users of the device.

[0048] In building a user-perceived eye model, an embodiment of the registration process may perform a physiologically accurate reconstruction of the corneal surface and may also perform dynamic pupil modeling. The pupil moves with multiple degrees of freedom as it contracts and dilates. Dynamic pupil modeling captures and processes the user's pupil across those degrees of freedom. The registration process adjusts brightness to stimulate changes in pupil diameter. Thus, a mathematical representation of the corneal surface and a dynamic model of the pupil are captured in the user-perceived eye model generated by the registration process. By reconstructing an accurate representation of the corneal surface as well as the dynamic pupil behavior of a specific user to generate a user-perceived eye model, the registration process may improve the accuracy of processes that use eye models, such as gaze tracking, compared to conventional systems that use a general "universal" eye model.

[0049] An embodiment of the registration process may prompt the user to look in different directions when capturing images of the eye. From each captured image, a set of features may be derived, such as the lens and pupil outlines. Prompting the user to look in different directions may provide good coverage of the eye outline. The pupil may move in many degrees of freedom, and thus the display brightness may be changed, so that the process may observe those states and regress to generate a pupil model.

[0050] The user-perceived eye model may be generated during a registration process of a device, such as a head mounted device (HMD). In some embodiments, the registration process may be performed as an initial process while the user is using the device, for example, when the user places the HMD on their head. Alternatively, the registration process may be a continuous or iterative process that builds and improves the eye model while the user is using the device.

[0051] Embodiments of an eye model registration method for generating a user eye model using two different methods are described. In the first method, an initial eye model is input, which can be a general eye model. The user is prompted to look at a target location (corresponding to a gesture), where images of the eye are captured, in some embodiments, at different levels of display brightness. The method then seeks the best eye center so that the features constructed or calculated by the gesture match the features collected by the corresponding image. An updated eye model is constructed. The updated eye model can be tested, and if it is not satisfactory (e.g., if the error rate is above a threshold), the updated eye model can be input to the beginning of the method for optimization. In the second method, the initial eye model is not used. Instead, multiple iterations of capturing and processing images at different locations (and at different levels of display brightness) are performed. In the first iteration, the eye center is estimated from a set of captured eye gestures, and an initial user-perceived eye model is constructed. The eye model is then used, for example, in a gaze tracking process to test the eye model. If the eye model is not satisfactory, for example, if the error rate of the process is above a threshold when the eye model is used in the process, another iteration of capturing and processing eye gestures is performed to optimize the eye model. These iterations can continue until the error rate is below the threshold.

[0052] The embodiment captures an image of the user's eye. Based on the current version of the eye model, corneal features and pupil positioning are calculated. Those calculated features are then compared to features that can actually be derived from the captured eye image to determine how far the model deviates; the model can then be updated based on this comparison.

[0053] In an embodiment, the features being tracked may be determined using a combination of information obtained from images captured by the eye camera and an existing eye model to compute the eye pose. The eye pose that gives the best match for the feature is selected. The eye model parameters are fixed and then the eye pose is estimated (e.g., with 5 DoF).

[0054] Eye pose is used to align features with the local system. The goal is to generate a robust eye model given the local system. Because features in the world system are observed in the eye image, those features need to be aligned with the local system. Once the eye pose is obtained, eye instances from different gaze directions can be aligned. From this, the corneal surface and pupil models can be optimized.

[0055] Eye pose refers to the position and orientation of the eye. Whereas, the eye model indicates what features of the eye (e.g. corneal contour and pupil shape) would be seen from the perspective of the eye camera at a given pose.

[0056] The eye camera system may include active illumination (e.g., from FIG. 5A to FIG. 5FThe active illumination is performed by an LED (shown), from which the reflection on the corneal surface can be observed. The refracted pupil contour can also be observed. The light reflection (observation of the point where the light reflects on the eye surface) varies based on the eye posture. Thus, the eye posture can be derived from the positioning of the reflection or flash produced by the active illumination LED and also from the pupil contour.

[0057] Embodiments of the methods and apparatus described herein for user eye registration may be used, for example, in an HMD of a computer generated reality (XR) system, such as a mixed or augmented reality (MR) system or a virtual reality (VR) system. In conventional systems, a general human eye model is typically used as a basis. In an embodiment, instead of using a general human eye model, a user perceived eye model is constructed from images of the user's eyes captured during the registration process.

[0058] In some embodiments, a ground-truth target may be used during the enrollment process to guide the user and build an eye model. However, embodiments are described in which a user-perceived eye model may be built without using a ground-truth target. Removing the dependency on a ground-truth target may provide a better user experience and also provide more freedom to the system designer.

[0059] The system implementing the user eye model registration method may include, but is not limited to, at least one eye camera (e.g., infrared (IR) or near infrared (NIR) camera, RGB or RGB-D camera, etc.), an illumination source including a light emitting element (e.g., IR or NIR LED or LED of other wavelengths), a controller, and a display. The implementation scheme of the user eye model registration method may be, for example, in FIG. 5A to FIG. 5F can be implemented in any of the systems shown. FIG. 7A to FIG. 9 Example devices and systems are shown that can implement embodiments of the user eye model registration method.

[0060] Example System

[0061] Fig. 7A and Figure 7B is an example of a method that may include, according to some embodiments, Figures 1 to 6 Block diagram of the illustrated components and devices implementing the methods as illustrated in these figures.An example application of the methods described herein for user eye model registration is in a handheld device 3000 such as a smartphone, tablet or tablet computer. Fig. 7A A side view of an example device 3000 is shown, and Figure 7BAn example top view of an example device 3000 is shown. The device 3000 may include, but is not limited to, a display screen (not shown), a controller 3060 including one or more processors, a memory 3070, a posture, motion and orientation sensor (not shown), and one or more cameras or sensing devices, such as a visible light camera and a depth sensor (not shown). A camera 3080 and an illumination source 3040 may be attached to or integrated into the device 3000, and the device 3000 may be held by a user and positioned so that the camera 3080 may capture an image of the user's eyes when illuminated by the illumination source 3050. The captured image may be processed, for example, by the controller 3060 to generate a user eye model. The user eye model may then be used in another process such as gaze tracking or biometric authentication.

[0062] Please note that if Fig. 7A and Figure 7B The device 3000 shown is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the device 3000 may vary, and the location, number, type, and other features of the components of the device 3000 may vary.

[0063] Figure 8 An example head mounted device (HMD) according to some embodiments is illustrated, which example HMD may include Figures 1 to 6 4000 and implements the methods illustrated in these figures. HMD 4000 may, for example, be a component in a mixed or augmented reality (MR) system. Figure 8 The illustrated HMD 4000 is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the HMD 4000 may differ, and the location, number, type, and other features of the components of the HMD 4000 may vary. In some embodiments, the HMD 4000 may include, but is not limited to, a display and two optical lenses (eyepieces) (not shown) mounted in a wearable housing or frame. Figure 8 As shown, HMD 4000 can be positioned on the user's head 4090 so that the display is set in front of the user's eyes 4092. The user looks at the display through the eyepiece. HMD 4000 may also include sensors (e.g., eye tracking sensors) that collect information about the user's environment (video, depth information, lighting information, etc.) and information about the user. The sensors may include, but are not limited to, one or more eye cameras 4040 (e.g., infrared (IR) cameras) that capture the view of the user's eyes 4092, one or more scene (visible light) cameras (e.g., RGB cameras) that capture images of the real world environment in the field of view in front of the user (not shown), and one or more ambient light sensors (not shown) that capture lighting information of the environment.

[0064] The controller 4060 of the MR system may be implemented in the HMD 4000, or alternatively may be implemented at least in part by an external device (e.g., a computing system) that is communicatively coupled to the HMD 4000 via a wired or wireless interface. The controller 4060 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), and / or other components for processing and rendering video and / or images. The controller 4060 may render frames (each frame including a left image and a right image) including virtual content based at least in part on input obtained from a sensor, and may provide the frames to a display. Fig. 9 Components of an HMD and an MR system according to some embodiments are further illustrated.

[0065] In some embodiments, an imaging system for an MR system may include, but is not limited to, one or more eye cameras 4040 and an IR light source 4030. An IR light source 4030 (e.g., an IR LED) may be positioned in the HMD 4000 (e.g., around the eyepiece or elsewhere in the HMD 4000) to illuminate the user's eyes 4092 with IR light. At least one eye camera 4040 (e.g., an IR camera, such as a 400×400 pixel digital camera or a 600×600 pixel digital camera that operates at 850 nm or 940 nm or at some other IR wavelength or wavelength combination and captures frames at, for example, a rate of 60-120 frames per second (FPS)) is located at each side of the user's 4090 face. In various embodiments, eye cameras 4040 may be positioned on each side of the face of the user 4090 in the HMD 4000 to provide a direct view of the eye 4092, a view of the eye 4092 through the eyepiece, or a view of the eye 4092 via reflection from a thermal mirror or other reflective component. Note that the positions and angles of the eye cameras 4040 are given by way of example and are not intended to be limiting. Although Figure 8 A single eye camera 4040 is shown located on each side of the face of the user 4090 , but in some embodiments, there may be two or more eye cameras 4040 on each side of the face of the user 4090 .

[0066] A portion of the IR light emitted by the light source 4030 reflects from the eye of the user 4090 and is captured by the eye camera 4040 to image the user's eye 4092. The image captured by the eye camera 4040 may be analyzed by the controller 4060 to detect features (e.g., pupil and corneal features), location, and movement of the user's eye 4092, and / or detect other information about the eye 4092, such as pupil dilation. The registration process described herein may be performed by the controller 4060 to generate a model of the user's eye. The eye model may then be used in other processes. For example, the eye model may be used to estimate a gaze point on a display from eye tracking information; the estimated gaze point may be used to enable a scene camera of the HMD 4000 to expose an image of a scene based on a region of interest (ROI) corresponding to the gaze point. As another example, the estimated gaze point may enable gaze-based interaction with content displayed on the display. As another example, in some embodiments, the brightness of the displayed image may be adjusted based on the user's pupil dilation as determined by the imaging system. The HMD 4000 may implement such Figures 1 to 6 One or more of the methods shown in to capture and process images of a user's eye 4090 and generate an eye model based on the captured images.

[0067] like Figure 8 The illustrated embodiment of the HMD 4000 may be used, for example, for augmented or mixed (AR) applications to provide an augmented or mixed reality view to a user 4090. The HMD 4000 may include, for example, one or more sensors positioned on an outer surface of the HMD 4000 that collect information about the external environment of the user 4090 (video, depth information, lighting information, etc.); the sensors may provide the collected information to a controller 4060 of the MR system. The sensors may include one or more visible light cameras (e.g., RGB cameras) that capture video of the user's environment, which may be used to provide the user 4090 with a virtual view of his or her real environment. In some embodiments, the video stream of the real environment captured by the visible light camera may be processed by the controller 4060 of the HMD 4000 to render an augmented or mixed reality frame including virtual content overlaid on a view of the real environment, and the rendered frame may be provided to the display system of the HMD 4000.

[0068] Fig. 9 is a block diagram illustrating an example MR system according to some embodiments, which may include Figures 1 to 6, and implement the methods as illustrated in these figures. In some embodiments, the MR system may include HMD 5000, such as headphones, helmets, goggles or glasses. HMD 5000 can implement any of various types of display technologies. For example, HMD 5000 may include a display system that displays a frame including a left image and a right image on a screen or display (not shown) viewed by a user through an eyepiece (not shown). The display system may be, for example, a DLP (digital light processing), LCD (liquid crystal display) or LCoS (liquid crystal on silicon) technology display system. In order to create a three-dimensional (3D) effect in a 3D virtual view, objects at different depths or distances in the two images may be offset to the left or right as a function of triangulation of the distance, with closer objects offset more than farther objects. It should be noted that in some embodiments, other types of display systems may be used.

[0069] In some embodiments, the HMD 5000 may include a controller 5060 configured to implement the functions of the MR system and generate frames (each frame including a left image and a right image) provided to the display of the HMD. In some embodiments, the HMD 5000 may also include a memory 5062 configured to store software (code 5064) of the MR system executable by the controller 5060, and data 5068 that can be used by the MR system when executed on the controller 5060. In some embodiments, the HMD 5000 may also include one or more interfaces (e.g., Bluetooth technology interface, USB interface, etc.), and one or more interfaces (e.g., Bluetooth technology interface, USB interface, etc.) are configured to communicate with external devices via wired or wireless connections. In some embodiments, at least a portion of the functions described for the controller 5060 can be implemented by an external device. The external device may be or may include any type of computing system or computing device, such as a desktop computer, a notebook or laptop computer, a tablet or tablet device, a smart phone, a handheld computing device, a game controller, a gaming system, and the like.

[0070] In various embodiments, the controller 5060 may be a single processor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number). The controller 5060 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. For example, in various embodiments, the controller 5060 may include a general-purpose processor or embedded processor that implements any instruction set architecture in a variety of instruction set architectures (ISAs) such as x86, PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each processor may implement the same ISA together, but this is not required. In some embodiments, the controller 5060 may be implemented as a system on chip (SoC). For example, in some embodiments, a processor, memory, I / O interface (e.g., structure), etc. may be implemented in a single SoC including multiple components integrated into a single chip. For example, a SoC may include multiple CPU cores, a multi-core GPU, a multi-core neural engine, a cache, one or more memories, etc. integrated into a single chip. In some embodiments, the SoC implementation implements a reduced instruction set computing (RISC) architecture or any other suitable architecture. The controller 5060 may adopt any microarchitecture, including scalar, superscalar, pipeline, superpipeline, out of order, in order, speculative, non-speculative, etc., or a combination thereof. The controller 5060 may include a circuit that implements microcode technology. The controller 5060 may include one or more processing cores that are each configured to execute instructions. The controller 5060 may include one or more levels of cache, which may be of any size and any configuration (set association, direct mapping, etc.). In some embodiments, the controller 5060 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuit. Typically, a GPU may be configured to render an object to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). The GPU may include one or more graphics processors that may execute graphics software to perform some or all of the graphics operations or hardware acceleration of certain graphics operations. In some embodiments, the controller 5060 may include one or more other components for processing and rendering video and / or images, such as an image signal processor (ISP), an encoder / decoder (codec), etc.

[0071] The memory 5062 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., or low power versions of SDRAM, such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single inline memory module (SIMM), a dual inline memory module (DIMM), etc. Alternatively, these devices may be mounted in a chip stack configuration, a package stack configuration, or a multi-chip module configuration with an integrated circuit implementing the system.

[0072] In some embodiments, the HMD 5000 may include one or more sensors that collect information about the user's environment (video, depth information, lighting information, etc.). The sensors may provide the information to the controller 5060 of the MR system. In some embodiments, the sensors may include, but are not limited to, visible light cameras (e.g., video cameras) and ambient light sensors.

[0073] The HMD 5000 may be positioned on the user's head such that the display and the eyepieces are disposed in front of the user's eyes 5092A and 5092B. IR light sources 5030A and 5030B (e.g., IR LEDs) may be positioned in the HMD 5000 (e.g., around the eyepieces or elsewhere in the HMD 5000) to illuminate the user's eyes 5092A and 5092B with IR light. Eye cameras 5040A and eye tracking cameras 5040B (e.g., IR cameras, such as 400×400 pixel digital cameras or 600×600 pixel digital cameras that operate at 850 nm or 940 nm or some other IR wavelength and capture frames at a rate of, for example, 60-120 frames per second (FPS)) may be located on each side of the user's face. In various embodiments, the eye camera 5040 may be positioned in the HMD 5000 to provide a direct view of the eye 5092, a view of the eye 5092 through the eyepiece 5020, or a view of the eye 5092 via reflection from a thermal mirror or other reflective component. Note that the positions and angles of the eye camera 5040A and the eye tracking camera 5040B are given by way of example and are not intended to be limiting. In some embodiments, there may be a single eye camera 5040 located on each side of the user's face. In some embodiments, there may be two or more eye cameras 5040 on each side of the user's face. For example, in some embodiments, a wide angle camera 5040 and a narrower angle camera 5040 may be used on each side of the user's face. A portion of the IR light emitted by the light sources 5030A and 5030B is reflected from the user's eyes 5092A and 5092B, received at the corresponding eye cameras 5040A and 5040B, and captured by the eye cameras 5040A and 5040B to image the user's eyes 5092A and 5092B. Eye information captured by the cameras 5040A and 5040B may be provided to the controller 5060. The controller 5060 may analyze the eye information (e.g., images of the user's eyes 5092A and 5092B) to determine eye positioning and movement and / or other features of the eyes 5092A and 5092B. In some embodiments, in order to accurately determine the position of the user's eyes 5092A and eyes 5092B relative to the eye camera 5040A and the eye tracking camera 5040B, the controller 5060 can use the images captured by the eye camera 5040A and the eye camera 5040B to perform 3D reconstruction to generate a 3D model of the user's eyes 5092A and eyes 5092B. The 3D model of the eyes 5092A and eyes 5092B indicates the 3D positioning of the eyes 5092A and eyes 5092B relative to the eye camera 5040A and the eye camera 5040, which, for example, allows the eye tracking algorithm executed by the controller to accurately track eye movements.HMD 5000 can achieve such as. Figures 1 to 6 One or more of the methods shown in to capture and process images of a user's eye 5090 and generate an eye model based on the captured images.

[0074] The eye information obtained and analyzed by the controller 5060 can be used by the controller to implement various VR or AR system functions. For example, the gaze point on the display can be estimated from the images captured by the eye camera 5040A and the eye camera 5040B and the eye model; the estimated gaze point can be used to cause one or more scene cameras of the HMD 5000 to expose an image of the scene based on a region of interest (ROI) corresponding to the gaze point. For another example, the estimated gaze point can enable gaze-based interaction with virtual content displayed on the display. As another example, in some embodiments, the brightness of the displayed image can be adjusted based on the pupil dilation of the user as determined by the imaging system.

[0075] In some embodiments, the HMD 5000 may be configured to render and display frames to provide an augmented or mixed reality (MR) view for the user based at least in part on the sensor input. The MR view may include rendering the user's environment, including rendering real objects in the user's environment based on video captured by one or more cameras, the one or more cameras capturing high-quality, high-resolution video of the user's environment for display. The MR view may also include virtual content (e.g., virtual objects, virtual labels for real objects, an avatar of the user, etc.) generated by the MR system and synthesized with the displayed view of the user's real environment.

[0076] like Fig. 9 The embodiments of the HMD 5000 shown in FIG. 5 may also be used in virtual reality (VR) applications to provide a VR view to a user. In these embodiments, a controller 5060 of the HMD 5000 may render or obtain a virtual reality (VR) frame including virtual content, and the rendered frame may be displayed to provide a virtual reality (as opposed to mixed reality) experience to the user. In these systems, the rendering of the VR frame may be affected based on a gaze point determined from an imaging system.

[0077] Extended Reality

[0078] A real environment is an environment that a person can perceive (e.g., see, hear, feel) without using a device. For example, an office environment may include furniture such as tables, chairs, and filing cabinets; structural members such as doors, windows, and walls; and objects such as electronic devices, books, and writing instruments. A person in a real environment can perceive various aspects of the environment and may be able to interact with objects in the environment.

[0079] On the other hand, an extended reality (XR) environment is partially or fully simulated using electronic devices. For example, in an XR environment, a user can see or hear computer-generated content that partially or completely replaces the user's perception of the real environment. In addition, the user can interact with the XR environment. For example, the user's movement can be tracked, and virtual objects in the XR environment can change in response to the user's movement. As another example, a device that presents the XR environment to the user can determine that the user is moving their hand toward the virtual location of a virtual object, and can move the virtual object in response. In addition, the user's head position and / or eye gaze can be tracked, and the virtual object can be moved to stay in the user's line of sight.

[0080] Examples of XR include augmented reality (AR), virtual reality (VR), and mixed reality (MR). XR can be considered a spectrum of realities, where VR, on the one hand, allows users to fully immerse themselves in the real environment, replacing it with virtual content, and on the other hand, users can experience the real environment without the aid of equipment. In between are AR and MR, which mix virtual content with the real environment.

[0081] VR generally refers to a type of XR that fully immerses the user and replaces the user's real environment. For example, VR can be presented to the user using a head-mounted device (HMD), which may include a near-eye display for presenting a virtual visual environment to the user and headphones for presenting a virtual audible environment. In a VR environment, the user's movements can be tracked and cause the user's observation of the environment to change. For example, a user wearing an HMD can walk in a real environment, and the user will appear to be walking in the virtual environment they are experiencing. In addition, the user can be represented by an avatar in the virtual environment, and the HMD can use various sensors to track the user's movements to animate the user's avatar.

[0082] AR and MR refer to a class of XR that includes some mix of real environment and virtual content. For example, a user might hold a tablet computer that includes a camera that captures images of the user's real environment. The tablet computer may have a display that displays images of the real environment mixed with images of virtual objects. AR or MR may also be presented to the user through an HMD. The HMD may have an opaque display, or may use a transparent display, which allows the user to see the real environment through the display while displaying virtual content overlaid on the real environment.

[0083] There are many types of devices that allow users to experience various forms of XR. Examples include HMDs, heads-up displays (HUDs), projector-based systems, smart windows, tablets, desktop or laptop computers, smart watches, earbuds / headphones, controllers that may include tactile devices, and many other devices. As mentioned above, any of the HMDs or other devices listed above may include an opaque display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or a micro-LED display) or a perspective display. The perspective display may have a medium through which light is directed to the user's eyes. The medium may include one or more of a waveguide, a holographic medium, an optical combiner, an optical reflector, and other optical components. Display sources such as OLEDs, micro-LEDs, liquid crystal on silicon (LCOS), optical scanners, and digital light projection (DLP) may be used to generate images and propagate images through the medium.

[0084] Devices for XR may also include audio output devices, such as speakers that present audio (including spatial audio) to the user, haptic devices that stimulate the user's sense of touch, and other devices that stimulate any of the user's senses. In addition, the device may include numerous sensors, including cameras, microphones, depth sensors, eye tracking sensors, environmental sensors, input sensors, and other sensors to allow the device to understand the user and the real environment.

[0085] In different embodiments, the method described herein can be implemented in software, hardware or a combination thereof. In addition, the order of the frames of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art who benefit from the present disclosure, various modifications and changes can obviously be made. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions and improvements are possible. Therefore, multiple examples can be provided for the components described as a single example in this article. The boundaries between various components, operations and data repositories are arbitrary to a certain extent, and specific operations are shown in the context of a specific example configuration. Other allocations of functions are expected, and they may fall within the scope of the appended claims. Finally, the structure and function presented as discrete components in the example configuration may be implemented as a combined structure or component. These and other variations, modifications, additions and improvements may fall within the scope of the embodiments defined in the following claims.

Claims

1. A system comprising: one or more cameras configured to capture images of the eye; A memory storing a set of eye models; and A controller comprising one or more processors configured to: capturing, by the one or more cameras, one or more images of the eye; determining one or more characteristics of the eye from the captured image; determining an eye model from the set of eye models that provides a best match to the eye by evaluating the determined eye characteristics against eye characteristics of the eye model; as well as The eye model providing the best match is implemented in a gaze tracking process, in which features of the eye are derived from one or more images of the eye captured by the one or more cameras, and a gaze direction is estimated based on the derived features and the implemented eye model.

2. The system of claim 1 , wherein the controller is configured to implement the eye model in the gaze tracking process when the eye model providing the best match satisfies a threshold.

3. The system of claim 1, wherein the eye model that provides the best match is the eye model with the highest match score that satisfies a threshold.

4. The system of claim 3, wherein a matching score of the eye model is determined based on a residual error between the eye characteristics of the eye model and the eye characteristics of the eye determined from the captured image.

5. The system of claim 1, wherein the controller is further configured to implement an eye model registration process when the eye model providing the best match does not satisfy a threshold.

6. The system of claim 1, wherein the controller is configured to determine the eye model from the set of eye models that provides the best match in response to detecting the presence of the eye in the captured image.

7. The system of claim 1, wherein the one or more characteristics of the eye determined from the captured image comprise eye features.

8. The system of claim 7, wherein the one or more processors are configured to: determining two or more eye gestures from two or more images of the eye at different orientations captured by the one or more cameras, wherein the eye gestures indicate a current eye position and orientation relative to the one or more cameras; and The eye features are calculated based at least in part on two or more eye gestures.

9. The system of claim 7, wherein the eye features include a corneal profile and pupil features of the eye.

10. The system of claim 1, wherein the system is a head mounted device (HMD), a handheld device, or a wall mounted device.

11. The system of claim 1 , further comprising: one or more second cameras configured to capture an image of a second eye; The controller is further configured to: capturing, by the second camera, one or more second images of the second eye; determining one or more characteristics of the second eye from the captured second image; determining a second eye model from the set of eye models that provides the best match to the second eye by evaluating the determined second eye characteristic against the eye characteristics of the eye model; as well as The second eye model providing the best match is implemented in a second gaze tracking process, in which features of the second eye are derived from one or more second images of the second eye captured by the one or more second cameras, and a second gaze direction is estimated based on the derived features and the implemented second eye model.

12. A method comprising: capturing one or more images of the eye using one or more cameras positioned in front of the eye, the cameras being located on a device configured to be positioned near the head; determining, by a computer processor coupled to the head mounted device, one or more characteristics of the eye from the captured image; determining, by the computer processor, a match score for each eye model in a set of eye models stored in a memory coupled to the computer processor, wherein the match score is determined based on a comparison of eye features determined from the captured image with eye features in the eye model; as well as A gaze tracking process is implemented by the camera in the device, wherein the gaze tracking process utilizes the eye model with the highest matching score, and wherein the gaze tracking process derives features of the eye from one or more images of the eye captured by the one or more cameras, and estimates a gaze direction based on the derived features and the implemented eye model.

13. The method according to claim 12, further comprising: When the matching score of the eye model with the highest matching score satisfies a threshold, the eye model is utilized in the gaze tracking process.

14. The method of claim 12, wherein the eye features include corneal contour and pupil features of the eye.

15. The method of claim 12, wherein the matching score for each eye model is determined based on a residual error between the eye features in the eye model and the eye features determined from the captured image.

16. The method of claim 12, wherein determining the one or more features of the eye from the captured image comprises: determining two or more eye poses from the two or more captured images of the eye at different orientations, wherein the eye poses indicate a current eye position and orientation relative to the one or more cameras; as well as The eye feature is calculated based at least in part on the two or more eye gestures.

17. A system comprising: one or more cameras configured to capture images of the right eye and the left eye; A memory storing a set of eye models for the right eye and the left eye; and A controller comprising one or more processors configured to: capturing, by the one or more cameras, one or more images of the right eye and the left eye; determining one or more eye features of the right eye from the captured image; determining a first matching score for each eye model of the right eye in the set of eye models, wherein the matching score is determined based on a comparison of eye features of the right eye determined from the captured image with eye features of the right eye in the eye models; implementing a first gaze tracking process for the right eye using the eye model of the right eye having the highest first matching score, wherein the first gaze tracking process derives features of the right eye from one or more images of the right eye captured by the one or more cameras, and estimates a gaze direction based on the derived features and the implemented eye model; determining one or more eye features of the left eye from the captured image; determining a second match score for each eye model of the left eye in the set of eye models, wherein the match score is determined based on a comparison of eye features of the left eye determined from the captured image with eye features of the left eye in the eye models; as well as The eye model of the left eye with the highest second matching score is used to implement a second gaze tracking process for the left eye, wherein the second gaze tracking process derives features of the left eye from one or more images of the left eye captured by the one or more cameras, and estimates a gaze direction based on the derived features and the implemented eye model.

18. The system of claim 17, wherein the one or more cameras include at least a first camera and a second camera, the first camera being configured to capture a first image of the right eye and the second camera being configured to capture a second separate image of the left eye.

19. The system of claim 17, further comprising: An illumination source includes a plurality of light emitting elements configured to emit light toward the right eye and the left eye for imaging by the camera.

20. The system of claim 19, wherein the light emitting element comprises an infrared (IR) light source, and wherein the one or more cameras are IR cameras.

Images