Eye tracking and gaze estimation using off-axis cameras

Abstract

Techniques related to computing a gaze vector of a user of a wearable device are disclosed. A neural network can be trained through first and second training steps. The neural network can include a set of feature encoding layers and a plurality of sets of task-specific layers, each set of task-specific layers operating on an output of the set of feature encoding layers. During the first training step, a first image of a first eye can be provided to the neural network, eye segmentation data can be generated using the neural network, and the set of feature encoding layers can be trained. During the second training step, a second image of a second eye is provided to the neural network, network output data can be generated using the neural network, and the plurality of sets of task-specific layers can be trained.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 888,953, filed August 19, 2019, entitled “EYENET: A MULTI-TASK DEEP NETWORK FOR OFF-AXIS EYE GAZE ESTIMATION AND SEMANTIC USER UNDERSTANDING,” U.S. Provisional Application No. 62 / 926,241, filed October 25, 2019, entitled “METHOD AND SYSTEM FOR PERFORMING EYE TRACKING USING AN OFF-AXIS CAMERA,” and U.S. Provisional Application No. 62 / 935,584, filed November 14, 2019, entitled “METHOD AND SYSTEM FOR PERFORMING EYE TRACKING USING AN OFF-AXIS CAMERA,” the entire contents of which are incorporated by reference for all purposes. BACKGROUND

[0003] Modern computing and display technologies have facilitated the development of systems for so called “virtual reality” or “augmented reality” experiences, wherein digitally reproduced images or portions of images are presented to a user in a manner wherein they seem to be, or can be perceived as, real. A virtual reality, or “VR,” scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR,” scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world. In some scenarios, the entirety of the user’s field of view (FOV) is dominated by the digitally reproduced image information; in other scenarios, only a portion of the user’s FOV is filled with the digitally reproduced image information.

[0004] Despite the advances in these display technologies, there is a continuing need in the art for improved methods, systems, and devices related to augmented reality systems, particularly display systems. SUMMARY

[0005] The present disclosure relates generally to systems and methods for eye tracking. More specifically, embodiments of the present disclosure provide systems and methods for performing eye tracking in head-mounted virtual reality (VR), mixed reality (MR), and / or augmented reality (AR) devices for gaze estimation. Embodiments of the present disclosure enable the use of energy and bandwidth efficient content rendering to drive a multi-focal display in a manner that is effective and unobtrusive to the needs of the user. Although the present disclosure is described with reference to AR devices, the present disclosure is applicable to a variety of applications in computer vision and image display systems.

[0006] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. As used in the following, any reference to a series of examples will be understood to be a reference to each of those examples individually (e.g., "Examples 1 to 4" will be understood to be "Examples 1, 2, 3, or 4").

[0007] Example 1 is a method of training a neural network having a set of feature encoding layers and a plurality of sets of task-specific layers, each set of task-specific layers operating on an output of the set of feature encoding layers, the method comprising: performing a first training step comprising: providing a first image of a first eye to the neural network; generating, using the neural network, eye segmentation data based on the first image, wherein the eye segmentation data comprises a segmentation of the first eye into a plurality of regions; and training the set of feature encoding layers using the eye segmentation data; and performing a second training step comprising: providing a second image of a second eye to the neural network; generating, using the set of feature encoding layers and each of the plurality of sets of task-specific layers, network output data based on the second image; and training the plurality of sets of task-specific layers using the network output data.

[0008] Example 2 is the method of Example 1, wherein the first training step is performed during a first duration, and the second training step is performed during a second duration after the first duration.

[0009] Example 3 is the method of Example 1, wherein the plurality of regions comprises one or more of a background region, a sclera region, a pupil region, or an iris region.

[0010] Example 4 is the method of Example 1, wherein performing the first training step further comprises: training a single set of task-specific layers of the plurality of sets of task-specific layers using the eye segmentation data.

[0011] Example 5 is the method of Example 4, wherein the single set of task-specific layers is the only set of task-specific layers of the plurality of sets of task-specific layers that is trained during the first training step.

[0012] Example 6 is the method of example 1, wherein performing the first training step further comprises: receiving eye segmentation ground truth (GT) data; and comparing the eye segmentation data to the eye segmentation GT data.

[0013] Example 7 is the method of example 1, wherein the set of feature encoding layers are not trained during the second training step.

[0014] Example 8 is the method of example 1, wherein the network output data comprises two-dimensional (2D) pupil data corresponding to the second eye.

[0015] Example 9 is the method of example 1, wherein the network output data comprises glint detection data corresponding to the second eye.

[0016] Example 10 is the method of example 1, wherein the network output data comprises corneal center data corresponding to the second eye.

[0017] Example 11 is the method of example 1, wherein the network output data comprises blink prediction corresponding to the second eye.

[0018] Example 12 is the method of example 1, wherein the network output data comprises gaze classification corresponding to the second eye.

[0019] Example 13 is the method of example 1, wherein the network output data comprises second eye segmentation data, the second eye segmentation data comprising a second segmentation that segments the second eye into a second plurality of regions.

[0020] Example 14 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any of examples 1-13.

[0021] Example 15 is a system comprising: one or more processors; and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any of examples 1-13.

[0022] Example 16 is a method of training a neural network for classifying a user’s gaze, the method comprising: capturing an image of an eye; providing the image of the eye to the neural network; using the neural network, generating a gaze classification corresponding to the eye based on the image of the eye, wherein the gaze classification is one of a plurality of possible gaze classifications; determining a ground truth (GT) gaze classification; computing error data based on a difference between the gaze classification and the GT gaze classification; and modifying the neural network based on the error data.

[0023] Example 17 is the method of example 16, wherein the image of the eye is captured using a camera of the wearable display device.

[0024] Example 18 is the method of example 16, wherein determining the GT gaze classification comprises receiving user input indicating the GT gaze classification.

[0025] Example 19 is the method of example 16, wherein determining the GT gaze classification comprises determining that an instruction delivered to the user indicates the GT gaze classification.

[0026] Example 20 is the method of example 16, further comprising delivering an instruction to the user indicating the GT gaze classification prior to capturing the image of the eye.

[0027] Example 21 is the method of example 16, wherein modifying the neural network comprises modifying a set of weights of the neural network.

[0028] Example 22 is the method of example 21, wherein the set of weights is modified using backpropagation.

[0029] Example 23 is the method of example 16, wherein the neural network is modified based on a magnitude of the error data.

[0030] Example 24 is the method of example 16, further comprising outputting light to the eye by a plurality of infrared (IR) light emitting diodes (LEDs) such that the image of the eye includes a plurality of glints.

[0031] Example 25 is the method of example 16, wherein the image of the eye includes a plurality of glints resulting from light output by a plurality of infrared (IR) light emitting diodes (LEDs).

[0032] Example 26 is the method of example 16, wherein the image of the eye does not include eyebrows of a user of the eye.

[0033] Example 27 is the method of example 16, wherein the plurality of possible gaze classifications includes at least one of neutral, happy, disinterested, or sensitive.

[0034] Example 28 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any of examples 16-27.

[0035] Example 29 is a system comprising: one or more processors; and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any of examples 16-27.

[0036] Example 30 is a method of training a neural network for computing a gaze vector, the method comprising: capturing an image of an eye; processing the image of the eye to produce an optical axis corresponding to the eye; providing the optical axis to the neural network; using the neural network, generating a gaze vector corresponding to the eye based on the optical axis; determining gaze vector ground truth (GT) data; computing error data based on a difference between the gaze vector and the gaze vector GT data; and modifying the neural network based on the error data.

[0037] Example 31 is the method of example 30, wherein the image of the eye is captured using a camera of a wearable display device.

[0038] Example 32 is the method of example 30, wherein the gaze vector GT data is determined based on a location at which a target is displayed on a screen.

[0039] Example 33 is the method of example 30, wherein determining the gaze vector GT data comprises: receiving user input indicative of the gaze vector GT data.

[0040] Example 34 is the method of example 30, wherein determining the gaze vector GT data comprises: determining that an instruction delivered to a user is indicative of the gaze vector GT data.

[0041] Example 35 is the method of example 30, further comprising: prior to capturing the image of the eye, delivering an instruction to a user indicative of the gaze vector GT data.

[0042] Example 36 is the method of example 30, further comprising: displaying a target at a location on a screen, wherein the gaze vector GT data is determined based on the location.

[0043] Example 37 is the method of example 30, wherein modifying the neural network comprises: modifying a set of weights of the neural network.

[0044] Example 38 is the method of example 37, wherein the set of weights is modified using backpropagation.

[0045] Example 39 is the method of example 30, wherein the neural network is modified based on a magnitude of the error data.

[0046] Example 40 is the method of example 30, further comprising: outputting light to the eye by a plurality of infrared (IR) light emitting diodes (LEDs) such that the image of the eye includes a plurality of glints.

[0047] Example 41 is the method of example 30, wherein the image of the eye includes a plurality of glints produced by light output by a plurality of infrared (IR) light emitting diodes (LEDs).

[0048] Example 42 is the method of example 30, wherein the gaze vector comprises at least one angle.

[0049] Example 43 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any of examples 30 to 42.

[0050] Example 44 is a system comprising: one or more processors; and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any of examples 30 to 42.

[0051] Many benefits are achieved by way of the present disclosure over conventional techniques. For example, through eye images, user’s eye gaze estimation and simultaneous understanding enables energy and bandwidth efficient content rendering (gaze point rendering), drives multi-focal displays for more realistic content rendering (minimizes visual vergence accommodation conflict), and provides an effective and unobtrusive method for understanding user’s expressions. An additional benefit is that the estimation using trained networks works well in conjunction with classic eye tracking pipelines. It has been demonstrated that the estimation using trained networks can be used in a geometric eye tracking system to improve its overall robustness and accuracy.

[0052] Further, the results of the multi-stage eye tracking model described herein can drive other important applications in AR / VR / MR. For example, cornea prediction can be used for gaze point rendering, and eye segmentation is useful for rendering eyes in avatar-based social kit applications. Although it can be both inaccurate and difficult to collect gaze target GT data for a large number of topics, by decoupling the training of intermediate predictions (pupil and cornea estimation) from the final 3D gaze vector estimation pipeline, the data collection herein is significantly simpler. Since errors in end-to-end deep networks can be difficult to interpret, the intermediate estimation using trained networks in each stage improves interpretability. Other benefits of the present disclosure will be readily apparent to those skilled in the art. BRIEF DESCRIPTION OF DRAWINGS

[0053] A further understanding of the nature and advantages of various embodiments can be realized by reference to the following drawings. In the drawings, like reference numerals can designate similar structures or features. Further, various components of the same type can be distinguished by adding a dash and a second label that distinguishes among the components of the same type. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.

[0054] Figure 1An augmented reality (AR) scene viewed through a wearable AR device is shown.

[0055] Figure 2 Various features of the AR device are shown.

[0056] Figure 3 A standard bi-spherial model of a human eye is shown.

[0057] Figure 4 A schematic diagram of an AR device is shown.

[0058] Figure 5 A schematic diagram of a system for computing a gaze vector that incorporates a multi-task neural network is shown.

[0059] Figure 6 A schematic diagram of a multi-task neural network is shown.

[0060] Figure 7 Systems and techniques for generating a blink prediction using features from different time steps are shown.

[0061] Figure 8 A schematic diagram of an AR device operating in a training mode is shown.

[0062] Figure 9A And Figure 9B A schematic diagram of sequential training steps for training a multi-task neural network is shown.

[0063] Figure 10 A schematic diagram of an AR device operating in a runtime mode is shown.

[0064] Figure 11 A schematic diagram of a gaze vector neural network is shown.

[0065] Figure 12 A training pipeline is shown.

[0066] Figure 13 A method of training a neural network having a set of feature encoding layers and a plurality of sets of task-specific layers is shown.

[0067] Figure 14 A method of training a neural network for classifying a user's eye behavior is shown.

[0068] Figure 15 A method of training a neural network for computing a gaze vector is shown.

[0069] Figure 16 A method of computing a gaze vector using a neural network is shown.

[0070] Figure 17 A method of training a neural network is shown.

[0071] Figure 18 A simplified computer system is shown. DETAILED DESCRIPTION

[0072] User's eye gaze estimation and simultaneous semantic understanding through eye images is an important component of virtual reality (VR) and mixed reality (MR); enabling energy efficient rendering, multi-focal display, and effective interaction with 3D content. In head-mounted VR / MR devices, the eyes can be imaged off-axis to avoid blocking the user's gaze, which can make making eye-related inferences very challenging. In various embodiments described herein, a single deep neural network is provided that solves multiple heterogeneous tasks related to eye gaze estimation and semantic user understanding for off-axis camera setups. Tasks can include eye segmentation, blink detection, emotion expression classification, infrared radiation (IR) light-emitting diode (LED) glint detection, and pupil and cornea center estimation. To train the neural network end-to-end, both human-labeled supervision and model-based supervision can be used.

[0073] The process of estimating accurate gaze involves appearance-based computations (segmentation, keypoint detection, e.g., pupil center, glints), followed by geometry-based computations (e.g., estimating cornea, pupil center, and gaze vector in three dimensions). Current eye trackers use classic computer vision techniques (without learning) to estimate the pupil boundary / center, and then compute gaze based on those estimates. The estimation described herein using a trained network is significantly more accurate than classic techniques. According to some embodiments described herein, a single deep network is trained to jointly estimate multiple quantities related to eye and gaze estimation for off-axis eye images.

[0074] Figure 1 An augmented reality (AR) scene as viewed through a wearable AR device according to embodiments described herein is shown. An AR scene 100 is depicted, in which a user of an AR technology sees a real-world park-like setting 106 featuring people, trees, buildings in the background, and a specific platform 120. In addition to these items, the user of the AR technology also perceives that he "sees" a robotic statue 110 standing on the real-world platform 120 and a flying cartoon-like avatar character 102 (which appears to be an avatar of a bumblebee), even though these elements (character 102 and statue 110) do not exist in the real world. Due to the extreme complexity of the human visual perception and nervous systems, it is challenging to create VR or AR technologies that facilitate a comfortable sense of presence of virtual image elements within a rich presentation of other virtual or real-world imagery elements.

[0075] Figure 2Various features of AR device 200 are shown in accordance with some embodiments of the present disclosure. In some embodiments, AR device 200 can include a projector 214 configured to project virtual image light 222 (light associated with virtual content) onto eyepiece 202 to cause a user to perceive one or more virtual objects (e.g., character 102 and statue 110) as being located at a certain location within the user’s environment (e.g., at one or more depth planes). The user can perceive these virtual objects as being next to world object 230. AR device 200 can also include an off-axis camera 240 and one or more emitters 262 mounted to AR device 200 and pointed at the user’s eye. Emitters 262 can include IR LEDs that send light that is not visible to the user’s eye but that off-axis camera 240 can detect. In some embodiments, emitters 262 can include LEDs that send light that is visible to the user’s eye so that off-axis camera 240 does not need to have the ability to detect light in the IR spectrum. Thus, off-axis camera 240 can be a camera with or without IR detection capabilities.

[0076] During operation of AR device 200, off-axis camera 240 can detect information (e.g., capture images) that results in an estimate of a gaze vector 238 corresponding to the user’s eye. Gaze vector 238 can be computed for each image frame and, in various embodiments, can be represented as a two-dimensional (2D) or three-dimensional (3D) value. For example, as shown, gaze vector 238 can be represented using a spherical coordinate system by a polar angle Θ and an azimuthal angle φ. Alternatively or additionally, gaze vector 238 can be represented using a 3D Cartesian coordinate system by X, Y, and Z values. Gaze vector 238 can intersect eyepiece 202 at an intersection point 239, which can be computed based on the position of the user’s eye, the position of eyepiece 202, and gaze vector 238. In some instances, projector 214 can adjust virtual image light 222 to increase image brightness and / or clarity around intersection point 239 relative to other regions of the field of view. Figure 2

[0077] As shown, a set of four emitters 262 are placed in and around the display and their reflections (glints) are detected using off-axis camera 240. This setup is replicated for the user’s left eye. The detected glints are used to estimate important geometric quantities in the eye that are not directly observable from eye camera images. As shown, there can be a large angle between the user’s gaze and the camera axis. This makes eye gaze estimation challenging due to increased eccentricity of the pupil, partial occlusion by the eyelids and eyelashes, and glint interference due to ambient lighting. Figure 2

[0078] Figure 3 ​​A standard double-sphere model 300 of a human eye is shown. According to the model 300, an eyeball sphere 302 can completely or partially enclose an inner cornea sphere 304. A cornea center 306 can be the geometric center of the cornea sphere 304. A pupil center 308 can correspond to the pupil opening or pupil center of the eye and can be enclosed by the cornea sphere 304. An optical axis 310 of the eye can be a vector formed by connecting the cornea center 306 and the pupil center 308. A gaze vector 238 (alternatively referred to as a visual axis) can be formed by connecting the cornea center 306 and a fovea 312 at the back of the eye. Since the fovea 312 is typically unknown and difficult to estimate, the gaze vector 238 can be calculated using the optical axis 310 and a user-specific calibration angle K. The calibration angle K can be a one-dimensional (ID), 2D, or 3D value and can be calibrated for a particular user during a calibration phase when the AR device 200 is first operated by that user. Once the calibration angle K is calculated for a particular user, it is assumed to be fixed. Thus, estimating the optical axis 310 using the cornea center 306 and the pupil center 308 can be an important foundation for gaze tracking.

[0079] Figure 4 A schematic diagram of an AR device 200 is shown, in accordance with some embodiments of the disclosure. The AR device 200 can include a left eyepiece 202A, a right eyepiece 202B, a left forward-facing world camera 206A directly attached to or near the left eyepiece 202A, a right forward-facing world camera 206B directly attached to or near the right eyepiece 202B, a left side-facing world camera 206C, a right side-facing world camera 206D, and a processing module 250. An emitter 262 can be mounted to one or both of the eyepieces 202, and in some embodiments can be split into a left emitter 262A mounted directly to or near the left eyepiece 202A and a right emitter 262B mounted directly to or near the right eyepiece 202B (e.g., mounted to a frame of the AR device 200). In some instances, the AR device 200 can include a single or multiple off-axis cameras 260, such as a centrally positioned off-axis camera 260 or a left off-axis camera 260A directly mounted to or near the left eyepiece 202A and a right off-axis camera 260A directly mounted to or near the right eyepiece 202B as shown. Figure 4

[0080] Some or all of the components of the AR device 200 can be head-mounted, such that the projected images can be viewed by a user. In one particular implementation, Figure 4 ​All of the components of the illustrated AR device 200 are mounted to a single device (e.g., a single headset) that can be worn by a user. In another implementation, the processing module 250 is physically separate from the other components of the AR device 200 and communicatively coupled to the other components of the AR device 200 through a wired or wireless connection. For example, the processing module 250 can be mounted in various configurations, such as fixedly attached to a frame, fixedly attached to a helmet or hat worn by the user, embedded in a headset, or otherwise removably attached to the user (e.g., in a backpack-type configuration, in a waistband-coupling type configuration, etc.).

[0081] The processing module 250 can include at least one processor 252 and associated digital memory, such as nonvolatile memory (e.g., flash memory), which can be utilized to assist in the processing, caching, and storage of data. The data can include data captured from sensors (which can be, for example, operatively coupled to the AR device 200), such as image capture devices (e.g., cameras 206 and off-axis camera 260), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyros. For example, the processing module 250 can receive image(s) 220 from the cameras 206, or more specifically, left front-facing world camera 206A for left front image(s) 220A, right front-facing world camera 206B for right front image(s) 220B, left side-facing world camera 206C for left side image(s) 220C, and right side-facing world camera 206D for right side image(s) 220D. In some embodiments, the image(s) 220 (or images received from the off-axis camera 260) can include a single image, a pair of images, a video including a stream of images, a video including a stream of pairs of images, etc. The image(s) 220 (or images received from the off-axis camera 260) can be generated periodically and transmitted to the processing module 250 upon power up of the AR device 200, or can be generated in response to instructions transmitted by the processing module 250 to the one or more cameras.

[0082] In some embodiments, the functionality of the processing module 250 can be implemented by two or more groups of electronic hardware components (e.g., multiple groups of one or more processors, storage devices, etc.) that are disposed apart but communicatively coupled. For example, the functionality of the processing module 250 can be performed by electronic hardware components disposed within a headset in conjunction with electronic hardware components disposed within a computing device that is physically connected to the headset, one or more electronic devices (e.g., a smartphone, a computer, a peripheral device, a smart appliance, etc.) within an environment of the headset, one or more remotely located computing devices (e.g., a server, a cloud computing device, etc.), or a combination thereof.

[0083] Eyepieces 202A and 202B can each include a transparent or semi-transparent waveguide configured to direct light from projectors 214A and 214B, respectively. Specifically, processing module 250 can cause left projector 214A to output left virtual image light 222A onto left eyepiece 202A, and can cause right projector 214B to output right virtual image light 222B onto right eyepiece 202B. In some embodiments, each eyepiece 202 can include multiple waveguides corresponding to different colors and / or different depth planes.

[0084] Cameras 206A and 206B can be positioned to capture images that substantially overlap the fields of view of the user's left and right eyes, respectively. Accordingly, cameras 206A and 206B can be placed near the user's eyes, but not so close as to obscure the user's field of view. Alternatively or additionally, cameras 206A and 206B can be positioned so as to be aligned with the in-coupling locations of virtual image light 222A and 222B, respectively. Cameras 206C and 206D can be positioned to capture images to the side of the user, e.g., within the user's peripheral vision or outside the user's peripheral vision. The image(s) 220C and 220D captured using cameras 206C and 206D need not necessarily overlap the image(s) 220A and 220B captured using cameras 206A and 206B. Cameras 260A and 260B can be positioned to capture images of the user's left and right eyes, respectively. The images captured by cameras 260 can show the user's eyes in their entirety or some portion of the user's eyes.

[0085] During operation of AR device 200, processing module 250 can use multitask neural network 256 to compute gaze vector 238. In some embodiments, multitask neural network 256 can be stored in a non-transitory memory associated with or otherwise accessible to at least one processor 252 of processing module 250. Multitask neural network 256 can be an artificial neural network, a convolutional neural network, or any type of computational system that can be "learned" by processing examples. For example, multitask neural network 256 can be trained by processing artificially prepared training data that represents ground truth (GT) data. After processing each training data, multitask neural network 256 is able to generate an output that more closely approximates the GT data.

[0086] In some embodiments, the multi-task neural network 256 includes a collection of connected nodes that are capable of sending signals from one node to another. For example, the multi-task neural network 256 can include several different layers of such nodes. As described in further detail below, in some embodiments, the multi-task neural network 256 can include encoder layers and decoder layers. In some embodiments, one or more encoder layers of the multi-task neural network 256 can be stored in a non-transitory memory associated with a first set of one or more processors, while one or more decoder layers of the multi-task neural network 256 can be stored in a non-transitory memory associated with a second set of one or more processors that are disposed separately from the first set of one or more processors but communicatively coupled to one another. For example, the first set of one or more processors can include one or more processors disposed within a head-mounted device, while the second set of one or more processors can include one or more processors disposed within a computing device that is physically connected to the head-mounted device, one or more electronic devices (e.g., a smartphone, a computer, a peripheral device, a server, a cloud computing device, etc.) that are physically separate from the head-mounted device, or a combination thereof. The training and use of the multi-task neural network 256 is described further below.

[0087] Figure 5 A schematic diagram of a system for computing a gaze vector incorporating the multi-task neural network 256 is shown. In some embodiments, an input image I(x, y, c) is captured by an off-axis camera 260 and provided as input to the multi-task neural network 256. The input image I(x, y, c) can have H x W x C dimensions, where H is the number of pixels in the vertical direction, W is the number of pixels in the horizontal direction, and C is the number of channels of the image (e.g., equal to 3 for an RGB image and equal to 1 for a grayscale image). The multi-task neural network 256 can process the input image I(x, y, c) and can generate network output data 264 based on the input image I(x, y, c).

[0088] When the AR device 200 is operating in a runtime mode, the network output data 264 can be used in conjunction with the calibration angle K to compute the gaze vector 238. In some embodiments, a post-processing block 266 can perform one or more operations to compute the gaze vector 238. In other embodiments, or in the same embodiments, the calibration angle K can be provided as input to the multi-task neural network 256 along with the input image I(x, y, c), and the gaze vector 238 can be included directly in the network output data 264 or can be computed based on the network output data 264.

[0089] When the AR device 200 is operating in the training mode, the network output data 264 can be compared to the GT data 268. Error data 270 can be computed based on the comparison, and can represent a difference between the network output data 264 and the GT data 268, such that in some embodiments the magnitude of the error data 270 can be proportional to the difference between the network output data 264 and the GT data 268. The multi-task neural network 256 can be modified based on the error data 270 (e.g., using the modifier 272). In some embodiments, the magnitude of the modification to the multi-task neural network 256 can be proportional to the magnitude of the error data 270, such that a larger difference between the network output data 264 and the GT data 268 can correspond to a larger modification to the multi-task neural network 256.

[0090] In some embodiments, some or all of the operations described herein as being associated with the training mode can be performed independent of the AR device 200. For example, in such embodiments, the multi-task neural network 256 can be at least partially trained prior to manufacturing and / or distribution of the AR device 200, and subsequently loaded onto the AR device 200 at the time of manufacturing and / or distribution of the AR device 200. In at least some of these embodiments, the multi-task neural network 256 can be at least partially trained with data from a relatively large number of subjects and by one or more computing devices other than the AR device 200. In some such embodiments, the AR device 200 can perform one or more of the operations described herein as being associated with the training mode in order to further train the pre-loaded multi-task neural network 256 with data from the particular user of the AR device 200. This can allow one or more portions of the multi-task neural network 256 to become personalized to each user of the AR device 200. In some embodiments, the AR device 200 can store a personalized version of the multi-task neural network 256 for each user of the AR device 200. Thus, in these embodiments, the AR device 200 can store multiple different versions of the multi-task neural network 256 for multiple different users, and can use the version of the multi-task neural network 256 associated with the current user of the AR device 200 at runtime.

[0091] Figure 6A schematic diagram of a multi-task neural network 256 is shown, which includes various layers 257. In some embodiments, the multi-task neural network 256 includes a feature encoding base network consisting of a feature encoding layer 258 (alternatively referred to as an encoder layer) and six task branches consisting of task-specific layers 294 (alternatively referred to as a decoder layer). The six task branches correspond to (1) pupil center estimation and glint localization to generate 2D pupil center data 274, (2) semantic segmentation of eye parts to generate eye segmentation data 276, (3) pupil and glint presence classification to generate glint detection data 278, (4) 2D cornea estimation to generate cornea center data 280, (5) blink detection to generate blink prediction 296, and (6) emotional expression classification to generate eye gaze classification 298.

[0092] Network output data 264 may include Figure 6 One or more types of data may be generated based on the AR device 200. Based on whether the AR device 200 is operating in training mode or runtime mode, one or more types of data may not be used in subsequent processing. Alternatively or additionally, one or more types of data may not be generated by the multi-task neural network 256 to save processor usage, power, and / or memory. Alternatively or additionally, one or more types of data may not be generated based on user input. For example, certain applications operating on the AR device 200 may request that only specific types of data be generated, such as gaze classification 298.

[0093] In some embodiments, the feature encoding layer 258 can produce encoder features 282 that are shared across each task branch. In some implementations, an image feature extraction network and a feature pyramid (FPN) are used to capture information from different scales. In some implementations, the top-most features from the encoder (e.g., with a size of 20×15×256) can be used as input to the task branch.

[0094] In some embodiments, the multi-task neural network 256 includes three main appearance-based tasks in the multi-task learning model, which include: (1) eye part segmentation, (2) pupil and glint localization, and (3) pupil and glint existence classification. In some embodiments, eye part segmentation is defined as the task of assigning a class label from the following: background, sclera, iris, and pupil, to each pixel in the input image I(x, y, c). For this task, encoder features 282 corresponding to the last layer of features maps can be obtained from the encoder network (e.g., feature encoding layers 258) and upsampled to the same resolution as the input image I(x, y, c) using deconvolution layers. The resulting four-channel output can be converted to class probabilities independently for each pixel using a softmax layer. The loss can be the cross-entropy loss between the predicted probability distribution and the one-hot label obtained from the human-annotated ground truth.

[0095] In some embodiments, for the k-th class with GT class c, and the predicted probability p k (x, y) of a pixel x, y, the following loss is minimized:

[0096]

[0097] where I x,y [.] is the indicator function. The total loss can be the sum of the losses over all pixels in the image. The segmentation task serves as a bootstrapping phase for training the feature encoder layers 258, as it captures rich semantic information of the eye images. Separately, the eye part segmentation can aid the initial stages of any classic pipeline in localizing the search for glints (using the iris boundary) and estimating the pupil center (using the pupil boundary). In some implementations, the eye part segmentation can be useful for rendering digital avatars’ eyes.

[0098] The pupil and glint localization branch provides four pixel locations of glints and one pupil center, for a total of five keypoints. The network decoder layers for these two tasks (which can be similar to the eye part segmentation branch) can predict a set of five dense maps at the output corresponding to the five keypoints. Each dense map can be normalized to sum to unity over all pixels. Then, the cross-entropy loss can be computed over all pixels of each map during training. Once trained, the location of a particular glint or the center of the pupil is the pixel corresponding to the maximum probability at the output. In some embodiments, for each keypoint (four glints and one pupil center), the following loss is minimized:

[0099]

[0100] where I[.] is an indicator function that is zero everywhere except at the GT keypoint locations, p x,y is the predicted probability of a keypoint location, and the sum is over all pixels in the image.

[0101] In a real-world setting, glints and / or the pupil center can be occluded by closed eyelids, unwanted reflections can manifest as glints, and / or glints can not appear on the reflective corneal surface for certain gaze angles. Thus, it can be important to learn to robustly classify the presence or absence of glints and the pupil center. These predictions can effectively control whether glints should be used for corneal center estimation and similarly for 3D pupil center estimation.

[0102] For this task, encoder features 282 corresponding to the topmost feature map can be obtained from the encoder network (e.g., feature encoding layers 258), a convolutional layer can be used to reduce the number of feature channels, the reduced number of feature channels can be reshaped into a one-dimensional array, and a trainable fully connected layer (e.g., of size 1500 x 10) can be added to produce the output (e.g., of size 5 x 2). Each pair can represent one of four glints and / or the probability of the presence or absence of the pupil center. Binary cross-entropy loss can be used to learn from artificially labeled annotations.

[0103] With respect to corneal center estimation, the center of the cornea is a 3D geometric quantity that cannot be observed in a 2D image of the eye. Thus, unlike the pupil (pupil ellipse center) or glint landmarks, it is not possible to directly hand-label the projected location of the 3D corneal center on the image. Thus, a two-step approach can be employed to train the corneal 2D center prediction branch of the multi-task neural network 256. First, well-known geometric constraints and related known / estimated quantities (LEDs, glints) can be used to generate corneal 2D supervision. Then, this model-based supervision can be used to train the 2D corneal branch using the obtained for each frame.

[0104] Using the multi-task neural network 256 to predict the cornea relative to using geometric constraints during evaluation has two main benefits. First, this prediction is more robust because deep networks have a tendency to average out noise during training, whereas standard network- outside optimization can sometimes not converge. Second, since the corneal task branch consists of only a few fully connected layers, this prediction can only incur a small and constant time feed-forward computation.

[0105] The facial expression classification task involves classifying the emotional expression of a user from an input eye image. This task is particularly challenging because only the eye region of the user is available as input, not the eyebrows and / or the entire face as used in most emotion facial expression classification benchmarks. In some embodiments, the following individual emotional facial expressions are considered: happy, angry, disgusted, fearful, and surprised. These expressions can be grouped into 4 discrete dimensions: positive dimension (happy) discriminative dimension (angry and disgusted) sensitive dimension (fearful and surprised), and neutral dimension. As with other task branches, the feature encoding layer 258 is fixed and only the facial expression task branch (composed of multiple FC layers) is trained for expression classification. In some embodiments, this task branch is trained for each subject to produce a personalized model that yields better accuracy than a general model for a large number of subjects.

[0106] In some embodiments, the network output data 264 can include 2D pupil center data 274. In some embodiments, the 2D pupil center data 274 can include a 2D pupil center represented as a 2D value. For example, the 2D pupil center can include X and Y values corresponding to the computed location of the center of the pupil (e.g., pupil center 308) within a frame of the input image I(x,y,c). Alternatively or additionally, the 2D pupil center data 274 can include a matrix of HxW dimensions including binary values of 0 or 1 (value 1 corresponding to the computed location of the center of the pupil).

[0107] In some embodiments, the network output data 264 can include eye segmentation data 276. The eye segmentation data 276 can include a segmentation of the eye into a plurality of regions. In one particular implementation, the regions can include a background region, a sclera region, a pupil region, and an iris region. In another particular implementation, the regions can include a pupil region and a non-pupil region. In another particular implementation, the regions can include a pupil region, an eye region (including portions of the eye that are not part of the pupil region), and a background region.

[0108] In some embodiments, the eye segmentation data 276 can include a matrix of HxW dimensions including a finite set of values, such as 0, 1, 2, and 3 (corresponding to, e.g., a background region, a sclera region, a pupil region, and an iris region, respectively). In some embodiments, the eye segmentation data 276 includes an assignment of each pixel of the input image I(x,y,c) to a set of classes including background, sclera, pupil, and iris, which can be obtained in some embodiments by taking the last layer of the (decoder) multi-task neural network 256 and upsample it to the same resolution as the input image I(x,y,c) using deconvolution, which in turn is fed to a softmax cross-entropy loss over the feature channels, where each feature channel represents the probability of a pixel belonging to a certain class.

[0109] In some embodiments, the network output data 264 can include glint detection data 278. In some embodiments, the glint detection data 278 includes one or more glint locations represented as 2D or 3D values. For example, if only a single glint location is detected, the glint detection data 278 can include a single 2D value, or if four glint locations are detected, the glint detection data 278 can include four 2D values. In some embodiments, the glint detection data 278 can include X and Y values within a frame of the input image I(x,y,c) corresponding to the computed locations of detected glints. Alternatively or additionally, the glint detection data 278 can include a matrix with HxW dimensions including binary values of 0 or 1 (value 1 corresponding to the location of a detected glint).

[0110] In some embodiments, the network output data 264 can include corneal center data 280. In some embodiments, the corneal center data 280 can include a 2D corneal center represented as a 2D value or a 3D corneal center represented as a 3D value. For example, a 2D corneal center can include X and Y values within a frame of the input image I(x,y,c) corresponding to the computed location of the center of the cornea (e.g., corneal center 306). Alternatively or additionally, the corneal center data 280 can include a matrix with HxW dimensions including binary values of 0 or 1 (value 1 corresponding to the computed location of the corneal center).

[0111] In some embodiments, the network output data 264 can include blink prediction 296. In some embodiments, the blink prediction 296 includes binary values of 0 or 1 (e.g., corresponding to a prediction of eye open and eye blink, respectively). In some embodiments, the blink prediction 296 includes a probability associated with whether a blink occurred. Detecting blinks is a task that can be used to drive multi-focal displays and / or digital avatars. Blinks can be captured over a series of images so that temporal information can be used to distinguish blinks from events such as saccades (rapid lateral movements of the eyes).

[0112] In general, it can be difficult to accurately locate blink events where the eyes are fully closed, especially at standard frame rates of 30 frames per second. In other cases, it can be important to detect the onset of a blink to reduce latency between detection and application. In some embodiments, a simple definition of a blink can be the state of the eye when the upper eyelid covers more than 50% of the entire pupil area. This can be a useful working definition for non-expert human labelers. Given the blink definition above, the encoder features 282 generated by the feature encoding layer 258 trained for tasks such as eye segmentation transfer well to the blink detection task. In some embodiments, the topmost layer (shared representation) of a pre-trained feature encoding network (e.g., the feature encoding layer 258) is used to train the blink detection branch.

[0113] Figure 7 Systems and techniques are shown for generating blink predictions 296 using features from different time steps. In the illustrated embodiment, encoder features from three consecutive time steps T-2, T-1, and T are fed as inputs a T-2 , a T-1 , and a T to a three-layer fully connected network that classifies the current frame (at time T) as a blink or a wide open eye and produces an output y T indicating a blink or a wide open eye. While longer time window lengths can be employed, they result in diminishing returns in prediction accuracy. Recurrent neural networks (RNNs) and long short-term memory (LSTM) have similar training and testing performance, however, network 700 provides lower computational requirements.

[0114] Figure 8 A schematic diagram of AR device 200 operating in a training mode is shown. When AR device 200 operates in the training mode, network output data 264 includes eye segmentation data 276, glint detection data 278, and cornea center data 280. The particular input images I(x,y,c) used to generate these network outputs can also be manually inspected by one or more individuals, who can prepare GT data 268 before, after, or concurrently with generation of network output data 264 by multi-task neural network 256. For example, an individual can inspect a displayed version of input image I(x,y,c) on an electronic device, such as a personal computer or a smartphone. A program or application on the electronic device can ask the individual a set of questions related to input image I(x,y,c), and the individual can input his / her responses using an input device, such as a mouse, a keyboard, a touchscreen, etc.

[0115] While observing and inspecting input image I(x,y,c), an individual can prepare 2D pupil center GT data 283 by identifying the outline of the pupil using an input device. This can include the individual placing an elliptical boundary on the pupil and causing the pupil center to be automatically computed based on the placed elliptical boundary. 2D pupil center GT data 283 can be prepared so as to have the same format and dimensions as 2D pupil center data 274 (e.g., X and Y values). Further, while observing and inspecting input image I(x,y,c), an individual can prepare eye segmentation GT data 284 by deciding that a first region of the image should be assigned as a background region, a second region should be assigned as a sclera region, a third region should be assigned as a pupil region, a fourth region should be assigned as an iris region. Eye segmentation GT data 284 can be prepared so as to have the same format and dimensions as eye segmentation data 276 (e.g., a matrix with HxW dimensions that includes a finite set of values, such as 0, 1, 2, and 3 corresponding to different regions).

[0116] Further, in observing and inspecting the input image I(x,y,c), the individual can prepare the glint detection GT data 286 by deciding how many glint locations are present in the input image I(x,y,c) and the location of each glint location. The glint detection GT data 286 can be prepared to have the same format and size as the glint detection data 278 (e.g., a set of 2D values), or if some number of glint locations are detected (e.g., four), the glint detection GT data 286 can include that number of 2D values. In some embodiments, the glint detection GT data 286 can include X and Y values corresponding to the calculated locations of detected glints within a frame of the input image I(x,y,c). Alternatively or additionally, the glint detection GT data 286 can include a matrix having HxW dimensions that includes binary values of 0 or 1 (value 1 corresponding to the location of a detected glint).

[0117] In one particular implementation, the GT data 268 can be obtained by having an individual or a group of individuals face a 3x3 dot grid at two different depths (a near depth at, e.g., 3 meters and a further plane at, e.g., 6 meters). Under a given prompt, the individual is asked to focus their gaze on one of the 18 3D points, which allows for the GT data 268 for the gaze vector 238 to be collected for each frame (to determine overall accuracy later). The captured images of the individual’s eyes can be analyzed (using a camera of an AR device worn by the individual) to allow the GT data 268 to include eye segmentation and glint location information. Since there is diminishing returns in annotating segmentation, glint and pupil center for each frame at 30 or 60 Hz, a number (e.g., 200) of left or right eye image frames can be uniformly sampled for each individual to manually annotate segmentation, glint presence or absence, glint 2D location and pupil 2D location. In one particular experimental run, 87,000 annotated images were used in a dataset to train and validate the performance of the multi-task neural network 256.

[0118] In some embodiments, the error data 270 can include first error data 270A computed based on a difference between the 2D pupil center data 274 and 2D pupil center GT data, second error data 270B computed based on a difference between the eye segmentation data 276 and eye segmentation GT data 284, third error data 270C based on a difference between the glint detection data 278 and glint detection GT data 286, and fourth error data 270C generated by a geometric constraint engine 288. Inputs to the geometric constraint engine 288 include one or more of cornea center data 280, glint detection data 278, emitter position data 290, and camera intrinsic parameters 291. The emitter position data 290 can include fixed positions of the emitters 262 and / or emission directions of the emitters 262. The emitter position data 290 can be determined at the time of manufacturing of the AR device 200 and / or during a calibration phase. The camera intrinsic parameters 291 can include optical centers and / or focal lengths of the off-axis cameras 260, among other possibilities. The camera intrinsic parameters 291 can be determined at the time of manufacturing of the off-axis cameras 260 and / or during a calibration phase.

[0119] The geometric constraint engine 288 can perform various operations to assess consistency between different generated data (glint detection data 278 and cornea center data 280) and calibration data (emitter position data 290), the output of the geometric constraint engine 288, i.e., the fourth error data 270D, can be inversely related to a likelihood or consistency parameter. In some instances, a corneal sphere 304 is reconstructed using the glint detection data 278 and the emitter position data 290, and the fourth error data 270D is set as a distance between a center of the computed reconstructed sphere and the cornea center as indicated by the cornea center data 280.

[0120] In some embodiments, training of the multi-task neural network 256 is improved by sequentially training using only certain outputs of the multi-task neural network 256 during different training iterations. In a first training step, the multi-task neural network 256 is trained using only the eye segmentation data 276. This can be done by modifying the multi-task neural network 256 using only the second error data 270B. Once the multi-task neural network 256 is sufficiently trained (i.e., accurate enough) for eye segmentation, a second training step is performed by additionally training the multi-task neural network 256 using the glint detection data 278. This can be done by modifying the multi-task neural network 256 using only the third error data 270C. Once the multi-task neural network 256 is sufficiently trained for eye segmentation and glint detection, a third training step is performed by additionally training the multi-task neural network 256 using the corneal center data 280. This can be done by modifying the multi-task neural network 256 using all of the error data 270. In some instances, the same training images and GT data can be used during the different training steps. In some embodiments, the AR device 200 remains in training mode until an accuracy threshold is met or a maximum iteration threshold is met (e.g., a number of training images used meets an iteration threshold).

[0121] Figure 9A and Figure 9B A diagram illustrating sequential training steps 902 for training the multi-task neural network 256 is shown. Referring to Figure 9A , a first training step 902-1 is shown. During the first training step 902-1, the feature encoding layer 258 and the task-specific layer 294-2 (corresponding to the decoder layer used to generate the eye segmentation data 276) are trained independent of the remaining task-specific layers 294. For example, during a training iteration, an input image I(x,y,c) can be provided to the multi-task neural network 256 and can also be presented to a person who can prepare eye segmentation GT data 284. Using the second error data 270B computed based on a difference between the eye segmentation data 276 and the eye segmentation GT data 284, the modifier 272 can modify the weights associated with the feature encoding layer 258 and the task-specific layer 294-2 (e.g., using backpropagation) such that the second error data 270B is reduced during subsequent computations of the second error data 270B based on a difference between the eye segmentation data 276 and the eye segmentation GT data 284. During the first training step 902-1, the modifier 272 does not modify the weights associated with the task-specific layers 294-1, 294-3, 294-4, 294-5, or 294-6.

[0122] Referring to Figure 9B, a second training step 902-2 is shown. In some embodiments, the second training step 902-2 is performed after the first training step 901-1. During the second training step 902-2, one or more of the task-specific layers 294-1, 294-3, 294-4, 294-5, and 294-6 are trained independently of the feature encoding layer 258 and the task-specific layer 294-2. For example, during a first training iteration, an input image I(x, y, c) may be provided to the multi-task neural network 256 and may also be presented to an individual who may prepare the associated ground truth data 268. Using the error data 270 calculated based on the difference between the network output data 264 and the ground truth data 268, the modifier 272 may modify the weights associated with the task-specific layers 294-1, 294-3, 294-4, 294-5, and / or 294-6 (e.g., using backpropagation) to reduce the error data 270 during subsequent calculations of the error data 270. During the second training step 902 - 2 , the modifier 272 does not modify the weights associated with the feature encoding layer 258 or the task-specific layer 294 - 2 , although in some embodiments, the task-specific layer 294 - 2 may be fine-tuned during the second training step 902 - 2 , as shown by the dashed line.

[0123] In some embodiments, task-specific layer 294-1, task-specific layer 294-2, and task-specific layer 294-3 may each include one or more convolutional layers and one or more deconvolutional layers. In some embodiments, task-specific layer 294-4 and task-specific layer 294-6 may be architecturally similar or identical to each other, but may be trained as two separate branches. In at least some of these embodiments, task-specific layer 294-4 and task-specific layer 294-6 may each include one or more convolutional layers. Additionally, in some embodiments, task-specific layer 294-5 may be architecturally similar or identical to that of neural network 700, as described above with reference to FIG. Figure 7 described.

[0124] like Figure 6 As shown, some outputs of the multi-task neural network 256 can be obtained with fewer executed operations than other outputs of the multi-task neural network 256. For example, the cornea center data 280 can be obtained with fewer computations than other outputs, and the eye segmentation data 276 can be obtained with more computations than other outputs. Therefore, one advantage of first training the multi-task neural network 256 using the eye segmentation data 276 is that some layers used only to compute the eye segmentation data 276 can be fine-tuned without being affected by feedback from other outputs.

[0125] Figure 10A schematic diagram of the AR device 200 operating in the runtime mode is shown. When the AR device 200 operates in the runtime mode, the network output data 264 can include eye segmentation data 276, glint detection data 278, and cornea center data 280. These outputs can be used in conjunction with the calibration angle kappa to compute the gaze vector 238 using the post-processing block 266. In some embodiments, the post-processing block 266 can be split into a first post-processing block 266A, a second post-processing block 266B, and a third post-processing block 266C. The first post-processing block 266A receives the 2D pupil center data 274 and the eye segmentation data 276 as inputs and computes the 3D pupil center 292. The second post-processing block 266B receives the 3D pupil center 292 and the cornea center data 280 as inputs and computes the optical axis 310. The third post-processing block 266C receives the optical axis 310 and the calibration angle kappa as inputs and computes the gaze vector 238.

[0126] The accuracy of the multi-task neural network 256 has been demonstrated, e.g., as described in U.S. Provisional Application No. 62 / 935,584. The following table shows one example of accuracy for eye segmentation, which provides eye segmentation confusion matrix percentage values, where the average accuracy for all four classes is over 97.29%.

[0127] GT / Pred Pupil Iris Sclera BG Pupil 96.25 3.75 0.00 0.00 Iris 0.04 99.03 0.93 0.00 Sclera 0.00 3.27 96.71 0.02 BG 0.01 0.72 2.09 97.18

[0128] These results are very accurate both in quantitative and qualitative assessment. This can be important because the segmentation boundaries can be used to generate precise pupil 2D center position training data through a carefully adjusted ellipse fitting process, especially for the case of partially occluded pupils. The segmentation predictions can also be used by a classical geometric pipeline, which can be used as a baseline for gaze estimation comparisons.

[0129] As another example, the following table shows accuracy for pupil and glint detection, which shows quantitative results for predicting pixel locations using each of the multi-task neural network 256 (“NN 256”) and the classical pipeline.

[0130]

[0131] When the images are from ideal settings, both the multi-task neural network 256 and the classical predictions are accurate, with errors close to zero. However, when the images have severe reflections or the user gaze is far from the central target, the multi-task neural network 256 is able to first very accurately detect the presence or absence of glints and provide a robust labeling of glints, while the classical approach has poor absence indication and false labeling of glints, resulting in much higher errors under the Euclidean error metric.

[0132] Figure 11 A schematic diagram of a gaze vector neural network 1102 is shown, which can generate gaze vectors 238 based on the calibration angle K and the optical axis 310. In some embodiments, the gaze vector neural network 1102 can replace or be incorporated into the post-processing block 266C. In one implementation, the gaze vector neural network 1102 includes 5 layers and approximately 30,000 parameters or weights. In some embodiments, the gaze vector neural network 1102 is trained only on calibration frames.

[0133] During training, the gaze vector 238 can be compared to gaze vector GT data 1104. Error data 1106 can be computed based on the comparison, and can represent a difference between the gaze vector 238 and the gaze vector GT data 1104, such that in some embodiments, a magnitude of the error data 1106 can be proportional to a difference between the gaze vector 238 and the gaze vector GT data 1104. The gaze vector neural network 1102 can be modified based on the error data 1106 (e.g., using modifier 1108). In some embodiments, a magnitude of the modification to the gaze vector neural network 1102 can be proportional to a magnitude of the error data 1106, such that a larger difference between the gaze vector 238 and the gaze vector GT data 1104 can correspond to a larger modification to the gaze vector neural network 1102.

[0134] In some embodiments, the gaze vector GT data 1104 can be obtained by a user looking at a target generated on a screen. For example, a user can wear an AR device 200, which can include the previously trained multi-task neural network 256. During a training iteration, the user can be instructed to look at a target located on a display while wearing the AR device 200. Input images I(x,y,c) of the user’s eyes can be captured and used to generate the optical axis 310. Based on the optical axis 310 (and optionally based on the calibration angle K), a gaze vector 238 can be generated by the gaze vector neural network 1102. The gaze vector GT data 1104 can be determined based on a relationship between the wearable device and the target generated on the display. For example, an orientation between the AR device 200 and the display can be determined based on one or more sensors, such as a camera and / or an inertial measurement unit, and the determined orientation can be used to compute an actual gaze vector of the user’s eyes.

[0135] During subsequent training iterations, the target can be moved to a new location on the display, new input images I(x,y,c) of the user's eyes can be captured, and the gaze vector neural network 1102 can be modified using the newly computed error data 1106. During various training iterations, the target can be moved to various locations on the screen in order to train the gaze vector neural network 1102 to robustly estimate gaze vectors over a large range of gaze angles. In some embodiments, various lighting conditions and / or user moods can be employed during the training process to be combined with various gaze vectors, resulting in a robustly trained network.

[0136] An example of the accuracy of gaze estimation can be demonstrated by the following table, which shows the gaze error for each of 9 targets clustered on different target planes, where the overall gaze estimation metric is defined as the angular error (e.g., in arcminutes) between the true gaze vector and the estimated gaze vector.

[0137]

[0138]

[0139] It is clear that the estimation using the multi-task neural network 256 and the gaze vector neural network 1102 is significantly better and similar in all directions. This can be primarily attributed to the robust glint and cornea 2D estimation along with the use of the gaze vector neural network 1102.

[0140] Figure 12 A training pipeline 1200 is shown in accordance with some embodiments of the present application. In some instances, the full training can take several steps, as the framework receives GT from different sources and because the model-based supervision uses estimates from the training network itself. For example, the model first trains the eye segmentation and glint prediction, then uses the trained model to predict glints on all unlabelled data. Next, using these predicted glints and the known position of the LEDs, the cornea position is inferred based on standard eye models and geometry. Since the supervision of the training model to predict cornea prediction comes from using previously trained models along with standard eye models and geometry, this technique can be referred to as model-based supervision.

[0141] At step 1202, the encoder-decoder network is first trained with eye segmentation labels (e.g., eye segmentation GT data 284), as it provides the richest semantic information and is the most complex supervised task to train accurately.

[0142] At step 1204, all supervised tasks are trained. Further, at step 1204, human-labeled glint data (e.g., glint detection GT data 286), pupil 2D center data (2D pupil center GT data 283), and eye segmentation data (e.g., eye segmentation GT data 284) can be used together to jointly train each of these three supervised tasks. In some instances, initialization with weights trained from eye segmentation can result in more stable training than random initialization.

[0143] At step 1206, glint predictions (e.g., glint detection data 278) are made for all frames, and these glint predictions are used together with known locations of LEDs (e.g., emitter location data 290) to generate cornea 2D GT (generated within geometry constraint engine 288) at step 1208 for training the cornea branch (e.g., using fourth error data 270D) at step 1210. It should be noted that the cornea branch is trained with data from the entire training set population, and is further personalized (fine-tuned) at each subject calibration phase.

[0144] After 3D pupil center is predicted at step 1212, the optical axis is inferred at step 1214 using the predicted cornea (personalized) and pupil 3D centers from the calibration frames. Using gaze target GT, the gaze vector neural network 1102 is trained at step 1216 to convert the optical axis to the visual axis. During runtime, the predicted cornea and pupil 2D centers are obtained from the multi-task neural network 256. These quantities are used to lift to 3D to obtain the optical axis, which is then fed into the gaze mapping network to infer the predicted gaze direction.

[0145] The blink and facial expression classification tasks are trained on top of the intermediate features of the main feature encoding branch. Blink detection is a temporal task that requires capturing three consecutive eye images and extracting their intermediate features. With a set of pre-computed features, the blink detection branch is trained separately while the main feature encoding branch of the multi-task neural network 256 remains frozen. A similar procedure is followed during runtime. For facial expression classification, the main feature encoding branch is frozen and only the expression classification layers are trained using the expression data. Expression prediction is produced during runtime along with all other tasks.

[0146] Figure 13A method 1300 is shown for training a neural network (e.g., the multi-task neural network 256) having a set of feature encoding layers (e.g., the feature encoding layer 258) and a plurality of sets of task-specific layers (e.g., the task-specific layer 294), each set of task-specific layers operating on the output of the feature encoding layer set (e.g., the encoder features 282). The steps of the method 1300 need not be performed in the order shown, and one or more steps of the method 1300 may be omitted during execution of the method 1300. In some embodiments, one or more steps of the method 1300 may be performed by the processing module 250 or some other component of the AR device 200.

[0147] At step 1302, a first training step (e.g., first training step 902-1) is performed. In some embodiments, the first training step is performed during a first duration. In some embodiments, step 1302 includes steps 1304, 1306, and / or 1308.

[0148] At step 1304, a first image of the first eye (e.g., input image I(x, y, c)) is provided to the neural network. In some embodiments, the first image is captured by and / or received from a camera (e.g., off-axis camera 260). In some embodiments, method 1300 includes the step of capturing the first image of the first eye using the camera. In some embodiments, method 1300 includes the step of sending the first image of the first eye from the camera to a processing module (e.g., processing module 250).

[0149] At step 1306, based on the first image, a neural network is used to generate eye segmentation data (eg, eye segmentation data 276). In some embodiments, the eye segmentation data includes segmenting the first eye into a plurality of regions.

[0150] At step 1308, the feature encoding layer set is trained using the eye segmentation data. In some embodiments, during the first training step, a single task-specific layer set from the plurality of task-specific layer sets is also trained using the eye segmentation data. In some embodiments, error data (e.g., error data 270B) is calculated based on the difference between the eye segmentation data and the eye segmentation GT data (e.g., eye segmentation GT data 284). In some embodiments, the error data is used to train the feature encoding layer set.

[0151] At step 1310, a second training step (e.g., second training step 902-2) is performed. In some embodiments, the second training step is performed during a second duration. In some embodiments, the second duration is after the first duration. In some embodiments, step 1310 includes steps 1312, 1314, and / or 1316.

[0152] At step 1312, a second image of a second eye (e.g., input image I(x,y,c)) is provided to the neural network. The second eye can be the same or different than the first eye. In some embodiments, the second image is captured by and / or received from a camera. In some embodiments, the method 1300 includes a step of capturing the second image of the second eye using a camera. In some embodiments, the method 1300 includes a step of sending the second image of the second eye from the camera to the processing module.

[0153] At step 1314, based on the second image, network output data (e.g., network output data 264) is generated using each of the set of feature encoding layers and the plurality of sets of task-specific layers.

[0154] At step 1316, the plurality of sets of task-specific layers are trained using the network output data. In some embodiments, the set of feature encoding layers are not trained during the second training step. In some embodiments, error data (e.g., error data 270) is computed based on a difference between the network output data and GT data (e.g., GT data 268). In some embodiments, the error data is used to train the plurality of sets of task-specific layers.

[0155] Figure 14 A method 1400 of training a neural network (e.g., multi-task neural network 256) for classifying user eye gazes is shown. The steps of the method 1400 need not be performed in the order shown, and one or more steps of the method 1400 can be omitted during performance of the method 1400. In some embodiments, one or more steps of the method 1400 can be performed by the processing module 250 or some other component of the AR device 200.

[0156] At step 1402, an image of an eye (e.g., input image I(x,y,c)) is captured. In some embodiments, the first image is captured by and / or received from a camera (e.g., off-axis camera 260). In some embodiments, the method 1400 includes a step of capturing the image of the eye using a camera. In some embodiments, the method 1400 includes a step of sending the image of the eye from the camera to a processing module (e.g., processing module 250).

[0157] At step 1404, the image of the eye is provided to the neural network. In some embodiments, providing the image of the eye to the neural network can include providing data representing the image of the eye as input to a set of operations implementing the neural network.

[0158] At step 1406, a gaze classification corresponding to the eye (e.g., gaze classification 298) is generated by the neural network. In some embodiments, the gaze classification is one of a plurality of possible gaze classifications.

[0159] At step 1408, a GT gaze classification is determined (e.g., GT data 268). In some embodiments, determining the GT gaze classification includes receiving user input indicating the GT gaze classification. For example, a user can indicate that they are exhibiting a “happy” expression through an input device. In some embodiments, determining the GT gaze classification includes determining that an instruction communicated to the user indicates the GT gaze classification. For example, a user can be instructed to exhibit a “happy” facial expression through a display device.

[0160] At step 1410, error data is calculated based on a difference between the gaze classification and the GT gaze classification (e.g., error data 270).

[0161] At step 1412, the neural network is modified based on the error data. In some embodiments, modifying the neural network includes modifying a set of weights of the neural network. In some embodiments, the set of weights can be modified using backpropagation. In some embodiments, a set of task-specific layers (e.g., task-specific layers 294-6) of the neural network can be modified based on the error data.

[0162] Figure 15 A method 1500 of training a neural network (e.g., gaze vector neural network 1102) for calculating a gaze vector (e.g., gaze vector 238) is shown. The steps of method 1500 need not be performed in the order shown, and one or more steps of method 1500 can be omitted during performance of method 1500. In some embodiments, one or more steps of method 1500 can be performed by processing module 250 or some other component of AR device 200.

[0163] At step 1502, an image of an eye is captured (e.g., input image I(x,y,c)). In some embodiments, the first image is captured by and / or received from a camera (e.g., off-axis camera 260). In some embodiments, method 1500 includes a step of capturing an image of an eye using a camera. In some embodiments, method 1500 includes a step of sending an image of an eye from a camera to a processing module (e.g., processing module 250).

[0164] At step 1504, the image of the eye is processed to produce an optical axis corresponding to the eye. In some embodiments, processing the image of the eye can include generating 2D pupil center data (e.g., 2D pupil center data 274), eye segmentation data (e.g., eye segmentation data 276), and / or cornea center data (e.g., cornea center data 280) using a multi-task neural network (e.g., multi-task neural network 256).

[0165] The optical axis is provided to the neural network at step 1506. In some embodiments, providing the optical axis to the neural network may include providing data representing the optical axis as input to a set of operations implementing the neural network.

[0166] At step 1508, a gaze vector corresponding to the eye is generated by the neural network. In some embodiments, the gaze vector includes at least one angle.

[0167] At step 1510, gaze vector GT data (e.g., gaze vector GT data 1104) is determined. In some embodiments, the gaze vector GT data is determined based on the location of the target displayed on the screen. In some embodiments, determining the gaze vector GT data includes receiving user input indicating the gaze vector GT data. For example, the user may be looking at a particular target among multiple targets displayed on the screen and provide input regarding which target the user is looking at.

[0168] At step 1512 , error data (eg, error data 1106 ) is calculated based on the difference between the gaze vector and the gaze vector GT data.

[0169] At step 1514, the neural network is modified based on the error data. In some embodiments, modifying the neural network includes modifying a set of weights of the neural network. In some embodiments, backpropagation can be used to modify the set of weights.

[0170] Figure 16 A method 1600 for calculating a gaze vector using a neural network is shown. The steps of method 1600 need not be performed in the order shown, and one or more steps of method 1600 may be omitted during execution of method 1600. In some embodiments, one or more steps of method 1600 may be performed by processing module 250 or some other component of AR device 200.

[0171] At step 1602, an input image of a user's eye (e.g., input image I(x, y, c)) is received. In some embodiments, the input image is received from a camera (e.g., off-axis camera 260). The camera can be mounted on an optical device and / or can be a component of an optical device. In some embodiments, method 1600 includes the step of capturing the input image of the user's eye using the camera. In some embodiments, method 1600 includes the step of sending the input image from the camera to a processing module (e.g., processing module 250).

[0172] At step 1604, the input image of the eye is provided to a neural network (e.g., the multi-task neural network 256). In some embodiments, the input image is provided to a processor that implements the neural network. The processor can be a specialized processor (e.g., a neural network processor) that has an architecture that allows certain operations (e.g., convolutions, matrix multiplications) that are typically performed by a neural network to be performed faster than with a general-purpose processor. For example, the specialized processor can include a systolic array having a plurality of processing elements for performing various arithmetic operations on different pixels of the input image in parallel or simultaneously.

[0173] At step 1606, network output data (e.g., the network output data 264) is generated using the neural network. The network output data can include data corresponding to the overall output of the neural network as well as the outputs of intermediate layers of the neural network. For example, the network output data can include certain data (e.g., the eye segmentation data 276) derived from the overall output of the neural network and certain data (e.g., the blink prediction 296 and the corneal center data 280) derived from the outputs of intermediate layers of the neural network. Additionally or alternatively, the network output data can include certain data (e.g., the glint detection data 278 and the 2D pupil center data 274) derived from the outputs of different intermediate layers of the neural network as well as one or more additional layers that do not involve processing of the overall output of the neural network.

[0174] At step 1608, a 3D pupil center (e.g., the 3D pupil center 292) is computed based on the network output data. In some embodiments, the 3D pupil center is computed based on the 2D pupil data and the eye segmentation data.

[0175] At step 1610, an optical axis (e.g., the optical axis 310) associated with the eye of the user is computed based on the network output data. In some embodiments, the optical axis is computed based on the 3D pupil center and certain data (e.g., the corneal center data 280) in the network output data.

[0176] At step 1612, a gaze vector corresponding to the eye (e.g., gaze vector 238) is computed based on the network output data. In some embodiments, only certain components of the network output data (e.g., 2D pupil center data 274, eye segmentation data 276, and cornea center data 280) are used to compute the gaze vector, while other components of the network output data (e.g., glint detection data 278) are not used in the computation. In some embodiments, computing the gaze vector can include one or more post-processing steps. For example, a 3D pupil center (e.g., 3D pupil center 292) can first be computed based on one or more components of the network output data (e.g., 2D pupil center data 274 and eye segmentation data 276). Second, an optical axis (e.g., optical axis 310) can be computed based on the 3D pupil center and an additional component of the network output data (e.g., cornea center data 280). Next, the gaze vector can be computed based on the optical axis and a calibration angle corresponding to the user.

[0177] Figure 17 A method 1700 of training a neural network is shown. The steps of method 1700 need not be performed in the order shown, and one or more steps of method 1700 can be omitted during performance of method 1700. In some embodiments, one or more steps of method 1700 can be performed by processing module 250 or some other component of AR device 200.

[0178] At step 1702, a plurality of training input images (e.g., input images I(x,y,c)) is received. The plurality of training input images can be received from a camera (e.g., off-axis camera 260), or can be artificially generated or acquired for training purposes. Each training image of the plurality of training images can be an image of an eye. Step 1702 can be similar to step 1602.

[0179] Steps 1704-1712 can be performed for each training input image of the plurality of training input images. At step 1704, the training input image is provided to the neural network (e.g., multi-task neural network 256). Step 1704 can be similar to step 1604.

[0180] At step 1706, training network output data (e.g., network output data 264) is generated using the neural network. Step 1706 can be similar to step 1606.

[0181] At step 1708, GT data (e.g., GT data 268) is received from a user input device. The GT data can include one or more components (e.g., 2D pupil center GT data 283, eye segmentation GT data 284, glint detection GT data 286) corresponding to one or more components of the training network output data.

[0182] At step 1710, error data (e.g., error data 270) is computed based on a difference between the training network output data and the GT data. The error data can include one or more components corresponding to one or more components of the GT data and / or the training network output data (e.g., first error data 270A, second error data 270B, third error data 270C, fourth error data 270D).

[0183] At step 1712, the neural network is modified based on the error data. In some embodiments, the magnitude of the modification to the neural network is proportional to the magnitude of the error data, such that a larger difference between the training network output data and the GT data can correspond to a larger modification to the neural network. In some embodiments, the neural network can be trained using a backpropagation algorithm that computes one or more weight updates to the weights of the neural network.

[0184] Figure 18 A simplified computer system 1800 according to embodiments described herein is shown. As Figure 18 The computer system 1800 shown can be incorporated into a device as described herein, such as the AR device 200. Figure 18 A diagram of one embodiment of a computer system 1800 that can perform some or all of the steps of the methods provided by the various embodiments is provided. It should be noted Figure 18 is merely intended to provide a generalized illustration of various components of a computing device with which the various embodiments can be utilized. It should be noted Figure 18 The individual system elements can be implemented in a relatively

[0185] The computer system 1800 shown includes a hardware elements that can electrically couple via bus 1805 or can otherwise be in communication with one another. The hardware elements can include one or more processors 1810, including, without limitation, one or more general-purpose processors and / or one or more special- purpose processors (such as digital signal processing chips, graphics acceleration processing units), and / or the like; one or more input devices 1815, which can include, without limitation, a mouse, a keyboard, a camera, and / or the like; and one or more output devices 1820, which can include, without limitation, a display device, a printer, and / or the like.

[0186] The computer system 1800 can further include and / or be in communication with one or more non-transitory storage devices 1825, which can include, without limitation, local and / or network accessible storage, and / or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device such as a random access memory ("RAM") and / or a read-only memory ("ROM"), which can be programmable, flash- updateable, and / or the like. Such storage devices can be configured to implement any appropriate data stores, including without limitation, various file systems, database structures, and / or the like.

[0187] The computer system 1800 might also include a communications subsystem 1830, which can include without limitation a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset such as a Bluetooth® device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc. and / or the like. The communications subsystem 1830 can include one or more input and / or output communication interfaces that allow for exchange of data with a network (such as the networks described below, just to name one), other computer systems, television, and / or any other device(s) described herein. Depending on the desired functionality and / or other implementation issues, a portable electronic device or similar device can communicate image and / or other information via the communications subsystem 1830. In other embodiments, a portable electronic device (e.g., a first electronic device) can be incorporated into the computer system 1800, for example, as an electronic device of the input devices 1815. In some embodiments, the computer system 1800 will further include a working memory 1835, which can include a RAM or ROM device, as described above. TM

[0188] The computer system 1800 also can comprise software elements, shown as being currently located within the working memory 1835, including an operating system 1840, device drivers, executable libraries, and / or other code, such as one or more application programs 1845, which can comprise computer programs provided by various embodiments, and / or can be designed to implement methods, and / or configure systems provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the method(s) discussed above might be implemented as code and / or instructions executable by a computer and / or a processor within a computer; in an aspect, then, such code and / or instructions can be used to configure and / or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.

[0189] ​A set of these instructions and / or code might be stored on a non-transitory computer-readable storage medium, such as the storage device(s) 1825 described above. In some cases, the storage medium might be included in the computer system, such as computer system 1800. In other embodiments, the storage medium might be separate from a computer system (e.g., a removable medium, such as a compact disc), and / or provided in an installation package, such that the storage medium can be used to program, configure and / or adapt a general purpose computer with the instructions / code stored thereon. These instructions might take the form of executable code, which is executable by the computer system 1800 and / or might take the form of source and / or installed code, which, upon compilation and / or installation on the computer system 1800 (e.g., using any of a variety of generally available compilers, installation programs, compression / decompression utilities, etc.), then takes the form of executable code.

[0190] It will be apparent to those skilled in the art that substantial variations can be made in accordance with specific requirements. For example, customized hardware might also be used, and / or particular elements might be implemented in hardware, software (including portable software, such as applets, etc.), or both. Further, connection to other computing devices such as network input / output devices can be employed.

[0191] As mentioned above, in one aspect, some embodiments can employ a computer system (such as the computer system 1800) to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all of the procedures of such methods are performed by the computer system 1800 in response to processor 1810 executing one or more sequences of instructions contained in the working memory 1835. Such instructions can be read into the working memory 1835 from another computer readable medium, such as one or more of the storage device(s) 925. The processor 1810 can execute the sequences of instructions contained in the working memory 1835 to cause the functioning of the computer system according to the methods described herein. Alternatively, hardwired circuitry can be used in place of or in combination with software instructions to implement processes consistent with the methods described herein. The processes described herein are not inherently related to any particular computer or other apparatus. In particular, various systems can be used with computer system 1800, or alternatively with any other general purpose computer, such as a server, desktop, mobility device, laptop etc. having one or more processors.

[0192] The terms "machine-readable medium" and "computer-readable medium," as used as herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using the computer system 1800, various computer-readable media can be involved in providing instructions / code to the processor(s) 1810 for execution and / or might be used to store and / or carry such instructions / code. In many implementations, a computer-readable medium is a physical and / or tangible storage medium. Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, or any combination thereof. Non-volatile media includes, for example, optical or magnetic disks, such as the storage device(s) 1825. Volatile media includes, for example, dynamic memory, such as the working memory 1835.

[0193] Common forms of physical and / or tangible computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0194] Various forms of computer-readable media can be involved in carrying one or more sequences of one or more instructions to the processor(s) 1810 for execution. Merely by way of example, the instructions can initially be carried on a magnetic disk and / or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by the computer system 1800.

[0195] The communications subsystem 1830 and / or its components typically will receive signals, and the bus 1805 might then carry the signals and / or data, instructions, etc. carried by the signals to the working memory 1835, from which the processor(s) 1810 retrieves and executes the instructions. The instructions received by the working memory 1835 can optionally be stored on a non-transitory storage device 1825 either before or after execution by the processor(s) 1810.

[0196] The methods, systems, and devices discussed above are examples. Various configurations can omit, substitute, or add various procedures or components as appropriate. For instance, in alternative configurations, the methods can be performed in an order different from that described, and / or various stages can be added, omitted, and / or combined. Also, features described with respect to certain configurations can be combined in various other configurations. Different aspects and elements of the configurations can be combined in a similar manner. Also, technology evolves and, thus, many of the elements are examples and do not limit the scope of the disclosure or claims.

[0197] In the description specific details are set forth in order to provide a thorough understanding of the example configurations (including implementations). However, configurations can be practiced without these specific details. For example, well known circuits, processes, algorithms, structures, and techniques have not been described in detail so as not to obscure the configurations. This description provides example configurations only, and is not intended to limit the scope, applicability or configurations of the claims. Rather, this description will provide those skilled in the art with an enabling description for implementing described techniques. Various changes can be made to the function and arrangement of elements without departing from the spirit or scope of the disclosure.

[0198] Also, configurations can be described as a process that is depicted as a schematic flowchart or flow diagram. Although each configuration can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be rearranged. A process can have additional steps not included in the figure. Furthermore, examples of the methods can be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks can be stored in a non-transitory computer-readable medium such as a storage medium. Processors can perform the described tasks.

[0199] Having described several example configurations, modifications, alternative constructions, and equivalents can be used without departing from the spirit of the disclosure. For example, the above elements can be components of a larger system, wherein other rules can take precedence or otherwise modify the application of the techniques. Also, a number of steps can be undertaken before, during, or after the above elements are considered. The above description is not intended to limit the scope of the claims.

[0200] As used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Thus, for example, reference to "a user" includes a plurality of such users, and reference to "a processor" includes reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.

[0201] Also, the words "comprise," "comprising," "include," "including," and "includes" when used in this specification and in the following claims are intended to specify the presence of stated features, integers, components, or steps, but they do not preclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.

[0202] It should also be understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and that are to be included within the spirit and purview of this application and scope of the appended claims.

Claims

1. A method for training a neural network, the method comprising: Perform the first training step, including: A first image of a first eye is provided as input to a neural network having a set of feature encoding layers connected to a plurality of sets of task-specific layers, the plurality of sets of task-specific layers including at least three sets of task-specific layers operating on outputs generated by the set of feature encoding layers, the plurality of sets of task-specific layers comprising: The first set of task-specific layers, which output two-dimensional pupil data, a second set of task-specific layers that output eye segmentation data, the eye segmentation data comprising segmenting the eye into a plurality of regions, the plurality of regions comprising one or more of: a background region, a sclera region, a pupil region, or an iris region, and a third set of task-specific layers that output corneal center data; generating eye segmentation data for the first eye based on the first image of the first eye as input, using the set of feature encoding layers and the set of task-specific layers of the neural network, the eye segmentation data for the first eye comprising a segmentation of the first eye into the plurality of regions; and training the set of feature encoding layers using the eye segmentation data by modifying weights associated with the set of feature encoding layers; and Perform the second training step, including: providing a second image from a second eye to the neural network as input; generating, using the set of feature encoding layers, the first set of task-specific layers, and the third set of task-specific layers of the neural network and based on the second image of the second eye as input, network output data comprising two-dimensional pupil data corresponding to the second eye and corneal center data corresponding to the second eye; and training the plurality of sets of task-specific layers using the network output data by modifying weights associated with the plurality of sets of task-specific layers, wherein training the plurality of sets of task-specific layers during the second training step comprises using emitter position data corresponding to a set of emitters, wherein the neural network is trained such that the set of feature encoding layers is trained during the first training step but remains fixed during the second training step.

2. The method according to claim 1, wherein The first training step is performed during a first duration, and the second training step is performed during a second duration following the first duration.

3. The method according to claim 1, wherein Executing the first training step further includes: The second set of task-specific layers is trained using the eye segmentation data for the first eye.

4. The method according to claim 3, wherein: The second task-specific layer set is the only task-specific layer set of the plurality of task-specific layer sets that is trained during the first training step.

5. The method according to claim 1, wherein Executing the first training step further includes: Receive eye segmentation annotation GT data; and The eye segmentation data for the first eye is compared with the eye segmentation GT data.

6. The method according to claim 1, wherein The network output data includes glint detection data corresponding to the second eye.

7. The method according to claim 1, wherein The network output data includes a blink prediction corresponding to the second eye.

8. The method according to claim 1, wherein The network output data includes an eye classification corresponding to the second eye, the eye classification being one of a plurality of possible eye classifications, the plurality of possible eye classifications including at least one of neutral, happy, discriminatory, or sensitive.

9. The method according to claim 1, wherein The network output data comprises eye segmentation data for the second eye, the eye segmentation data for the second eye comprising a second segmentation of the second eye into the plurality of regions.

10. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for training a neural network, wherein: The operations include: Perform the first training step, including: A first image of a first eye is provided as input to a neural network having a set of feature encoding layers connected to a plurality of sets of task-specific layers, the plurality of sets of task-specific layers including at least three sets of task-specific layers operating on outputs generated by the set of feature encoding layers, the plurality of sets of task-specific layers comprising: The first set of task-specific layers, which output two-dimensional pupil data, a second set of task-specific layers that output eye segmentation data, the eye segmentation data comprising segmenting the eye into a plurality of regions, the plurality of regions comprising one or more of: a background region, a sclera region, a pupil region, or an iris region, and a third set of task-specific layers that output corneal center data; generating eye segmentation data for the first eye based on the first image of the first eye as input, using the set of feature encoding layers and the set of task-specific layers of the neural network, the eye segmentation data for the first eye comprising a segmentation of the first eye into the plurality of regions; and training the set of feature encoding layers using the eye segmentation data for the first eye by modifying weights associated with the set of feature encoding layers; and Perform the second training step, including: providing a second image from a second eye to the neural network as input; generating, using the set of feature encoding layers, the first set of task-specific layers, and the third set of task-specific layers of the neural network and based on the second image of the second eye as input, network output data comprising two-dimensional pupil data corresponding to the second eye and corneal center data corresponding to the second eye; and training the plurality of task-specific layer sets using the network output data by modifying weights associated with the plurality of task-specific layer sets, wherein training the plurality of task-specific layer sets during the second training step comprises using emitter position data corresponding to a set of emitters, wherein the neural network is trained such that the set of feature encoding layers is trained during the first training step but remains fixed during the second training step.

11. The non-transitory computer readable medium of claim 10, wherein: The first training step is performed during a first duration, and the second training step is performed during a second duration following the first duration.

12. A system comprising: one or more processors; as well as A non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for training a neural network, wherein the operations comprise: Perform the first training step, including: A first image of a first eye is provided as input to a neural network having a set of feature encoding layers connected to a plurality of sets of task-specific layers, the plurality of sets of task-specific layers including at least three sets of task-specific layers operating on outputs generated by the set of feature encoding layers, the plurality of sets of task-specific layers comprising: The first set of task-specific layers, which output two-dimensional pupil data, a second set of task-specific layers that output eye segmentation data, the eye segmentation data comprising segmenting the eye into a plurality of regions, the plurality of regions comprising one or more of: a background region, a sclera region, a pupil region, or an iris region, and a third set of task-specific layers that output corneal center data; generating eye segmentation data for the first eye based on the first image of the first eye as input, using the set of feature encoding layers and the set of task-specific layers of the neural network, the eye segmentation data for the first eye comprising a segmentation of the first eye into the plurality of regions; and training the set of feature encoding layers using the eye segmentation data by modifying weights associated with the set of feature encoding layers; and Perform the second training step, including: providing a second image from a second eye to the neural network as input; generating, using the set of feature encoding layers, the first set of task-specific layers, and the third set of task-specific layers of the neural network and based on the second image of the second eye as input, network output data comprising two-dimensional pupil data corresponding to the second eye and corneal center data corresponding to the second eye; and training the plurality of task-specific layer sets using the network output data by modifying weights associated with the plurality of task-specific layer sets, wherein training the plurality of task-specific layer sets during the second training step comprises using emitter position data corresponding to a set of emitters, wherein the neural network is trained such that the set of feature encoding layers is trained during the first training step but remains fixed during the second training step.

Images