Eye Tracking and Gaze Estimation Using an Off-Axis Camera

Abstract

Techniques related to calculating a gaze vector of a user of a wearable device are disclosed. A neural network may be trained through first and second training steps. The neural network may include a set of feature encoding layers and a set of multiple task-specific layers that each operate on an output of the set of feature encoding layers. During the first training step, a first image of a first eye may be provided to the neural network, eye segmentation data may be generated using the neural network, and the set of feature encoding layers may be trained. During the second training step, a second image of a second eye may be provided to the neural network, network output data may be generated using the neural network, and the set of task-specific layers may be trained.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Provisional Application No. 62 / 888,953, filed August 19, 2019 and entitled "EYENET: A MULTI-TASK DEEP NETWORK FOR OFF-AXIS EYE GAZE ESTIMATION AND SEMANTIC USER UNDERSTANDING," U.S. Provisional Application No. 62 / 926,241, filed October 25, 2019 and entitled "METHOD AND SYSTEM FOR PERFORMING EYE TRACKING USING AN OFF-AXIS CAMERA," and U.S. Provisional Application No. 62 / 935,584, filed November 14, 2019 and entitled "METHOD AND SYSTEM FOR PERFORMING EYE TRACKING USING AN OFF-AXIS CAMERA," the contents of which are incorporated herein by reference in their entireties for all purposes. [Background technology]

[0002] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or may be perceived as real. Virtual reality, or "VR", scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality, or "AR", scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user.

[0003] Despite the advances made in these display technologies, there remains a need in the art for improved methods, systems, and devices relating to augmented reality systems, and particularly display systems. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure generally relates to a system and method for eye tracking. More specifically, embodiments of the present disclosure provide a system and method for implementing eye tracking for gaze estimation in head-mounted virtual reality (VR), mixed reality (MR), and / or augmented reality (AR) devices. The embodiments of the present disclosure enable the use of energy- and bandwidth-efficient rendering of content to drive multi-focal displays in an effective and non-obtrusive manner to user needs. Although the present disclosure is described with reference to an AR device, the present disclosure is applicable to various applications in computer vision and image display systems.

[0005] A summary of the invention is provided with reference to a series of examples listed below. As used below, any reference to a series of examples should be understood as a reference to each of those examples separately (e.g., "Examples 1-4" should be understood as "Examples 1, 2, 3, or 4").

[0006] Example 1 is a method of training a neural network having a set of feature encoding layers and a set of multiple task-specific layers each operating on an output of the set of feature encoding layers, the method including performing a first training step including providing a first image of a first eye to the neural network and using the neural network to generate eye segmentation data based on the first image, the eye segmentation data including a segmentation of the first eye into a plurality of regions, and training the set of feature encoding layers using the eye segmentation data; and performing a second training step including providing a second image of a second eye to the neural network and using the set of feature encoding layers and each of the sets of multiple task-specific layers to generate network output data based on the second image, and using the network output data to train the set of multiple task-specific layers.

[0007] Example 2 is the method of Example 1, wherein the first training step is performed for a first duration and the second training step is performed for a second duration after the first duration.

[0008] Example 3 is the method of example 1, in which the plurality of regions includes one or more of a background region, a sclera region, a pupil region, or an iris region.

[0009] Example 4 is the method of example 1, wherein performing the first training step further comprises training a single task-specific layer set of the multiple task-specific layer sets using the eye segmentation data.

[0010] Example 5 is the method according to example 4, in which the single task-specific layer set is the only task-specific layer set of the multiple task-specific layer sets trained during the first training step.

[0011] Example 6 is a method described in Example 1, wherein the step of performing the first training step further includes a step of receiving eye segmentation ground truth (GT) data and a step of comparing the eye segmentation data with the eye segmentation GT data.

[0012] Example 7 is the method of example 1, in which the set of feature encoding layers is not trained during the second training step.

[0013] Example 8 is the method of example 1, in which the network output data includes two-dimensional (2D) pupil data corresponding to a second eye.

[0014] Example 9 is the method of example 1, in which the network output data includes flash detection data corresponding to a second eye.

[0015] Example 10 is the method of example 1, in which the network output data includes corneal center data corresponding to a second eye.

[0016] Example 11 is the method of example 1, wherein the network output data includes an eyeblink prediction corresponding to a second eye.

[0017] Example 12 is the method of example 1, wherein the network output data includes an eye expression classification corresponding to a second eye.

[0018] Example 13 is the method of example 1, in which the network output data includes second eye segmentation data including a second segmentation of the second eye into a second plurality of regions.

[0019] Example 14 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of Examples 1-13.

[0020] Example 15 is a system comprising one or more processors and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform a method described in any of Examples 1-13.

[0021] Example 16 is a method for training a neural network to classify a user's eye expression, the method including the steps of capturing an image of the eye, providing the eye image to a neural network, using the neural network to generate an eye expression classification corresponding to the eye based on the eye image, the eye expression classification being one of a plurality of possible eye expression classifications, determining a ground truth (GT) eye expression classification, calculating error data based on a difference between the eye expression classification and the GT eye expression classification, and modifying the neural network based on the error data.

[0022] Example 17 is the method of example 16, wherein the image of the eye is captured using a camera of the wearable display device.

[0023] Example 18 is the method of example 16, in which determining the GT eye expression classification includes receiving a user input indicating the GT eye expression classification.

[0024] Example 19 is the method of example 16, wherein determining the GT eye expression classification includes determining that the instructions communicated to the user indicate a GT eye expression classification.

[0025] Example 20 is the method of example 16, further comprising the step of communicating instructions to a user prior to the step of capturing an image of the eye, the instructions indicating the GT eye expression classification.

[0026] Example 21 is the method of example 16, in which modifying the neural network includes modifying a set of weights of the neural network.

[0027] Example 22 is the method according to example 21, in which the set of weights is modified using backpropagation.

[0028] Example 23 is the method of example 16, in which the neural network is modified based on the magnitude of the error data.

[0029] Example 24 is the method of example 16, further comprising outputting light toward the eye by a plurality of infrared (IR) light emitting diodes (LEDs) such that the image of the eye includes a plurality of flashes of light.

[0030] Example 25 is the method of example 16, wherein the image of the eye includes a plurality of flashes of light produced by light output by a plurality of infrared (IR) light emitting diodes (LEDs).

[0031] Example 26 is the method of example 16, wherein the image of the eye does not include the eyebrows of the user's eye.

[0032] Example 27 is the method of example 16, wherein the plurality of possible eye expression classifications includes at least one of neutral, happy, discriminatory, or sensitive.

[0033] Example 28 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of Examples 16-27.

[0034] Example 29 is a system comprising one or more processors and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform a method described in any of Examples 16-27.

[0035] Example 30 is a method for training a neural network to calculate a gaze vector, the method including the steps of capturing an image of an eye, processing the image of the eye to produce an optical axis corresponding to the eye, providing the optical axis to a neural network, generating a gaze vector corresponding to the eye based on the optical axis using the neural network, determining gaze vector ground truth (GT) data, calculating error data based on a difference between the gaze vector and the gaze vector GT data, and modifying the neural network based on the error data.

[0036] Example 31 is the method of example 30, wherein the image of the eye is captured using a camera of the wearable display device.

[0037] Example 32 is the method of example 30, wherein the gaze vector GT data is determined based on where the target is displayed on the screen.

[0038] Example 33 is the method of example 30, in which the step of determining the gaze vector GT data includes a step of receiving a user input indicating the gaze vector GT data.

[0039] Example 34 is the method of example 30, in which the step of determining the gaze vector GT data includes the step of determining that instructions communicated to the user indicate the gaze vector GT data.

[0040] Example 35 is the method of example 30, further comprising the step of communicating instructions to a user indicating the gaze vector GT data prior to the step of capturing an image of the eye.

[0041] Example 36 is the method of example 30, further comprising displaying the target at a location on the screen, and the gaze vector GT data is determined based on the location.

[0042] Example 37 is the method of example 30, in which modifying the neural network includes modifying a set of weights of the neural network.

[0043] Example 38 is the method of example 37, in which the set of weights is modified using backpropagation.

[0044] Example 39 is the method of example 30, in which the neural network is modified based on the magnitude of the error data.

[0045] Example 40 is the method of example 30, further comprising outputting light toward the eye by a plurality of infrared (IR) light emitting diodes (LEDs) such that the image of the eye includes a plurality of flashes of light.

[0046] Example 41 is the method of example 30, wherein the image of the eye includes a plurality of flashes of light produced by light output by a plurality of infrared (IR) light emitting diodes (LEDs).

[0047] Example 42 is the method of example 30, in which the line of sight vector includes at least one angle.

[0048] Example 43 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of Examples 30-42.

[0049] Example 44 is a system comprising one or more processors and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform a method described in any of Examples 30-42.

[0050] A number of advantages over conventional techniques are achieved by the method of the present disclosure. For example, estimation and simultaneous understanding of the user's eye gaze through eye images enables energy and bandwidth efficient rendering of content (foveated rendering), drives multifocal displays for more realistic rendering of content (minimizing accommodative vergence-divergence conflicts), and provides an effective and unobtrusive way to understand user expressions. An additional advantage is that estimation using trained networks works well with classical eye tracking pipelines. It has been demonstrated that estimation using trained networks can be utilized within geometric eye tracking systems to improve their overall robustness and accuracy.

[0051] In addition, results from the multi-stage eye tracking model described herein can drive other essential applications in AR / VR / MR. For example, corneal predictions can be used for foveated rendering, and eye segmentation is useful for rendering eyes in avatar-based social suite apps. Collecting gaze-targeted GT data for a large number of subjects can be both imprecise and difficult, but data collection herein is significantly simplified by decoupling the training of intermediate predictions (pupil and corneal estimation) from the final 3D gaze vector estimation pipeline. Errors in end-to-end deep networks can be difficult to interpret, so intermediate estimations made at each stage using trained networks improve interpretability. Other advantages of the present disclosure will be readily apparent to those skilled in the art. [Brief description of the drawings]

[0052] A further understanding of the nature and advantages of the various embodiments may be realized by reference to the following figures. In the accompanying figures, similar components or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. If only a first reference label is used herein, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label.

[0053] [Figure 1] FIG. 1 illustrates an augmented reality (AR) scene as viewed through a wearable AR device.

[0054] [Diagram 2] FIG. 2 illustrates various features of the AR device.

[0055] [Diagram 3] FIG. 3 illustrates the standard double spherical model of the human eye.

[0056] [Figure 4] FIG. 4 illustrates a schematic diagram of an AR device.

[0057] [Diagram 5] FIG. 5 illustrates a schematic diagram of a system for calculating gaze vectors that incorporates a multi-tasking neural network.

[0058] [Figure 6] FIG. 6 illustrates a schematic diagram of a multitask neural network.

[0059] [Figure 7] FIG. 7 illustrates systems and techniques for generating eyeblink predictions using features from separate time steps.

[0060] [Figure 8]FIG. 8 illustrates a schematic diagram of an AR device operating in training mode.

[0061] [Figure 9A] 9A and 9B illustrate a schematic diagram of sequential training steps for training a multi-task neural network. [Figure 9B] 9A and 9B illustrate a schematic diagram of sequential training steps for training a multi-task neural network.

[0062] [Figure 10] FIG. 10 illustrates a schematic diagram of an AR device operating in runtime mode.

[0063] [Figure 11] FIG. 11 illustrates a schematic diagram of an eye gaze vector neural network.

[0064] [Figure 12] Figure 12 illustrates the training pipeline.

[0065] [Figure 13] FIG. 13 illustrates a method for training a neural network having a set of feature encoding layers and a set of multiple task-specific layers.

[0066] [Figure 14] FIG. 14 illustrates a method for training a neural network to classify a user's eye expressions.

[0067] [Figure 15] FIG. 15 illustrates a method for training a neural network to calculate gaze vectors.

[0068] [Figure 16] FIG. 16 illustrates a method for calculating gaze vectors using a neural network.

[0069] [Figure 17] FIG. 17 illustrates a method for training a neural network.

[0070] [Figure 18] FIG. 18 illustrates a simplified computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0071] Detailed Description of the Invention User eye gaze estimation and simultaneous semantic understanding through eye images are key components in virtual reality (VR) and mixed reality (MR), enabling energy-efficient rendering, multifocal displays, and effective interaction with 3D content. In head-mounted VR / MR devices, the eyes may be imaged off-axis to avoid blocking the user's line of sight, which may make it very difficult to derive eye-related inferences. In various embodiments described herein, a single deep neural network is provided that solves multiple heterogeneous tasks related to eye gaze estimation and semantic user understanding for off-axis camera settings. The tasks may include eye segmentation, eye blink detection, emotion expression classification, infrared radiation (IR) light-emitting diode (LED) flash detection, and pupil and corneal center estimation. Both manual labeling and model-based teaching may be employed to train the neural network end-to-end.

[0072] The process of estimating accurate gaze involves appearance-based calculations (segmentation, feature point detection, e.g., pupil center, phosphene), followed by geometry-based calculations (e.g., estimating the cornea, pupil center, and gaze vector in three dimensions). Current eye trackers use classical computer vision techniques (without learning) to estimate the pupil border / center, and then calculate gaze based on those estimates. Estimation using the trained networks described herein is significantly more accurate than classical techniques. According to some embodiments described herein, a single deep network is trained to estimate multiple quantities related to the eye and gaze estimation together for off-axis eye images.

[0073] FIG. 1 illustrates an augmented reality (AR) scene as viewed through a wearable AR device according to an embodiment described herein. An AR scene 100 is depicted in which a user of the AR technology sees a real-world park-like setting 106 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the AR technology also perceives that they "see" a robotic figure 110 standing on the real-world platform 120 and a flying cartoon-like avatar character 102 that appears to be an anthropomorphic bumblebee, although these elements (the character 102 and the figure 110) do not exist in the real world. Due to the significant complexity of the human visual perception and nervous system, it is difficult to produce VR or AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0074] FIG. 2 illustrates various features of an AR device 200 according to some embodiments of the present disclosure. In some embodiments, the AR device 200 may include a projector 214 configured to project a virtual image light 222 (light associated with virtual content) onto the eyepiece 202 so that the user perceives one or more virtual objects (e.g., the character 102 and the figure 110) as being located at a location (e.g., one or more depth planes) in the user's environment. The user may perceive these virtual objects along with world objects 230. The AR device 200 may also include an off-axis camera 240 and one or more emitters 262 mounted on the AR device 200 and directed toward the user's eye. The emitter 262 may include an IR LED that transmits light that is invisible to the user's eye but detectable by the off-axis camera 240. In some embodiments, emitter 262 may comprise an LED that transmits light that is visible to the user's eye, such that off-axis camera 240 need not have the ability to detect light in the IR spectrum. Thus, off-axis camera 240 may be a camera with or without IR detection capability.

[0075] During operation of the AR device 200, the off-axis camera 240 may detect information (e.g., capture images) that leads to an estimation of a gaze vector 238 corresponding to the user's eye. The gaze vector 238 may be calculated for each image frame and, in various embodiments, may be represented as a two-dimensional (2D) or three-dimensional (3D) value. For example, as illustrated in FIG. 2, the gaze vector 238 may be represented using a spherical coordinate system with a polar angle θ and an azimuth angle φ. Alternatively, or in addition, the gaze vector 238 may be represented using a 3D Cartesian coordinate system with X, Y, and Z values. The gaze vector 238 may intersect with the eyepiece 202 at an intersection 239, which may be calculated based on the location of the user's eye, the location of the eyepiece 202, and the gaze vector 238. In some instances, the projector 214 may adjust the virtual image light 222 to improve image brightness and / or clarity around the intersection 239 relative to other areas of the field of view.

[0076] As shown, a set of four emitters 262 are placed in and around the display, and their reflections (flashes) are detected using the off-axis camera 240. This setup is also replicated for the user's left eye. The detected flashes are used to estimate important geometric quantities within the eye that are not directly observable from the eye camera images. As shown in FIG. 2, there may be a large angle between the user's line of sight and the camera axis. This makes eye gaze estimation difficult due to increased decentration of the pupil, partial occlusion caused by eyelids and eyelashes, and flashes artifacts caused by environmental lighting.

[0077] FIG. 3 illustrates a standard double spherical model 300 of the human eye. According to the model 300, the eye globe 302 may completely or partially encompass the inner corneal sphere 304. The corneal center 306 may be the geometric center of the corneal sphere 304. The pupil center 308 may correspond to the pupil opening or pupil center of the eye and may be encompassed by the corneal sphere 304. The optical axis 310 of the eye may be a vector formed by connecting the corneal center 306 and the pupil center 308. The gaze vector 238 (alternatively referred to as the visual axis) may be formed by connecting the corneal center 306 and the fovea 312 at the back of the eye. Because the fovea 312 is generally unknown and difficult to estimate, the gaze vector 238 may be calculated using the optical axis 310 and a user-specific calibration angle κ. The calibration angle κ may be a one-dimensional (1D), 2D, or 3D value and may be calibrated for a particular user during a calibration phase when the AR device 200 is first operated by the user. Once the calibration angle κ is calculated for a particular user, it is assumed to be fixed. Thus, estimating the optical axis 310 using the corneal center 306 and the pupil center 308 may be a key problem underlying eye tracking.

[0078] 4 illustrates a schematic diagram of an AR device 200 according to some embodiments of the disclosure. The AR device 200 may include a left eyepiece 202A, a right eyepiece 202B, a left front facing world camera 206A mounted directly on or near the left eyepiece 202A, a right front facing world camera 206B mounted directly on or near the right eyepiece 202B, a left facing world camera 206C, a right facing world camera 206D, and a processing module 250. The emitters 262 may be mounted on one or both of the eyepieces 202, and in some embodiments may be separated into a left emitter 262A mounted directly on or near the left eyepiece 202A and a right emitter 262B mounted directly on or near the right eyepiece 202B (e.g., mounted to a frame of the AR device 200). In some instances, the AR device 200 may include a single or multiple off-axis cameras 260, such as a centrally positioned off-axis camera 260, or as illustrated in FIG. 4, a left off-axis camera 260A mounted directly on or near the left eyepiece 202A, and a right off-axis camera 260A mounted directly on or near the right eyepiece 202B.

[0079] Some or all of the components of the AR device 200 may be mounted on the head so that the projected image can be viewed by the user. In one particular implementation, all of the components of the AR device 200 shown in FIG. 4 are mounted on a single device (e.g., a single headset) that can be worn by the user. In another implementation, the processing module 250 is physically separate from the other components of the AR device 200 and is communicatively coupled by wired or wireless connectivity. For example, the processing module 250 may be mounted in various configurations, such as fixedly attached to a frame, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user (e.g., in a backpack configuration, in a belt-connected configuration, etc.).

[0080] The processing module 250 may include at least one processor 252 and associated digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include data captured from sensors (e.g., operatively coupled to the AR device 200), such as image capture devices (e.g., camera 206 and off-axis camera 260), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. For example, the processing module 250 may receive images 220 from the camera 206, more specifically, a left front image 220A from a left front facing world camera 206A, a right front image 220B from a right front facing world camera 206B, a left side image 220C from a left side facing world camera 206C, and a right side image 220D from a right side facing world camera 206D. In some embodiments, the images 220 (or those received from the off-axis camera 260) may include a single image, a pair of images, a video including a stream of images, a video including a stream of paired images, and the like. The images 220 (or those received from the off-axis camera 260) may be generated and sent to the processing module 250 periodically while the AR device 200 is powered on, or may be generated in response to instructions sent by the processing module 250 to one or more of the cameras.

[0081] In some embodiments, the functionality of processing module 250 may be implemented by two or more sets of separately stored but communicatively coupled electronic hardware components (e.g., one or more processors, a set of storage devices, etc.). For example, the functionality of processing module 250 may be performed by electronic hardware components stored within the headset, in conjunction with electronic hardware components stored in a computing device physically tethered to the headset, one or more electronic devices in the headset's environment (e.g., a smartphone, a computer, a peripheral device, a smart appliance, etc.), one or more remotely located computing devices (e.g., a server, a cloud computing device, etc.), or a combination thereof.

[0082] Eyepieces 202A and 202B may include transparent or semi-transparent waveguides configured to direct light from projectors 214A and 214B, respectively. Specifically, processing module 250 may cause left projector 214A to output left virtual image light 222A onto left eyepiece 202A and right projector 214B to output right virtual image light 222B into right eyepiece 202B. In some embodiments, eyepieces 202 may include multiple waveguides, each corresponding to a different color and / or different depth planes.

[0083] Cameras 206A and 206B may be positioned to capture images that substantially overlap the field of view of the user's left and right eyes, respectively. Thus, the placement of cameras 206A and 206B may be near the user's eyes, but not so close as to obscure the user's field of view. Alternatively or in addition, cameras 206A and 206B may be positioned to align with the internal coupling locations of virtual image lights 222A and 222B, respectively. Cameras 206C and 206D may be positioned to capture images to the side of the user, e.g., within the user's peripheral vision or outside the user's peripheral vision. Images 220C and 220D captured using cameras 206C and 206D do not necessarily overlap with images 220A and 220B captured using cameras 206A and 206B. Cameras 260A and 260B may be positioned to capture images to the user's left and right eyes, respectively. The image captured by camera 260 may show the user's eye in its entirety or a portion of the user's eye.

[0084] During operation of the AR device 200, the processing module 250 may use a multitasking neural network 256 to calculate the gaze vector 238. In some embodiments, the multitasking neural network 256 may be stored in a non-transient memory associated with or otherwise accessible to at least one processor 252 of the processing module 250. The multitasking neural network 256 may be an artificial neural network, a convolutional neural network, or any type of computing system that may "learn" over time by processing examples. For example, the multitasking neural network 256 may be trained by processing manually prepared training data that represents ground truth (GT) data. After processing each of the training data, the multitasking neural network 256 may generate an output that more closely approximates the GT data.

[0085] In some embodiments, the multitasking neural network 256 comprises a collection of connected nodes capable of transmitting signals from one node to another. For example, the multitasking neural network 256 may include several different layers of such nodes. As described in further detail below, in some embodiments, the multitasking neural network 256 may include an encoder layer and a decoder layer. In some embodiments, the one or more encoder layers of the multitasking neural network 256 may be stored in a non-transient memory associated with one or more processors of a first set, while the one or more decoder layers of the multitasking neural network 256 may be stored separately but in a non-transient memory associated with one or more processors of a second set that are communicatively coupled to the one or more processors of the first set. For example, the first set of one or more processors may include one or more processors stored within the headset, while the second set of one or more processors may include one or more processors stored within a computing device physically tethered to the headset, one or more electronic devices (e.g., a smartphone, computer, peripheral device, server, cloud computing device, etc.) that are physically separate from the headset, or a combination thereof. The training and use of multitasking neural network 256 is further described below.

[0086] 5 illustrates a schematic diagram of a system for calculating gaze vectors incorporating a multi-task neural network 256. In some embodiments, an input image I(x,y,c) is captured by an off-axis camera 260 and provided as an input to the multi-task neural network 256. The input image I(x,y,c) may have dimensions H×W×C, where H is the number of pixels in the vertical direction, W is the number of pixels in the horizontal direction, and C is the number of channels of the image (e.g., for an RGB image, equal to 3, and for a grayscale image, equal to 1). The multi-task neural network 256 may process the input image I(x,y,c) and generate network output data 264 based on the input image I(x,y,c).

[0087] When the AR device 200 is operating in runtime mode, the network output data 264 may be used in conjunction with the calibration angle κ to calculate the gaze vector 238. In some embodiments, a post-processing block 266 may perform one or more operations to calculate the gaze vector 238. In other embodiments, or in the same embodiment, the calibration angle κ may be provided as an input to the multi-tasking neural network 256 along with the input image I(x,y,c), and the gaze vector 238 may be included directly in the network output data 264 or may be calculated based on the network output data 264.

[0088] When the AR device 200 is operating in training mode, the network output data 264 may be compared to the GT data 268. Error data 270 may be calculated based on the comparison and, in some embodiments, may represent a difference between the network output data 264 and the GT data 268, such that a magnitude of the error data 270 may be proportional to the difference between the network output data 264 and the GT data 268. The multi-tasking neural network 256 may be modified (e.g., using a modifier 272) based on the error data 270. In some embodiments, the magnitude of the modification to the multi-tasking neural network 256 may be proportional to the magnitude of the error data 270, such that a larger difference between the network output data 264 and the GT data 268 may correspond to a larger modification to the multi-tasking neural network 256.

[0089] In some embodiments, some or all of the operations described herein as associated with the training mode may be performed independently of the AR device 200. For example, in such embodiments, the multitasking neural network 256 may be at least partially trained prior to the manufacture and / or distribution of the AR device 200 and subsequently loaded onto the AR device 200 upon the manufacture and / or distribution of the AR device 200. In at least some of these embodiments, the multitasking neural network 256 may be trained, at least in part, with data from a relatively large population of subjects and with one or more computing devices different from the AR device 200. In some such embodiments, the AR device 200 may perform one or more of the operations described herein as associated with the training mode to further train the preloaded multitasking neural network 256 with data from a specific user of the AR device 200. This may allow one or more portions of the multitasking neural network 256 to be personalized for each user of the AR device 200. In some embodiments, the AR device 200 may store a personalized version of the multi-tasking neural network 256 for each user of the AR device 200. Thus, in these embodiments, the AR device 200 may store multiple different versions of the multi-tasking neural network 256 for multiple different users and may use the version of the multi-tasking neural network 256 that is associated with the current user of the AR device 200 at runtime.

[0090] 6 illustrates a schematic diagram of a multi-task neural network 256, which is comprised of various layers 257. In some embodiments, the multi-task neural network 256 comprises a feature encoding based network, which is comprised of a feature encoding layer 258 (alternatively referred to as an encoder layer), and six task branches, which are comprised of a task specific layer 294 (alternatively referred to as a decoder layer). The six task branches correspond to (1) pupil center estimation and glint localization, which produces 2D pupil center data 274, (2) eye part semantic segmentation, which produces eye segmentation data 276, (3) pupil and glint presence classification, which produces glint detection data 278, (4) 2D cornea estimation, which produces cornea center data 280, (5) eye blink detection, which produces eye blink prediction 296, and (6) emotion expression classification, which produces eye expression classification 298.

[0091] The network output data 264 may include one or more of the types of data shown in FIG. 6. Based on whether the AR device 200 is operating in a training mode or a runtime mode, one or more of the types of data may not be made available for subsequent processing. Alternatively, or in addition, one or more of the types of data may not be generated by the multitasking neural network 256 to conserve processor usage, power, and / or memory. Alternatively, or in addition, one or more of the types of data may not be generated based on user input. For example, an application running on the AR device 200 may require that only certain types of data, such as eye expression classification 298, be generated.

[0092] In some embodiments, the feature encoding layer 258 can produce encoder features 282 that are shared across each of the task branches. In some implementations, image feature extraction networks and feature pyramids (FPNs) are used to capture information from different scales. In some implementations, features from the top layer of the encoder (e.g., having size 20x15x256) may be used as input to the task branches.

[0093] In some embodiments, the multi-task neural network 256 includes three main appearance-based tasks in the multi-task learning model, including (1) eye part segmentation, (2) pupil and glint localization, and (3) pupil and glint presence classification. In some embodiments, eye part segmentation is defined as the task of assigning a class label from the following, namely background, sclera, iris, and pupil, to every pixel in the input image I(x,y,c). For this task, the encoder features 282, corresponding to the final layer feature map from the encoder network (e.g., feature encoding layer 258), may be obtained and upsampled to the same resolution as the input image I(x,y,c) using a deconvolution layer. The resulting four channel outputs may be converted to class probabilities, independently, pixel by pixel, using a softmax layer. The loss can be the cross-entropy loss between the predicted probability distribution and the one-hot label obtained from manually annotated ground truth (a one-hot label is a vector with all zeros except for one value, e.g., [0,0,1,0,0]).

[0094] In some embodiments, the following loss is calculated for a GT class c and a predicted probability p k With (x,y), the vector is minimized with respect to pixel x, y. [ka] In the formula, I x,y[.] is an indicator function. The overall loss can be the sum of the losses over all pixels in the image. The segmentation task serves as a bootstrap phase to train the feature encoder layer 258 as it captures the rich semantic information of the eye image. As such, eye part segmentation can be useful for the initial phase of any classical pipeline in terms of localizing the search for phosphenes (using the iris boundary) and estimating the pupil center (using the pupil boundary). In some implementations, eye part segmentation can be useful for rendering the eyes of a digital avatar.

[0095] The pupil and glen localization branches provide four glen and pupil center pixel locations for a total of five feature points. The network decoder layer for these two tasks, which may be similar to the eye part segmentation branch, may predict a set of five dense maps at the output corresponding to the five feature points. Each dense map may be normalized to sum to one across all pixels. Cross-entropy loss may then be calculated across all pixels of each map during training. Once trained, the pupil center or location of a particular glen will be the pixel that corresponds to the maximum probability in the output. In some embodiments, the following losses are minimized for each feature point (four glens and one pupil center): [ka] where I[.] is an exponential function that is zero everywhere except at the GT feature points, and p x,y is the predicted probability of the interest point location, and the summation is performed over all pixels in the image.

[0096] In realistic settings, phosphenes and / or pupil center may be occluded by eyelid closure, turbulent reflections may appear as phosphenes, and / or for some gaze angles, phosphenes may not appear on the reflective corneal surface. Thus, it may be important to learn to robustly classify the presence or absence of phosphenes and pupil center. These predictions can effectively gate whether phosphenes should be used for corneal center estimation as well as for 3D pupil center estimation.

[0097] For this task, the encoder features 282 corresponding to the top layer feature map from the encoder network (e.g., feature encoding layer 258) may be obtained, one convolutional layer may be used to reduce the number of feature channels, the reduced number of feature channels may be reconverted to a one-dimensional array, and one trainable fully connected layer (e.g., of size 1500×10) may be added to produce an output (e.g., of size 5×2). Each pair may represent the presence or absence probability for one of the four phosphenes and / or pupil centers. Binary cross-entropy loss may be used to learn from the human labeling ground truth.

[0098] Regarding corneal center estimation, the center of the cornea is a geometric quantity in 3D that cannot be observed in the 2D image of the eye. Therefore, unlike pupil (center of pupil oval) or glint labeling, it may not be possible to manually label the location projected on the image of the 3D corneal center directly. Therefore, a two-step method may be adopted to train the corneal 2D center prediction branch for the multi-task neural network 256. First, known geometric constraints and related known / estimated quantities (LED, glint) may be used to generate the corneal 2D teacher. Then, the 2D corneal branch may be trained using this model-based teacher, which is obtained frame by frame.

[0099] Using a multi-task neural network 256 to predict the cornea has two main advantages over the use of geometric constraints during evaluation. First, such predictions are more robust because deep networks tend to average out noise during training, and standard out-of-network optimization may not lead to convergence in some cases. Second, such predictions can incur only a few and constant time feedforward calculations because the cornea task branch consists of only a few fully connected layers.

[0100] The facial expression classification task involves classifying a user's emotional expressions from an input eye image. The task is particularly challenging because only the user's eye region is available as input, rather than the eyebrows and / or the entire face, as used in most emotional facial expression classification benchmarks. In some embodiments, the following individual emotional facial expressions are considered: happiness, anger, disgust, fear, and surprise. These expressions can be grouped into four discrete states: positive aspects (happiness), discriminative aspects (anger and disgust), sensitive aspects (fear and surprise), and neutral aspects. As with the other task branches, the feature encoding layer 258 is fixed and only the facial expression task branch (consisting of several FC layers) is trained for expression classification. In some embodiments, this task branch is trained per subject to produce a personalized model, which produces better accuracy than a general model for a large population of subjects.

[0101] In some embodiments, the network output data 264 may include 2D pupil center data 274. In some embodiments, the 2D pupil center data 274 may include 2D pupil centers represented as 2D values. For example, the 2D pupil centers may include X and Y values ​​in a frame of input images I(x,y,c) that correspond to a calculated location of the pupil center (e.g., pupil center 308). Alternatively, or in addition, the 2D pupil center data 274 may include a matrix, having dimensions H×W, comprising binary values ​​of 0 or 1 (where a value of 1 corresponds to the calculated location of the pupil center).

[0102] In some embodiments, the network output data 264 may include eye segmentation data 276. The eye segmentation data 276 may include a segmentation of the eye into a number of regions. In one particular implementation, the regions may include a background region, a sclera region, a pupil region, and an iris region. In another particular implementation, the regions may include a pupil region and a non-pupillary region. In another particular implementation, the regions may include a pupil region, an eye region (including parts of the eye that are not part of the pupil region), and a background region.

[0103] In some embodiments, the eye segmentation data 276 may include a matrix having dimensions H×W with a finite set of values ​​such as 0, 1, 2, and 3 (e.g., corresponding to background, sclera, pupil, and iris regions, respectively). In some embodiments, the eye segmentation data 276 includes an assignment of all pixels of the input image I(x,y,c) to a set of classes including background, sclera, pupil, and iris, which in some embodiments may be obtained by taking the last layer of the (decoder) multi-task neural network 256 and upsampling it to the same resolution as the input image I(x,y,c) using deconvolution, which is then fed into a softmax cross-entropy loss across the feature channels, where each feature channel represents the probability of the pixel belonging to a class.

[0104] In some embodiments, the network output data 264 may include flash detection data 278. In some embodiments, the flash detection data 278 includes one or more flash locations represented as 2D or 3D values. For example, if only a single flash location is detected, the flash detection data 278 may include a single 2D value, or if four flash locations are detected, the flash detection data 278 may include four 2D values. In some embodiments, the flash detection data 278 may include X and Y values ​​in a frame of input image I(x,y,c) that correspond to the calculated location of the detected flash. Alternatively, or in addition, the flash detection data 278 may include a matrix, having dimensions H×W, with binary values ​​of 0 or 1 (a value of 1 corresponds to the location of the detected flash).

[0105] In some embodiments, the network output data 264 may include corneal center data 280. In some embodiments, the corneal center data 280 may include a 2D corneal center represented as a 2D value or a 3D corneal center represented as a 3D value. For example, the 2D corneal center may include X and Y values ​​in a frame of input images I(x,y,c) that correspond to a calculated location of the center of the cornea (e.g., corneal center 306). Alternatively or additionally, the corneal center data 280 may include a matrix, having dimensions H×W, with binary values ​​of 0 or 1 (a value of 1 corresponds to the calculated location of the center of the cornea).

[0106] In some embodiments, the network output data 264 may include an eyeblink prediction 296. In some embodiments, the eyeblink prediction 296 comprises a binary value of 0 or 1 (e.g., corresponding to predictions of eye open and blink, respectively). In some embodiments, the eyeblink prediction 296 comprises a probability associated with whether an eyeblink occurred. Detecting blinks is an appearance-based task that is useful for driving multifocal displays and / or digital avatars. Blinks can be captured across a sequence of images such that temporal information can be used to distinguish blinks from events such as saccades (rapid lateral movements of the eyes).

[0107] In general, it may be difficult to accurately locate a blink event, especially when the eye is completely closed, at a standard frame rate of 30 frames / second. In other cases, it may be important to detect the onset of a blink to reduce the latency between detection and application. In some embodiments, a simple definition of a blink may be the eye state when the upper eyelid covers over 50% of the entire pupil area. This may be a useful working definition for non-expert human labelers. Given the above definition of a blink, the encoder features 282 generated by the feature encoding layer 258 trained on a task such as eye segmentation transfer well to the blink detection task. In some embodiments, the top layer (shared representation) of a pre-trained feature encoding network (e.g., feature encoding layer 258) is used to train the blink detection branch.

[0108] 7 illustrates a system and technique for generating an eyeblink prediction 296 using features from separate time steps. In the illustrated embodiment, encoder features from three consecutive time steps at T-2, T-1, and T are fed into an input a T-2 , a T-1 , and a T Then, we classify the current frame (at time T) as an eye blink or eye open and indicate it with the output y TThe resulting trained neural network (RNN) and long short-term memory (LSTM) trains have similar training and testing performance, however network 700 offers lower computational requirements.

[0109] FIG. 8 illustrates a schematic diagram of the AR device 200 operating in training mode. When the AR device 200 is operating in training mode, the network output data 264 includes eye segmentation data 276, glimmer detection data 278, and corneal center data 280. The particular input images I(x,y,c) used to generate these network outputs may also be manually inspected by one or more individuals who may prepare the GT data 268 prior to, subsequent to, or in parallel with the generation of the network output data 264 by the multitasking neural network 256. For example, the individual may inspect a displayed version of the input image I(x,y,c) on an electronic device such as a personal computer or smartphone. A program or application on the electronic device may ask the individual a set of questions related to the input image I(x,y,c), and the individual may enter their responses using an input device such as a mouse, keyboard, touch screen, or the like.

[0110] When observing and inspecting the input image I(x,y,c), an individual may prepare 2D pupil center GT data 283 by identifying the outline of the pupil using an input device. This may include the individual placing an elliptical boundary over the pupil and having the pupil center automatically calculated based on the placed elliptical boundary. The 2D pupil center GT data 283 may be prepared to have the same format and dimensions (e.g., X and Y values) as the 2D pupil center data 274. In addition, when observing and inspecting the input image I(x,y,c), an individual may prepare eye segmentation GT data 284 by determining that a first region of the image should be assigned as a background region, a second region as a sclera region, a third region as a pupil region, and a fourth region as an iris region. The eye segmentation GT data 284 may be prepared to have the same format and dimensions as the eye segmentation data 276 (e.g., a matrix having dimensions H×W with a finite set of values ​​such as 0, 1, 2, and 3 corresponding to different regions).

[0111] Additionally, upon observing and inspecting the input image I(x,y,c), the individual may prepare flash detection GT data 286 by determining the number and respective locations of flash locations present within the input image I(x,y,c). The flash detection GT data 286 may be prepared to have the same format and dimensions (e.g., a set of 2D values) as the flash detection data 278, or if a certain number of flash locations are detected (e.g., four), the flash detection GT data 286 may include that number of 2D values. In some embodiments, the flash detection GT data 286 may include X and Y values ​​within the frame of the input image I(x,y,c) that correspond to the calculated locations of the detected flashes. Alternatively, or in addition, the flash detection GT data 286 may include a matrix, having dimensions H×W, with binary values ​​of 0 or 1 (a value of 1 corresponds to the location of the detected flash).

[0112] In one particular implementation, GT data 268 may be acquired by having an individual or group of individuals face a 3x3 grid of points at two distinct depths, a near depth at, for example, 3 meters, and a farther plane at, for example, 6 meters. In response to a given cue, the individual is asked to focus their gaze on one of these 18 3D points, which allows GT data 268 related to gaze vector 238 to be collected frame by frame (to later determine overall accuracy). Captured images of the individual's eyes (using the camera of the AR device worn by the individual) may be analyzed, allowing the GT data 268 to include eye segmentation and phosphene location information. Because there are diminishing returns in annotating segmentation, phosphenes, and pupil center for every frame in a 30 or 60 Hz recording, a number (e.g., 200) of left or right eye image frames may be uniformly sampled for each individual and manually annotated with segmentation, phosphenes present or absent, phosphenes 2D, and pupil 2D location. In one particular experimental run, 87,000 annotated images were used in the dataset to train a multitask neural network256 and validate its performance.

[0113] In some embodiments, the error data 270 may include a first error data 270A calculated based on a difference between the 2D pupil center data 274 and the 2D pupil center GT data, a second error data 270B calculated based on a difference between the eye segmentation data 276 and the eye segmentation GT data 284, a third error data 270C based on a difference between the flash detection data 278 and the flash detection GT data 286, and a fourth error data 270C generated by a geometric constraint engine 288. Inputs to the geometric constraint engine 288 include one or more of the corneal center data 280, the flash detection data 278, the emitter location data 290, and the camera specific parameters 291. The emitter location data 290 may include a fixed location of the emitter 262 and / or an emission direction of the emitter 262. The emitter location data 290 may be determined during manufacturing of the AR device 200 and / or during a calibration phase. The camera intrinsic parameters 291 may include, among other possibilities, the optical center and / or focal length of the off-axis camera 260. The camera intrinsic parameters 291 may be determined during manufacturing and / or a calibration phase of the off-axis camera 260.

[0114] The geometric constraint engine 288 may perform various operations to evaluate the consistency between the different generated data (phosphene detection data 278 and corneal center data 280) and the calibrated data (emitter location data 290), and the output of the geometric constraint engine 288, the fourth error data 270D, may be inversely proportional to the likelihood or consistency parameter. In some instances, the corneal sphere 304 is reconstructed using the phosphene detection data 278 and the emitter location data 290, and the fourth error data 270D is set to the calculated distance between the center of the reconstructed sphere and the corneal center as indicated by the corneal center data 280.

[0115] In some embodiments, the training of the multi-task neural network 256 is improved by sequentially training using only certain outputs of the multi-task neural network 256 during different training iterations. In a first training step, only the eye segmentation data 276 is used to train the multi-task neural network 256. This may be accomplished by modifying the multi-task neural network 256 using only the second error data 270B. Once the multi-task neural network 256 is sufficiently trained (i.e., sufficiently accurate) for eye segmentation, a second training step is performed by additionally training the multi-task neural network 256 using the glimmer detection data 278. This may be accomplished by modifying the multi-task neural network 256 using only the third error data 270C. Once the multi-task neural network 256 is sufficiently trained for eye segmentation and glimmer detection, a third training step is performed by additionally training the multi-task neural network 256 using the corneal center data 280. This may be accomplished by using all of the error data 270 to modify the multitasking neural network 256. In some instances, the same training images and GT data may be used during different training steps. In some embodiments, the AR device 200 remains in training mode until an accuracy threshold is met or a maximum iteration threshold is met (e.g., the number of training images used meets the iteration threshold).

[0116] 9A and 9B illustrate schematic diagrams of sequential training steps 902 for training the multi-task neural network 256. Referring to FIG. 9A, a first training step 902-1 is illustrated. During the first training step 902-1, the feature encoding layer 258 and the task-specific layer 294-2 (corresponding to the decoder layer for generating the eye segmentation data 276) are trained independently from the remaining task-specific layers 294. For example, during a training iteration, an input image I(x,y,c) may be provided to the multi-task neural network 256 and may be presented to an individual, which may prepare the eye segmentation GT data 284. Using the second error data 270B calculated based on the difference between the eye segmentation data 276 and the eye segmentation GT data 284, the corrector 272 may modify (e.g., using backpropagation) weights associated with the feature encoding layer 258 and the task-specific layer 294-2 such that the second error data 270B will be reduced during a subsequent calculation of the second error data 270B based on the difference between the eye segmentation data 276 and the eye segmentation GT data 284. During the first training step 902-1, the corrector 272 does not modify weights associated with the task-specific layers 294-1, 294-3, 294-4, 294-5, or 294-6.

[0117] 9B, a second training step 902-2 is illustrated. In some embodiments, the second training step 902-2 is performed after the first training step 901-1. During the second training step 902-2, one or more of the task-specific layers 294-1, 294-3, 294-4, 294-5, and 294-6 are trained independently of the feature encoding layer 258 and the task-specific layer 294-2. For example, during a first training iteration, an input image I(x,y,c) may be provided to the multi-task neural network 256 and presented to an individual, who may prepare associated GT data 268. Using the error data 270 calculated based on the difference between the network output data 264 and the GT data 268, the corrector 272 may modify (e.g., using backpropagation) weights associated with the task-specific layers 294-1, 294-3, 294-4, 294-5, and / or 294-6 such that the error data 270 will be reduced during subsequent calculations of the error data 270. During the second training step 902-2, the corrector 272 does not modify weights associated with the feature encoding layer 258 or the task-specific layer 294-2, although in some embodiments the task-specific layer 294-2 may be fine-tuned during the second training step 902-2, as indicated by the dashed line.

[0118] In some embodiments, the task-specific layer 294-1, the task-specific layer 294-2, and the task-specific layer 294-3 may each include one or more convolutional layers and one or more deconvolutional layers. In some embodiments, the task-specific layer 294-4 and the task-specific layer 294-6 may be architecturally similar or the same as one another, but may be trained as two separate branches. In at least some of these embodiments, the task-specific layer 294-4 and the task-specific layer 294-6 may each include one or more convolutional layers. Furthermore, in some embodiments, the task-specific layer 294-5 may be architecturally similar or the same as that of the neural network 700 as described above with reference to FIG. 7.

[0119] 6, some outputs of the multi-task neural network 256 may be obtained with fewer operations performed than other outputs of the multi-task neural network 256. For example, the corneal center data 280 may be obtained with fewer calculations than the other outputs, and the eye segmentation data 276 may be obtained with more calculations than the other outputs. Thus, one advantage of initially training the multi-task neural network 256 using the eye segmentation data 276 is that some layers that are used only for the calculation of the eye segmentation data 276 can be fine-tuned without being influenced by feedback from the other outputs.

[0120] FIG. 10 illustrates a schematic diagram of the AR device 200 operating in runtime mode. When the AR device 200 is operating in runtime mode, the network output data 264 may include eye segmentation data 276, flash detection data 278, and corneal center data 280. These outputs may be used in conjunction with the calibration angle κ to calculate the gaze vector 238 using a post-processing block 266. In some embodiments, the post-processing block 266 may be separated into a first post-processing block 266A, a second post-processing block 266B, and a third post-processing block 266C. The first post-processing block 266A receives the 2D pupil center data 274 and the eye segmentation data 276 as inputs and calculates the 3D pupil center 292. The second post-processing block 266B receives the 3D pupil center 292 and the corneal center data 280 as inputs and calculates the optical axis 310. The third post-processing block 266C receives the optical axis 310 and the calibration angle κ as input and calculates the line of sight vector 238.

[0121] The accuracy of the multi-task neural network 256 has been demonstrated, for example, as described in U.S. Provisional Application No. 62 / 935,584. An example of eye segmentation accuracy is shown in the table below, which provides eye segmentation confusion matrix percentage values, with the averaged accuracy for all four classes being greater than 97.29%. [Table 1] These results are very accurate from both a quantitative and qualitative evaluation point of view. This may be important because the segmentation boundaries can be used to generate precise pupil 2D center location training data by a carefully tuned ellipse fitting procedure, especially for the partially occluded pupil case. The segmentation predictions can also be used by a classical geometric pipeline, which can be used as a baseline for gaze estimation comparison.

[0122] As another example, the accuracy of pupil and glimmer detection is shown in the table below, which shows quantitative results for predicting pixel locations using a multi-tasking neural network 256 ("NN256") and a classical pipeline, respectively. [Table 2] When the images are from an ideal setting, the multi-tasking neural network 256 and classical predictions are all accurate, with near-zero errors. However, when the images have severe reflections or the user gaze moves away from the central target, the multi-tasking neural network 256 is initially able to detect the presence or absence of a flash very accurately and provide robust labeling of the flash, while the classical approach suffers from poor absence indications and mislabeling of the flash, resulting in much higher errors under our Euclidean error metric.

[0123] 11 illustrates a schematic diagram of a gaze vector neural network 1102 that may generate gaze vectors 238 based on the calibration angle κ and the optical axis 310. In some embodiments, the gaze vector neural network 1102 may replace or be incorporated into the post-processing block 266C. In one implementation, the gaze vector neural network 1102 includes five layers and approximately 30,000 parameters or weights. In some embodiments, the gaze vector neural network 1102 is trained only on the calibration frames.

[0124] During training, the gaze vector 238 may be compared to the gaze vector GT data 1104. Error data 1106 may be calculated based on the comparison and, in some embodiments, may represent a difference between the gaze vector 238 and the gaze vector GT data 1104, such that a magnitude of the error data 1106 may be proportional to the difference between the gaze vector 238 and the gaze vector GT data 1104. The gaze vector neural network 1102 may be modified (e.g., using a corrector 1108) based on the error data 1106. In some embodiments, a magnitude of the modification of the gaze vector neural network 1102 may be proportional to a magnitude of the error data 1106, such that a larger difference between the gaze vector 238 and the gaze vector GT data 1104 may correspond to a larger modification of the gaze vector neural network 1102.

[0125] In some embodiments, the gaze vector GT data 1104 may be acquired by a user looking at a target generated on a screen. For example, the user may wear the AR device 200, which may include the previously trained multitasking neural network 256. During the training iterations, the user may be instructed to look at a target located on the display while wearing the AR device 200. An input image I(x,y,c) of the user's eye may be captured and used to generate an optical axis 310. Based on the optical axis 310 (and optionally based on the calibration angle κ), a gaze vector 238 may be generated by the gaze vector neural network 1102. The gaze vector GT data 1104 may be determined based on a relationship between the wearable device and a target generated on the display. For example, an orientation between the AR device 200 and the display may be determined based on one or more sensors, such as a camera and / or an inertial measurement unit, and the determined orientation may be used to calculate the actual gaze vector of the user's eye.

[0126] During subsequent training iterations, the target may be moved to a new location on the display, a new input image I(x,y,c) of the user's eye may be captured, and the gaze vector neural network 1102 may be modified using the newly calculated error data 1106. During various training iterations, the target may be moved to various locations across the screen to train the gaze vector neural network 1102 to robustly estimate gaze vectors over a wide range of gaze angles. In some embodiments, various lighting conditions and / or user emotions may be employed in combination with various gaze vectors during the training process, resulting in a robustly trained network.

[0127] An example of the accuracy of gaze estimation can be demonstrated by the table below, which shows the gaze error for each of the nine targets aggregated across different target planes, where the overall gaze estimation metric is defined as the angular error (e.g., in arcmin units) between the true and estimated gaze vectors. [Table 3] It is clear that the estimations using the multi-task neural network 256 and the gaze vector neural network 1102 are significantly better and similar in all directions. This can be mainly attributed to the robust phosphene and corneal 2D estimation along with the use of the gaze vector neural network 1102.

[0128] FIG. 12 illustrates a training pipeline 1200 according to some embodiments of the present invention. In some instances, full training may take several steps because the framework receives GTs from different sources and because the model-based teacher uses estimates from the trained network itself. For example, the model first trains eye segmentation and phosphene prediction, and then uses the trained model to predict phosphenes for all unlabeled data. Next, using these predicted phosphenes and known locations of the LEDs, the corneal position is inferred based on a standard eye model and geometry. Because the teacher for training the model and predicting the corneal prediction occurs using a pre-trained model and a standard eye model and geometry, the technique may be referred to as a model-based teacher.

[0129] In step 1202, the encoder-decoder network is first trained with the eye segmentation labels (e.g., eye segmented GT data 284) because this provides the richest semantic information and is the most complex supervised task to train accurately.

[0130] In step 1204, all of the supervised tasks are trained. Furthermore, in step 1204, the human label flash data (e.g., flash detection GT data 286), the pupil 2D center data (2D pupil center GT data 283), and the eye segmentation data (e.g., eye segmentation GT data 284) may be used together to jointly train each of these three supervised tasks. In some instances, initialization with weights trained from eye segmentation may result in more stable training than random initialization.

[0131] In step 1206, flash prediction (e.g., flash detection data 278) is performed for every frame and used along with the known locations of the LEDs (e.g., emitter location data 290) to generate a corneal 2D GT (generated in the geometric constraint engine 288) in step 1208 to train the corneal bifurcation (e.g., using the fourth error data 270D) in step 1210. Note that the corneal bifurcation is trained using data from the entire training set population and is further personalized (fine-tuned) in a per-subject calibration phase.

[0132] After the 3D pupil center is predicted in step 1212, the predicted cornea (personalized) and pupil 3D centers from the calibration frame are used to infer the optical axis in step 1214. Using the gaze target GT, the gaze vector neural network 1102 is trained in step 1216 to convert the optical axis to the visual axis. During runtime, the predicted cornea and pupil 2D centers are obtained from the multi-task neural network 256. These quantities are boosted to 3D and used to obtain the optical axis, which is then fed into the gaze mapping network to infer the predicted gaze direction.

[0133] The eye blink and facial expression classification tasks are trained on the intermediate features of the principal feature encoding branch. Eye blink detection is a temporal task that involves capturing three consecutive eye images and extracting the intermediate features. Using a set of pre-computed features, the eye blink detection branch is trained separately while the principal feature encoding branch of the multi-task neural network 256 remains fixed. A similar procedure is followed at runtime. For facial expression classification, the principal feature encoding branch is fixed and only the expression classification layer is trained using expression data. Expression predictions are produced along with all other tasks during runtime.

[0134] 13 illustrates a method 1300 of training a neural network (e.g., multitask neural network 256) having a set of feature encoding layers (e.g., feature encoding layer 258) and a set of multiple task-specific layers (e.g., task-specific layer 294) that each operate on an output of the set of feature encoding layers (e.g., encoder features 282). The steps of method 1300 need not be performed in the order shown, and one or more steps of method 1300 may be omitted during performance of method 1300. In some embodiments, one or more steps of method 1300 may be performed by processing module 250 or some other component of AR device 200.

[0135] In step 1302, a first training step (e.g., first training step 902-1) is performed. In some embodiments, the first training step is performed for a first duration. In some embodiments, step 1302 includes steps 1304, 1306, and / or 1308.

[0136] In step 1304, a first image of the first eye (e.g., input image I(x,y,c)) is provided to the neural network. In some embodiments, the first image is captured by and / or received from a camera (e.g., off-axis camera 260). In some embodiments, method 1300 includes capturing the first image of the first eye using the camera. In some embodiments, method 1300 includes transmitting the first image of the first eye from the camera to a processing module (e.g., processing module 250).

[0137] In step 1306, eye segmentation data (e.g., eye segmentation data 276) is generated based on the first image using a neural network. In some embodiments, the eye segmentation data includes a segmentation of the first eye into multiple regions.

[0138] In step 1308, a set of feature encoding layers is trained using the eye segmentation data. In some embodiments, a single task-specific layer set of the multiple task-specific layer sets is also trained using the eye segmentation data during the first training step. In some embodiments, error data (e.g., error data 270B) is calculated based on the difference between the eye segmentation data and the eye segmentation GT data (e.g., eye segmentation GT data 284). In some embodiments, the error data is used to train the set of feature encoding layers.

[0139] In step 1310, a second training step (e.g., second training step 902-2) is performed. In some embodiments, the second training step is performed for a second duration. In some embodiments, the second duration is after the first duration. In some embodiments, step 1310 includes steps 1312, 1314, and / or 1316.

[0140] In step 1312, a second image of the second eye (e.g., input image I(x,y,c)) is provided to the neural network. The second eye may be the same as or different from the first eye. In some embodiments, the second image is captured by and / or received from a camera. In some embodiments, method 1300 includes capturing the second image of the second eye using the camera. In some embodiments, method 1300 includes transmitting the second image of the second eye from the camera to a processing module.

[0141] In step 1314, network output data (e.g., network output data 264) is generated based on the second image using the set of feature encoding layers and each of the sets of multiple task-specific layers.

[0142] In step 1316, a set of task-specific layers is trained using the network output data. In some embodiments, the set of feature encoding layers is not trained during the second training step. In some embodiments, error data (e.g., error data 270) is calculated based on the difference between the network output data and the GT data (e.g., GT data 268). In some embodiments, the error data is used to train the set of task-specific layers.

[0143] 14 illustrates a method 1400 of training a neural network (e.g., multitask neural network 256) to classify a user's eye expression. The steps of method 1400 do not have to be performed in the order shown, and one or more steps of method 1400 may be omitted during performance of method 1400. In some embodiments, one or more steps of method 1400 may be performed by processing module 250 or some other component of AR device 200.

[0144] In step 1402, an image of the eye (e.g., an input image I(x,y,c)) is captured. In some embodiments, the first image is captured by and / or received from a camera (e.g., off-axis camera 260). In some embodiments, method 1400 includes capturing the image of the eye using the camera. In some embodiments, method 1400 includes transmitting the image of the eye from the camera to a processing module (e.g., processing module 250).

[0145] At step 1404, the eye image is provided to a neural network. In some embodiments, providing the eye image to a neural network may include providing data representing the eye image as an input to a set of operations that implement the neural network.

[0146] In step 1406, an eye representation classification (e.g., eye representation classification 298) corresponding to the eye is generated by the neural network. In some embodiments, the eye representation classification is one of multiple possible eye representation classifications.

[0147] In step 1408, a GT eye expression classification (e.g., GT data 268) is determined. In some embodiments, determining the GT eye expression classification includes receiving a user input indicative of the GT eye expression classification. For example, a user may indicate through an input device that they exhibit a "happy" expression. In some embodiments, determining the GT eye expression classification includes determining that instructions communicated to the user indicate the GT eye expression classification. For example, a user may be instructed through a display device to exhibit a "happy" facial expression.

[0148] In step 1410, error data (eg, error data 270) is calculated based on the difference between the eye representation classification and the GT eye representation classification.

[0149] In step 1412, the neural network is modified based on the error data. In some embodiments, modifying the neural network includes modifying a set of weights of the neural network. In some embodiments, the set of weights may be modified using backpropagation. In some embodiments, a set of task-specific layers of the neural network (e.g., task-specific layer 294-6) may be modified based on the error data.

[0150] 15 illustrates a method 1500 of training a neural network (e.g., gaze vector neural network 1102) to calculate gaze vectors (e.g., gaze vector 238). The steps of method 1500 need not be performed in the order shown, and one or more steps of method 1500 may be omitted during performance of method 1500. In some embodiments, one or more steps of method 1500 may be performed by processing module 250 or some other component of AR device 200.

[0151] In step 1502, an image of the eye (e.g., an input image I(x,y,c)) is captured. In some embodiments, the first image is captured by and / or received from a camera (e.g., off-axis camera 260). In some embodiments, method 1500 includes capturing the image of the eye using the camera. In some embodiments, method 1500 includes transmitting the image of the eye from the camera to a processing module (e.g., processing module 250).

[0152] In step 1504, the eye image is processed to produce an optical axis corresponding to the eye. In some embodiments, processing the eye image may include using a multi-task neural network (e.g., multi-task neural network 256) to generate 2D pupil center data (e.g., 2D pupil center data 274), eye segmentation data (e.g., eye segmentation data 276), and / or corneal center data (e.g., corneal center data 280).

[0153] In step 1506, the optical axis is provided to a neural network. In some embodiments, providing the optical axis to the neural network may include providing data representing the optical axis as an input to a set of operations that implement the neural network.

[0154] In step 1508, gaze vectors corresponding to the eyes are generated by the neural network. In some embodiments, the gaze vector includes at least one angle.

[0155] In step 1510, gaze vector GT data (e.g., gaze vector GT data 1104) is determined. In some embodiments, the gaze vector GT data is determined based on where the target is displayed on the screen. In some embodiments, determining the gaze vector GT data includes receiving user input indicating the gaze vector GT data. For example, a user may look at a particular target of multiple targets displayed on the screen and provide input as the target the user is looking at.

[0156] In step 1512, error data (eg, error data 1106) is calculated based on the difference between the gaze vector and the gaze vector GT data.

[0157] In step 1514, the neural network is modified based on the error data. In some embodiments, modifying the neural network includes modifying a set of weights of the neural network. In some embodiments, the set of weights may be modified using backpropagation.

[0158] 16 illustrates a method 1600 of calculating a gaze vector using a neural network. The steps of method 1600 need not be performed in the order shown, and one or more steps of method 1600 may be omitted during performance of method 1600. In some embodiments, one or more steps of method 1600 may be performed by processing module 250 or some other component of AR device 200.

[0159] In step 1602, an input image of a user's eye (e.g., input image I(x,y,c)) is received. In some embodiments, the input image is received from a camera (e.g., off-axis camera 260). The camera may be mounted on and / or a component of an optical device. In some embodiments, method 1600 includes capturing the input image of the user's eye using the camera. In some embodiments, method 1600 includes transmitting the input image from the camera to a processing module (e.g., processing module 250).

[0160] In step 1604, an input image of the eye is provided to a neural network (e.g., multitasking neural network 256). In some embodiments, the input image is provided to a processor that implements the neural network. The processor may be a special purpose processor (e.g., a neural network processor) having an architecture that allows certain operations commonly performed by neural networks (e.g., convolution, matrix multiplication) to be performed faster than with a general purpose processor. For example, the special purpose processor may include a systolic array having multiple processing elements for performing various arithmetic operations in parallel or simultaneously on different pixels of the input image.

[0161] In step 1606, network output data (e.g., network output data 264) is generated using the neural network. The network output data may include data corresponding to the overall output of the neural network and to outputs of intermediate layers of the neural network. For example, the network output data may include some data (e.g., eye segmentation data 276) derived from the overall output of the neural network and some data (e.g., eye blink prediction 296 and corneal center data 280) derived from outputs of intermediate layers of the neural network. Additionally or alternatively, the network output data may include some data (e.g., flash detection data 278 and 2D pupil center data 274) derived from outputs of different intermediate layers of the neural network and one or more additional layers not involved in processing the overall output of the neural network.

[0162] In step 1608, a 3D pupil center (e.g., 3D pupil center 292) is calculated based on the network output data. In some embodiments, the 3D pupil center is calculated based on the 2D pupil data and the eye segmentation data.

[0163] In step 1610, an optical axis associated with the user's eye (e.g., optical axis 310) is calculated based on the network output data. In some embodiments, the optical axis is calculated based on the 3D pupil center and some data of the network output data (e.g., corneal center data 280).

[0164] In step 1612, a gaze vector corresponding to the eye (e.g., gaze vector 238) is calculated based on the network output data. In some embodiments, the gaze vector is calculated using only certain components of the network output data (e.g., 2D pupil center data 274, eye segmentation data 276, and corneal center data 280), while other components of the network output data (e.g., flash detection data 278) are not used in the calculation. In some embodiments, calculating the gaze vector may include one or more post-processing steps. For example, a 3D pupil center (e.g., 3D pupil center 292) may be first calculated based on one or more components of the network output data (e.g., 2D pupil center data 274 and eye segmentation data 276). Second, an optical axis (e.g., optical axis 310) may be calculated based on the 3D pupil center and an additional component of the network output data (e.g., corneal center data 280). Then, the gaze vector may be calculated based on the optical axis and a calibration angle corresponding to the user.

[0165] 17 illustrates a method 1700 of training a neural network. The steps of method 1700 do not have to be performed in the order shown, and one or more steps of method 1700 may be omitted during performance of method 1700. In some embodiments, one or more steps of method 1700 may be performed by processing module 250 or some other component of AR device 200.

[0166] In step 1702, a number of training input images (e.g., input image I(x,y,c)) are received. The multiple training input images may be received from a camera (e.g., off-axis camera 260) or may be artificially generated or retrieved for training purposes. Each of the multiple training images may be an image of an eye. Step 1702 may be similar to step 1602.

[0167] Steps 1704-1712 may be performed for each training input image of the plurality of training input images. In step 1704, the training input images are provided to a neural network (e.g., multi-task neural network 256). Step 1704 may be similar to step 1604.

[0168] In step 1706, training network output data (e.g., network output data 264) is generated using the neural network. Step 1706 may be similar to step 1606.

[0169] In step 1708, GT data (e.g., GT data 268) is received from a user input device. The GT data may include one or more components (e.g., 2D pupil center GT data 283, eye segmentation GT data 284, flash detection GT data 286) that correspond to one or more components of the training network output data.

[0170] In step 1710, error data (e.g., error data 270) is calculated based on the difference between the training network output data and the GT data. The error data may include one or more components (e.g., first error data 270A, second error data 270B, third error data 270C, fourth error data 270D) corresponding to one or more components of the GT data and / or the training network output data.

[0171] In step 1712, the neural network is modified based on the error data. In some embodiments, the magnitude of the modification to the neural network is proportional to the magnitude of the error data, such that a larger difference between the training network output data and the GT data may correspond to a larger modification to the neural network. In some embodiments, the neural network may be trained using a back-propagation algorithm that calculates one or more weight updates to the weights of the neural network.

[0172] FIG. 18 illustrates a simplified computer system 1800 according to an embodiment described herein. The computer system 1800 as illustrated in FIG. 18 may be incorporated into a device such as the AR device 200 as described herein. FIG. 18 provides a schematic illustration of one embodiment of a computer system 1800 that may perform some or all of the steps of the method provided by various embodiments. Note that FIG. 18 is intended only to provide a generalized illustration of various components, any or all of which may be utilized as desired. FIG. 18 thus broadly illustrates a situation in which individual system elements may be implemented in a relatively separated manner or in a relatively more integrated manner.

[0173] Computer system 1800 is shown to comprise hardware elements that may be electrically coupled via a bus 1805 or may otherwise communicate as needed. The hardware elements may include one or more processors 1810, including one or more general purpose processors and / or one or more special purpose processors, such as, but not limited to, digital signal processing chips, graphic acceleration processors, and / or the like, one or more input devices 1815, which may include, but are not limited to, a mouse, a keyboard, a camera, and / or the like, and one or more output devices 1820, which may include, but are not limited to, a display device, a printer, and / or the like.

[0174] Computer system 1800 may further include and / or communicate with one or more non-transitory storage devices 1825, which may comprise, but are not limited to, local and / or network accessible storage devices, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices such as random access memory ("RAM"), and / or read-only memory ("ROM"), which may be programmable, flash updatable, and / or the like. Such storage devices may be configured to implement any suitable data storage, including, but not limited to, various file systems, database structures, and / or the like.

[0175] The computer system 1800 may also include a communication subsystem 1830, which may include, but is not limited to, a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a chipset, such as, but not limited to, a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication facility, and / or the like. The communication subsystem 1830 may include one or more input and / or output communication interfaces, allowing data to be exchanged with networks, such as networks described below to give as an example, other computer systems, televisions, and / or any other devices described herein. Depending on the desired functionality and / or other implementation concerns, a portable electronic device or similar device may communicate images and / or other information via the communication subsystem 1830. In other embodiments, a portable electronic device, e.g., a first electronic device, may be incorporated into the computer system 1800, e.g., an electronic device, as an input device 1815. In some embodiments, the computer system 1800 further comprises a working memory 1835, which may include a RAM or ROM device as described above.

[0176] Computer system 1800 may also include software elements, shown as currently residing in working memory 1835, including operating system 1840, device drivers, executable libraries, and / or other code, such as one or more application programs 1845, that may comprise computer programs provided by various embodiments and / or that may be designed to implement methods and / or configure systems provided by other embodiments as described herein. Simply by way of example, one or more procedures described with respect to the methods discussed above may be implemented as code and / or instructions executable by a computer or a processor within a computer, and in certain aspects such code and / or instructions may then be used to configure and / or adapt a general-purpose computer or other device to perform one or more operations in accordance with the methods described.

[0177] A set of these instructions and / or code may be stored on a non-transitory computer-readable storage medium, such as storage device 1825 described above. In some cases, the storage medium may be incorporated within a computer system, such as computer system 1800. In other embodiments, the storage medium may be separate from the computer system, e.g., a removable medium, such as a compact disc, and / or may be provided within an installation package, such that the storage medium may be used to program, configure, and / or adapt a general-purpose computer with the instructions / code stored thereon. These instructions may take the form of executable code that is executable by computer system 1800, and / or may take the form of source and / or installable code that, upon compilation and / or installation onto computer system 1800, e.g., using any of a variety of commonly available compilers, installation programs, compression / decompression utilities, and the like, then takes the form of executable code.

[0178] It will be apparent to those skilled in the art that substantial variations may be made according to specific requirements. For example, customized hardware may also be used and / or particular elements may be implemented in hardware, software, including portable software such as applets, or both. Furthermore, connection to other computing devices, such as network input / output devices, may also be employed.

[0179] As mentioned above, in one aspect, some embodiments may employ a computer system, such as computer system 1800, to perform methods according to various embodiments of the present technology. According to one set of embodiments, some or all of the procedures of such methods are performed by computer system 1800 in response to processor 1810 executing one or more sequences of one or more instructions, which may be embedded in operating system 1840, and / or other code, such as application program 1845, contained in working memory 1835. Such instructions may be read into working memory 1835 from another computer-readable medium, such as one or more of storage devices 1825. By way of example only, execution of a sequence of instructions contained in working memory 1835 may cause processor 1810 to perform one or more procedures of the methods described herein. Additionally or alternatively, some of the methods described herein may be performed through specialized hardware.

[0180] The terms "machine-readable medium" and "computer-readable medium" as used herein refer to any medium that participates in providing data that causes a machine to operate in a tangible manner. In an embodiment implemented using computer system 1800, various computer-readable media may be involved in providing instructions / code to processor 1810 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, computer-readable media are physical and / or tangible storage media. Such media may take the form of non-volatile or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device 1825. Volatile media include dynamic memory, such as, but not limited to, working memory 1835.

[0181] Common forms of physical and / or tangible computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0182] Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 1810 for execution. By way of example only, the instructions may initially be carried on a magnetic and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by the computer system 1800.

[0183] The communications subsystem 1830 and / or its components generally receive signals and the bus 1805 may then convey the signals and / or the data, instructions, etc. carried by the signals to the working memory 1835, from which the processor 1810 retrieves and executes the instructions. The instructions received by the working memory 1835 may optionally be stored on a non-transitory storage device 1825 either before or after execution by the processor 1810.

[0184] The methods, systems, and devices discussed above are examples. Various configurations may omit, substitute, or add various procedures or components, as appropriate. For example, in alternative configurations, the method may be performed in a different order than described, and / or various steps may be added, omitted, and / or combined. Also, features described with respect to one configuration may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, technology evolves, and thus many of the elements are examples and do not limit the scope of the disclosure or the claims.

[0185] Specific details are given in the description to provide a thorough understanding of the example configurations, including the implementation. However, the configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the configurations. This description provides only example configurations and does not limit the scope, applicability, or configuration of the claims. Rather, the foregoing description of the configurations will provide those skilled in the art with an effective description for implementing the described techniques. Various changes may be made in the function and arrangement of elements without departing from the spirit or scope of the present disclosure.

[0186] Also, the configurations may be described as processes, depicted as schematic flow charts or block diagrams. Although the operations may be described as sequential processes, many of the operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. The processes may have additional steps not included in the figures. Furthermore, embodiments of the method may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a non-transitory computer-readable medium, such as a storage medium. A processor may perform the tasks described.

[0187] Although several example configurations have been described, various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the disclosure. For example, the aforementioned elements may be components of a larger system, and other rules may take precedence over or otherwise modify the application of the present technology. Also, some steps may take place before, during, or after the aforementioned elements are discussed. Thus, the foregoing description is not intended to be binding on the scope of the claims.

[0188] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Thus, for example, reference to a "user" includes a plurality of such users, reference to a "processor" includes reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.

[0189] Additionally, the words "comprise," "comprising," "contains," "containing," "include," "including," and "includes," when used in this specification and the claims that follow, are intended to specify the presence of stated features, integers, components, or steps, but they do not exclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.

[0190] It should also be understood that the examples and embodiments described herein are for illustrative purposes only, and that various modifications or changes in light thereof will be suggested to those skilled in the art and are within the spirit and scope of the present application and the appended claims.

Claims

1. A method for training a neural network, the method comprising: conducting a first training step, the first training step comprising: providing a first image of a first eye as an input to the neural network, the neural network including a set of feature encoding layers connected to a set of task-specific layers, the set of task-specific layers including at least two sets of task-specific layers operating on an output generated by the set of feature encoding layers; generating eye segmentation data based on the first image of the first eye as an input using the set of feature encoding layers and the set of task specific layers of the neural network, the eye segmentation data including a segmentation of the first eye into a plurality of regions; training the set of feature encoding layers using the eye segmentation data by modifying weights associated with the set of feature encoding layers; and conducting a second training step, the second training step comprising: providing a second image of a second eye as an input to the neural network; using the set of feature encoding layers and each of the sets of task-specific layers of the neural network to generate network output data based on the second image of the second eye as an input, the network output data including outputs of the sets of task-specific layers; training the set of task-specific layers using the network output data by modifying weights associated with the set of task-specific layers; Steps A method comprising:

2. 2. The method of claim 1, wherein the first training step is performed for a first duration and the second training step is performed for a second duration following the first duration.

3. The method of claim 1 , wherein the plurality of regions includes one or more of a background region, a sclera region, a pupil region, or an iris region.

4. 2. The method of claim 1 , wherein performing the first training step further comprises using the eye segmentation data to train a single task-specific layer set of the multiple task-specific layer sets.

5. 5. The method of claim 4, wherein the single set of task-specific layers is the only set of task-specific layers of the multiple sets of task-specific layers trained during the first training step.

6. The step of performing the first training step further comprises: receiving eye segmentation ground truth (GT) data; comparing the eye segmentation data with the eye segmentation GT data; The method of claim 1 , comprising:

7. The method of claim 1 , wherein the set of feature encoding layers is not trained during the second training step.

8. The method of claim 1 , wherein the network output data includes two-dimensional (2D) pupil data corresponding to the second eye.

9. The method of claim 1 , wherein the network output data includes flash detection data corresponding to the second eye.

10. The method of claim 1 , wherein the network output data includes corneal center data corresponding to the second eye.

11. The method of claim 1 , wherein the network output data includes an eyeblink prediction corresponding to the second eye.

12. 2. The method of claim 1, wherein the network output data includes an eye expression classification corresponding to the second eye, the eye expression classification being one of a plurality of possible eye expression classifications including at least one of neutral, happy, discriminative, or sensitive.

13. The method of claim 1 , wherein the network output data includes second eye segmentation data including a second segmentation of the second eye into a second plurality of regions.

14. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for training a neural network, the operations including: conducting a first training step, the first training step comprising: providing a first image of a first eye as an input to the neural network, the neural network including a set of feature encoding layers connected to a set of task-specific layers, the set of task-specific layers including at least two sets of task-specific layers operating on an output generated by the set of feature encoding layers; generating eye segmentation data based on the first image of the first eye as an input using the set of feature encoding layers and the set of task specific layers of the neural network, the eye segmentation data including a segmentation of the first eye into a plurality of regions; training the set of feature encoding layers using the eye segmentation data by modifying weights associated with the set of feature encoding layers; and conducting a second training step, the second training step comprising: providing a second image of a second eye as an input to the neural network; using the set of feature encoding layers and each of the sets of task-specific layers of the neural network to generate network output data based on the second image of the second eye as an input, the network output data including outputs of the sets of task-specific layers; training the set of task-specific layers using the network output data by modifying weights associated with the set of task-specific layers; Steps 16. A non-transitory computer readable medium comprising:

15. 15. The non-transitory computer-readable medium of claim 14, wherein the first training step is performed for a first duration and the second training step is performed for a second duration after the first duration.

16. 1. A system comprising: one or more processors; A non-transitory computer readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for training a neural network, the operations including: conducting a first training step, the first training step comprising: providing a first image of a first eye as an input to the neural network, the neural network including a set of feature encoding layers connected to a set of task-specific layers, the set of task-specific layers including at least two sets of task-specific layers operating on an output generated by the set of feature encoding layers; generating eye segmentation data based on the first image of the first eye as an input using the set of feature encoding layers and the set of task specific layers of the neural network, the eye segmentation data including a segmentation of the first eye into a plurality of regions; training the set of feature encoding layers using the eye segmentation data by modifying weights associated with the set of feature encoding layers; and conducting a second training step, the second training step comprising: providing a second image of a second eye as an input to the neural network; using the set of feature encoding layers and each of the sets of task-specific layers of the neural network to generate network output data based on the second image of the second eye as an input, the network output data including outputs of the sets of task-specific layers; training the set of task-specific layers using the network output data by modifying weights associated with the set of task-specific layers; Steps A non-transitory computer readable medium comprising: A system comprising:

Images