Method and system for performing eye tracking using an off-axis camera

Abstract

A system and method for estimating a gaze vector of an eye using a trained neural network is provided. An input image of the eye can be received from a camera. The input image can be provided to the neural network. Network output data can be generated using the neural network. The network output data can include two-dimensional (2D) pupil data, eye segmentation data, and / or corneal center data. The gaze vector can be calculated based on the network output data. The neural network can be previously trained by providing a training input image to the neural network, generating training network output data, receiving true (GT) data, calculating error data based on the difference between the training network output data and the GT data, and modifying the neural network based on the error data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 688,983, filed on June 22, 2018, entitled “METHOD AND SYSTEM FOR PERFORMINGEYE TRACKING USING AN OFF-AXIS CAMERA,” the contents of which are hereby incorporated by reference in their entirety. Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears to be real or can be perceived as real. Virtual reality (or "VR") scenes generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality (or "AR") scenes generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user.

[0004] Despite advances in these display technologies, there is a need in the art for improved methods, systems, and apparatus relating to augmented reality systems, and particularly display systems. Summary of the invention

[0005] The present disclosure generally relates to systems and methods for eye tracking. More specifically, embodiments of the present disclosure provide systems and methods for performing eye tracking for gaze estimation in head-mounted virtual reality (VR), mixed reality (MR), and / or augmented reality (AR) devices. Embodiments of the present invention enable the use of energy and bandwidth efficient content rendering to drive multifocal displays in a manner that is efficient for the needs of the user and does not interfere with the user. Although the present invention is described with reference to an AR device, the present disclosure is applicable to various applications in computer vision and image display systems.

[0006] In one aspect of the present invention, a method for estimating a gaze vector using a neural network is provided. The method may include receiving an input image of an eye from a camera. The method may also include providing an input image of an eye to a neural network. The method may further include generating network output data using the neural network. The method may further include: based on the network output data, calculating a gaze vector corresponding to the eye. In some embodiments, the network output data includes one or more of two-dimensional (2D) pupil data, eye segmentation data, and corneal center data. In some embodiments, the 2D pupil data includes a 2D pupil center. In some embodiments, the eye segmentation data includes segmentation of the eye into multiple regions, the multiple regions including one or more of a background region, a sclera region, a pupil region, and an iris region. In some embodiments, the corneal center data includes a 2D or 3D corneal center.

[0007] In some embodiments, the method further includes calculating a three-dimensional (3D) pupil center based on the 2D pupil data and the eye segmentation data. In some embodiments, the method further includes calculating an optical axis associated with the eye based on the 3D pupil center and the cornea center data. In some embodiments, the method further includes calculating a gaze vector associated with the eye based on the optical axis and a calibration angle corresponding to the user. In some embodiments, the neural network is previously trained by providing a training input image to the neural network for each training input image in a plurality of training input images, generating training network output data using the neural network, receiving ground-truth (GT) data, calculating error data based on a difference between the training network output data and the GT data, and modifying the neural network based on the error data.

[0008] In some embodiments, the training network output data includes one or more of training eye segmentation data, training flash detection data, and training cornea center data. In some embodiments, the GT data includes one or more of 2D pupil center GT data, eye segmentation GT data, and flash detection GT data. In some embodiments, the eye segmentation GT data includes a manually created segmentation of the eye into multiple training regions, the multiple training regions including one or more of a training background region, a training sclera region, a training pupil region, and a training iris region. In some embodiments, the flash detection GT data includes manually created identifications of flash locations.

[0009] In some embodiments, the error data includes the difference between the training 2D pupil center data and the 2D pupil center GT data, the difference between the training eye segmentation data and the eye segmentation GT data, the difference between the training flash detection data and the flash detection GT data, and the error based on the training cornea center data and one or more geometric constraints. In some embodiments, the method further includes calculating one or more geometric constraints based on one or more of the emitter position data, the camera intrinsic parameters, the flash detection data, and the cornea center data. In some embodiments, the emitter position data includes multiple emitter positions. In some embodiments, multiple training input images are captured by the camera.

[0010] In another aspect of the present invention, an optical device, such as an AR device, is provided. The optical device may include a camera, a plurality of infrared (IR) light emitting diodes (LEDs) directed toward the eyes of a user (when the device is worn), and a processor, the processor being communicatively coupled to the camera and configured to perform operations including: receiving an input image of the eye from the camera, providing the input image of the eye to a neural network, generating network output data using the neural network, and calculating a gaze vector corresponding to the eye based on the network output data. The processor may be further configured to perform any of the operations described with reference to the above method.

[0011] Compared with conventional techniques, many benefits are obtained through the present disclosure. First, the estimation using the trained network works well with the classic eye tracking pipeline. It has been shown that the estimation using the trained network can be inserted into the geometric eye tracking system to improve its overall robustness and accuracy. Second, the results of the multi-stage eye tracking model described in this article can drive other important applications in AR / VR / MR. For example, corneal prediction can be used for foveated rendering, and eye segmentation is useful for rendering eyes in avatar-based social kit applications. Third, although it may be inaccurate and difficult to collect continuous gaze target GT data for a large number of object subjects, the data collection here can be significantly simplified by separating the training of intermediate predictions (pupil and corneal estimation) from the final 3D gaze vector estimation pipeline. Fourth, although it is difficult to explain the errors in the end-to-end deep network, the intermediate estimates performed at each stage using the trained network are helpful for explanation. Other benefits of the present disclosure will be obvious to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A further understanding of the nature and advantages of the various embodiments may be achieved by reference to the following drawings. In the drawings, similar parts or features may have the same reference numerals. In addition, various components of the same type may be distinguished by following the reference label with a dash and a second label to distinguish between similar components. If only the first reference label is used in the specification, the description applies to any similar components having the same first reference label regardless of the second reference label.

[0013] Figure 1 An augmented reality (AR) scene viewed through a wearable AR device according to embodiments described herein is shown.

[0014] Figure 2 Various features of an AR device according to the present invention are shown.

[0015] Figure 3 A standard two-sphere model of the human eye is shown.

[0016] Figure 4 A schematic diagram of an AR device according to the present invention is shown.

[0017] Figure 5 A schematic diagram showing the configuration of a network.

[0018] Figure 6 A schematic diagram showing the various inputs and outputs of the network.

[0019] Figure 7 A schematic diagram showing an AR device operating in a training mode.

[0020] Figure 8 A schematic diagram showing an AR device operating in runtime mode.

[0021] Fig. 9 One possible structure of the network is shown.

[0022] Fig.10 A recurrent neural network (RNN) that generates blink predictions is shown.

[0023] Fig.11 A method for estimating the gaze vector using a network is shown.

[0024] Fig.12 A method for training a network for estimating gaze vectors is shown.

[0025] Fig.13 A simplified computer system is shown according to embodiments described herein. DETAILED DESCRIPTION

[0026] Eye tracking for gaze estimation is an important component of current and future generations of head-mounted virtual and mixed reality devices. It enables the use of energy and bandwidth efficient content rendering to drive multifocal displays to obtain more realistic content rendering, and becomes an effective and non-intrusive way to understand user needs. The process of estimating accurate gaze involves appearance-based calculations (segmentation, key point detection such as pupil center, flash), followed by model / geometry-based calculations (e.g., estimating the cornea, pupil center, and gaze vector in three dimensions). Current eye trackers use classical computer vision techniques (no learning required) to estimate pupil boundaries / centers, and then calculate gaze based on these estimates. Estimates using the trained network described herein are significantly more accurate than classical techniques. According to the present invention, a single deep network is trained to jointly estimate multiple quantities related to the eye and gaze estimation for off-axis eye images.

[0027] Figure 1 An augmented reality (AR) scene viewed through a wearable AR device according to an embodiment described herein is shown. An AR scene 100 is depicted in which a user of the AR technology sees a real-world park-like setting 106 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the AR technology also perceives that he "sees" a robotic statue 110 standing on the real-world platform 120, as well as a cartoon-like avatar character 102 flying by that appears to be an avatar of a bumblebee, even though these elements (character 102 and statue 110) do not exist in the real world. Because human visual perception and nervous systems are extremely complex, it is challenging to produce virtual reality (VR) or AR technology that facilitates comfortable, natural, and rich presentation of virtual image elements along with other virtual or real-world image elements.

[0028] Figure 2 Various features of an AR device 200 according to the present invention are shown. In some embodiments, the AR device 200 may include a projector 214 configured to project light onto the eyepiece 202 so that the virtual image 222 is perceived by the user along with the world object 230. The AR device 200 may also include an off-axis camera 240 and one or more emitters 262 mounted to the AR device 200 and directed toward the user's eyes. The emitter 262 may include an infrared (IR) light emitting diode (LED) that emits light that is not visible to the user's eyes but is detectable by the off-axis camera 240. In some embodiments, the emitter 262 may include an LED that emits light that is not visible to the user's eyes so that the off-axis camera 240 does not need to have the ability to detect light in the IR spectrum. In this way, the off-axis camera 240 may be a camera with or without IR detection capabilities.

[0029] During operation of the AR device 200, the off-axis camera 240 may detect information (e.g., captured images) that results in an estimate of a gaze vector 238 corresponding to the user's eyes. The gaze vector 238 may be calculated for each image frame, and in various embodiments, the gaze vector 238 may be represented as a two-dimensional (2D) or three-dimensional (3D) value. For example, Figure 2 As shown, gaze vector 238 can be represented by a spherical coordinate system consisting of polar angle θ and azimuth angle Alternatively or additionally, gaze vector 238 may be represented by X, Y, and Z values ​​using a 3D Cartesian coordinate system. Gaze vector 238 may intersect eyepiece 202 at intersection point 239, which may be calculated based on the position of the user's eye, the position of eyepiece 202, and gaze vector 238. In some cases, projector 214 may adjust virtual image 222 to improve image brightness and / or clarity around intersection point 239 in favor of other areas of the field of view.

[0030] Figure 3 A standard two-sphere model 300 of a human eye is shown. According to the model 300, an eyeball sphere 302 may completely or partially surround an inner corneal sphere 304. A corneal center 306 may be the geometric center of the corneal sphere 304. A pupil center 308 may correspond to the pupil opening or pupil center of the eye and may be surrounded by the corneal sphere 304. An optical axis 310 of the eye may be a vector formed by connecting the corneal center 306 and the pupil center 308. A gaze vector 238 (alternatively referred to as a visual axis) may be formed by connecting the corneal center 306 and a fovea 312 located at the back of the eye. Since the fovea 312 is generally unknown and difficult to estimate, the gaze vector 238 may be calculated using the optical axis 310 and a user-specific calibration angle κ. The calibration angle κ may be a one-dimensional (1D), 2D, or 3D value and may be calibrated for a specific user during a calibration phase when the user first operates the AR device 200. Once the calibration angle κ is calculated for a specific user, it is assumed to be fixed. Therefore, estimating the optical axis 310 using the cornea center 306 and the pupil center 308 is a potential key issue for gaze tracking.

[0031] Figure 4A schematic diagram of an AR device 200 according to the present invention is shown. The AR device 200 may include a left eyepiece 202A, a right eyepiece 202B, a left front-facing world camera 206A directly attached to or near the left eyepiece 202A, a right front-facing world camera 206B directly attached to or near the right eyepiece 202B, a left side-facing world camera 206C, a right side-facing world camera 206D, and a processing module 250. The emitter 262 may be mounted on one or both of the eyepieces 202, and in some embodiments may be divided into a left emitter 262A directly mounted on or near the left eyepiece 202A and a right emitter 262B directly mounted on or near the right eyepiece 202B (e.g., mounted to a frame of the AR device 200). In some cases, the AR device 200 may include a single or multiple off-axis cameras 260, such as a centrally positioned off-axis camera 260, or as Figure 4 As shown, a left off-axis camera 260A is mounted directly on or near the left eyepiece 202A and a right off-axis camera 260A is mounted directly on or near the right eyepiece 202B.

[0032] Some or all components of the AR device 200 may be head-mounted so that the projected image can be viewed by the user. In one specific implementation, Figure 4 All components of the AR device 200 shown in FIG. 2 are mounted on a single device (e.g., a single headset) that is wearable by a user. In another implementation, the processing module 250 is physically separated from the other components of the AR device 200 and is communicatively coupled to the other components of the AR device 200 through a wired or wireless connection. For example, the processing module 250 can be mounted in various configurations, such as fixedly attached to the frame, fixedly attached to a helmet or hat worn by the user, embedded in a headset, or otherwise removably attached to the user (e.g., in a backpack configuration, in a belt-coupled configuration, etc.).

[0033] The processing module 250 may include a processor 252 and a digital memory, such as a non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. The data may include data captured from a sensor (e.g., which may be operably coupled to the AR device 200), such as an image capture device (e.g., camera 206 and off-axis camera 260), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio device, and / or a gyroscope. For example, the processing module 250 may receive an image 220 from the camera 206, or more specifically, a left front image 220A from the left front world camera 206A, a right front image 220B from the right front world camera 206B, a left side image 220C from the left side world camera 206C, and a right side image 220D from the right side world camera 206D. In some embodiments, the images 220 (or those received from the off-axis cameras 260 ) may include a single image, a pair of images, a video including an image stream, a video including a pair of image streams, etc. The images 220 (or those received from the off-axis cameras 260 ) may be periodically generated and sent to the processing module 250 when the AR device 200 is powered on, or may be generated in response to instructions sent by the processing module 250 to one or more cameras.

[0034] The eyepieces 202A and 202B may include transparent or translucent waveguides configured to guide light from the projectors 214A and 214B, respectively. Specifically, the processing module 250 may cause the left projector 214A to output a left projected image 222A to the left eyepiece 202A, and may cause the right projector 214B to output a right projected image 222B to the right eyepiece 202B. In some embodiments, each of the eyepieces 202 may include a plurality of waveguides corresponding to different colors and / or different depth planes.

[0035] Cameras 206A and 206B can be positioned to capture images that substantially overlap the fields of view of the user's left eye and right eye, respectively. Therefore, the placement of cameras 206A and 206B can be near the user's eyes, but not so close that the user's field of view is blocked. Alternatively or additionally, cameras 206A and 206B can be positioned to align with the coupling positions of projected images 222A and 222B, respectively. Cameras 206C and 206D can be positioned to capture images to the side of the user, for example, in the user's peripheral vision or outside the user's peripheral vision. Images 220C and 220D captured using cameras 206C and 206D do not necessarily need to overlap with images 220A and 220B captured using cameras 206A and 206B. Cameras 260A and 260B can be positioned to capture images of the user's left eye and right eye, respectively. The image captured by camera 260 can show all or part of the user's eyes.

[0036] During operation of the AR device 200, the processing module 250 can use a neural network 256 to estimate the gaze vector 238. The neural network 256 can be an artificial neural network, a convolutional neural network, or any type of computing system that can gradually "learn" by processing examples. For example, the neural network 256 can be trained by processing manually prepared training data representing real (GT) data. After processing each training data, the neural network 256 is able to generate an output that is closer to the GT data. In some embodiments, the neural network 256 includes a collection of connected nodes that can transmit signals from one to another. The training and use of the neural network 256 are further described below.

[0037] Figure 5 A schematic diagram of a configuration of the neural network 256 is shown. An input image I(x, y, c) captured by the off-axis camera 260 may be provided as an input to the neural network 256. The input image I(x, y, c) may have a size of H×W×C, where H is the number of pixels in the vertical direction, W is the number of pixels in the horizontal direction, and C is the number of channels of the image (e.g., equal to 3 for RGB images and 1 for grayscale images). The neural network 256 may process the input image I(x, y, c) and may generate network output data 264 based on the input image I(x, y, c). When the AR device 200 operates in runtime mode, the network output data 264 may be used in conjunction with the calibration angle κ to calculate the gaze vector 238. In some embodiments, the post-processing block 266 may perform one or more operations to calculate the gaze vector 238. In other embodiments, or in the same embodiment, the calibration angle κ can be provided as an input to the neural network 256 along with the input image I(x,y,c), and the gaze vector 238 can be directly included in the network output data 264 or can be calculated based on the network output data 264.

[0038] When the AR device 200 is operating in the training mode, the network output data 264 can be compared to the GT data 268. Error data 270 can be calculated based on the comparison, and the error data 270 can represent the difference between the network output data 264 and the GT data 268, such that in some embodiments, the size of the error data 270 can be proportional to the difference between the network output data 264 and the GT data 268. The neural network 256 can be modified (e.g., using a modifier 272) based on the error data 270. In some embodiments, the size of the modification to the neural network 256 can be proportional to the size of the error data 270, such that a larger difference between the network output data 264 and the GT data 268 can correspond to a larger modification to the neural network 256.

[0039] Figure 6 A schematic diagram of various inputs and outputs of neural network 256 is shown. Network output data 264 may include Figure 6 One or more types of data shown. Based on whether the AR device is operating in training mode or in runtime mode, one or more types of data may not be utilized in subsequent processing. Alternatively, based on whether the AR device 200 is operating in training mode or in runtime mode, one or more types of data may not be generated by the neural network 256 to save processing power.

[0040] In some embodiments, the network output data 264 may include 2D pupil center data 274. In some embodiments, the 2D pupil center data 274 may include a 2D pupil center represented as a 2D value. For example, the 2D pupil center may include X and Y values ​​within a frame of the input image I(x,y,c) corresponding to the location of the calculated pupil center (e.g., pupil center 308). Alternatively or additionally, the 2D pupil center data 274 may include a matrix having dimensions of HxW that includes binary values ​​of 0 or 1 (a value of 1 corresponds to the location of the calculated pupil center).

[0041] In some embodiments, the network output data 264 may include eye segmentation data 276. The eye segmentation data 276 may include segmentation of the eye into a plurality of regions. In one particular implementation, the region may include a background region, a sclera region, a pupil region, and an iris region. In another particular implementation, the region may include a pupil region and a non-pupil region. In another particular implementation, the region may include a pupil region, an eye region (including a portion of the eye other than a portion of the pupil region), and a background region. In some embodiments, the eye segmentation data 276 may include a matrix having a dimension of HxW, which includes a finite set of values, such as 0, 1, 2, and 3 (corresponding to, for example, a background region, a sclera region, a pupil region, and an iris region, respectively). In some embodiments, the eye segmentation data 276 includes assigning each pixel of the input image I(x,y,c) to a set of categories including background, sclera, pupil, and iris, which in some embodiments can be obtained by extracting the last layer of the (decoder) neural network 256 and upsampling it to the same resolution as the input image I(x,y,c) using deconvolution, which in turn is fed to a softmax cross entropy loss across various feature channels, where each feature channel represents the probability of the pixel belonging to a certain category.

[0042] In some embodiments, the network output data 264 may include flash detection data 278. In some embodiments, the flash detection data 278 includes one or more flash locations represented as 2D or 3D values. For example, if only a single flash location is detected, the flash detection data 278 may include a single 2D value, or if four flash locations are detected, the flash detection data 278 may include four 2D values. In some embodiments, the flash detection data 278 may include X and Y values ​​within a frame of the input image I(x,y,c) corresponding to the calculated locations of the detected flashes. Alternatively or additionally, the flash detection data 278 may include a matrix having dimensions of HxW including binary values ​​of 0 or 1 (a value of 1 corresponds to the location of the detected flash).

[0043] In some embodiments, the network output data 264 may include cornea center data 280. In some embodiments, the cornea center data 280 may include a 2D cornea center represented as a 2D value or a 3D cornea center represented as a 3D value. For example, the 2D cornea center may include X and Y values ​​within a frame of the input image I(x,y,c) corresponding to the location of the calculated cornea center (e.g., cornea center 306). Alternatively or additionally, the cornea center data 280 may include a matrix having dimensions of HxW that includes binary values ​​of 0 or 1 (a value of 1 corresponds to the location of the calculated cornea center).

[0044] In some embodiments, the network output data 264 may include a flash and pupil indicator 282. In some embodiments, the flash and pupil indicator 282 includes a binary value of 0 or 1 (a value of 1 corresponds to a successful detection of a flash or pupil center and a value of 0 corresponds to an unsuccessful detection). In an alternative embodiment, a value of 1 may correspond to an occlusion of a flash or pupil center. In one particular implementation, the flash and pupil indicator 282 may include five binary values, the first four binary values ​​corresponding to four flashes and the last binary value corresponding to a pupil center. In some embodiments, the flash and pupil indicator 282 may be generated alternately based on the 2D pupil center data 274 and the flash detection data 278. In some embodiments, the flash and pupil indicator 282 may be obtained by taking the last layer of the encoder (all features are shared in the same box), reshaping, and adding two fully connected layers to produce 5x2 probabilities as binary classifications of whether some of the four flashes and one pupil center are missing.

[0045] Figure 7 and Figure 8Schematic diagrams of an AR device 200 operating in a training mode and an operating mode are shown, respectively. When the AR device 200 operates in the training mode, the network output data 264 includes eye segmentation data 276, flash detection data 278, and corneal center data 280. The specific input image I (x, y, c) used to generate these network outputs can also be manually inspected by one or more individuals, who can prepare the GT data 268 before, after, or while the network output data 264 is generated by the neural network 256. For example, an individual can examine a displayed version of the input image I (x, y, c) on an electronic device such as a personal computer or a smart phone. A program or application on the electronic device can ask the individual a set of questions related to the input image I (x, y, c), and the individual can use an input device (e.g., a mouse, keyboard, touch screen, etc.) to enter his or her responses.

[0046] While observing and examining the input image I(x,y,c), the individual may prepare 2D pupil center GT data 283 by identifying the outline of the pupil using an input device. This may include the individual placing an elliptical boundary over the pupil and automatically calculating the pupil center based on the placed elliptical boundary. The 2D pupil center GT data 283 may be prepared so that it has the same format and size (e.g., X and Y values) as the 2D pupil center data 274. Additionally, while observing and examining the input image I(x,y,c), the individual may prepare eye segmentation GT data 284 by deciding that a first region of the image should be designated as a background region, a second region as a sclera region, a third region as a pupil region, and a fourth region as an iris region. The eye segmentation GT data 284 may be prepared so that it has the same format and size (e.g., a matrix having a dimension of HxW, which includes a finite set of values, such as 0, 1, 2, and 3 corresponding to different regions) as the eye segmentation data 276.

[0047] Additionally, while observing and examining the input image I(x,y,c), the individual may prepare the flash detection GT data 286 by determining how many flash locations are present in the input image I(x,y,c) and the location of each. The flash detection GT data 286 may be prepared to have the same format and size as the flash detection data 278 (e.g., a set of 2D values), or if a certain number of flash locations (e.g., four) are detected, the flash detection GT data 286 may include that number of 2D values. In some embodiments, the flash detection GT data 286 may include X and Y values ​​within the frame of the input image I(x,y,c) corresponding to the calculated locations of the detected flashes. Alternatively or additionally, the flash detection GT data 286 may include a matrix having dimensions of HxW that includes binary values ​​of 0 or 1 (a value of 1 corresponds to the location of the detected flash).

[0048] In one particular implementation, GT data 268 may be obtained by having an individual or group of individuals face a 3x3 grid of points at two different depths (at a near depth at, for example, 3 meters and at a more distant plane at, for example, 6 meters). Given a prompt, the individual is asked to focus their gaze on one of these 18 3D points, which allows GT data 268 of the gaze vector 238 to be collected for each frame (in order to later determine overall accuracy). Images of the individual's eyes captured (using a camera of an AR device worn by the individual) may be analyzed to allow GT data 268 to include eye segmentation and glint position information. Due to the diminishing returns of annotating segmentation, glint, and pupil center for each frame at 30 or 60 Hz recording, a number (e.g., 200) of left or right eye image frames may be uniformly sampled for each individual to manually annotate segmentation, presence or absence of glint, glint 2D, and pupil 2D location. In one particular experimental run, 87,000 annotated images were used in the dataset to train and validate the performance of the neural network 256.

[0049] In some embodiments, the error data 270 may include first error data 270A calculated based on the difference between the 2D pupil center data 274 and the 2D pupil center GT data, second error data 270B calculated based on the difference between the eye segmentation data 276 and the eye segmentation GT data 284, third error data 270C calculated based on the difference between the flash detection data 278 and the flash detection GT data 286, and fourth error data 270C generated by the geometric constraint engine 288. The inputs to the geometric constraint engine 288 include one or more of the cornea center data 280, the flash detection data 278, the emitter position data 290, and the camera intrinsic parameters 291. The emitter position data 290 may include a fixed position of the emitter 262 and / or an emission direction of the emitter 262. The emitter position data 290 may be determined when the AR device 200 is manufactured and / or during a calibration phase. The camera intrinsic parameters 291 may include the optical center and / or focal length of the off-axis camera 260, among other possibilities. The camera intrinsic parameters 291 may be determined when the off-axis camera 260 is manufactured and / or during a calibration phase.

[0050] The geometric constraint engine 288 may perform various operations to evaluate the consistency between the different generated data (flash detection data 278 and cornea center data 280) and the calibration data (emitter position data 290), and the output of the geometric constraint engine 288, the fourth error data 270D, may be inversely related to the likelihood or consistency parameter. In some cases, the corneal sphere 304 is reconstructed using the flash detection data 278 and the emitter position data 290, and the fourth error data 270D is set to the calculated distance between the center of the reconstructed sphere and the cornea center, as indicated by the cornea center data 280.

[0051] In some embodiments, the training of the neural network 256 is improved by sequentially training using only certain outputs of the neural network 256 during different training iterations. In a first training step, only the eye segmentation data 276 is used to train the neural network 256. This can be achieved by modifying the neural network 256 using only the second error data 270B. Once the neural network 256 is sufficiently trained for eye segmentation (i.e., sufficiently accurate), a second training step can be performed to train the neural network 256 by additionally using the flash detection data 278. This can be achieved by modifying the neural network 256 using only the second error data 270B and the third error data 270C. Once the neural network 256 is sufficiently trained for eye segmentation and flash detection, a third training step can be performed to train the neural network 256 by additionally using the cornea center data 280. This can be achieved by modifying the neural network 256 using all the error data 270. In some cases, the same training images and GT data can be used during different training steps. In some embodiments, the AR device 200 remains in training mode until an accuracy threshold is met or a maximum iteration threshold is met (i.e., the number of training images used meets the maximum iteration threshold).

[0052] When the AR device 200 operates in the run mode, the network output data 264 includes 2D pupil center data 274, eye segmentation data 276, and cornea center data 280. These outputs can be used in conjunction with the calibration angle κ to calculate the gaze vector 238 using the post-processing block 266. In some embodiments, the post-processing block 266 can be divided into a first post-processing block 266A, a second post-processing block 266B, and a third post-processing block 266C. The first post-processing block 266A receives the 2D pupil center data 274 and the eye segmentation data 276 as input and calculates the 3D pupil center 292. The second post-processing block 266B receives the 3D pupil center 292 and the cornea center data 280 as input and calculates the optical axis 310. The third post-processing block 266C receives the optical axis 310 and the calibration angle κ as input and calculates the gaze vector 238.

[0053] Fig. 9One possible structure of a neural network 256 including a plurality of levels 257 is shown. Each level 257 may represent one or more operations performed on an input image I(x, y, c). In some embodiments, the neural network 256 may be composed of a feature encoding base network and five task branches. As shown, some outputs of the neural network 256 may be obtained at an earlier stage and perform fewer operations than other outputs of the neural network 256. For example, the glint and pupil indicator 282 and the cornea center data 280 may be obtained by fewer calculations than other outputs, and the eye segmentation data 276 may be obtained by more calculations than other outputs. Therefore, an advantage of first training the neural network 256 using the eye segmentation data 276 is that some levels used only to calculate the eye segmentation data 276 may be finely tuned without being affected by feedback from other outputs.

[0054] Fig.10 A recurrent neural network (RNN) 1000 is shown that takes features from the neural network 256 and produces a blink prediction. More specifically, the RNN 1000 takes features (e.g., outputs) of several consecutive frames (the neural network 256 may produce features for each frame) from the neural network 256 and predicts whether a blink will occur (probability) in the next 1, 2, and / or 3 frames based on the motion and flow of the features. In some embodiments, the RNN 1000 takes features only from the consecutive frames output by the neural network 256, rather than from the input image I (x, y, c) itself. The advantage of the RNN 1000 is that it can predict time series or data sequences (motion, tracking, etc.). Unlike a convolutional neural network (CNN), the RNN 1000 may have an internal memory state with an output that is fed back to itself to form a loop in order to process the sequence input data. In this way, the RNN 1000 "remembers" the history of events that occurred in the sequence, so it can better predict the next outcome. In the specific implementation shown, X t-1 , X t and X t+1 corresponds to features of consecutive frames from neural network 256, and O t-1 , O t and O t+1 is the probability of whether there will be a blink event in the next frame. Since the RNN 1000 utilizes high-level features produced by the neural network 256, it can be considered an additional post-processing step that can be included to improve the performance of the AR device 200.

[0055] Fig.11A method 1100 for estimating a gaze vector using a neural network is shown. The steps of the method 1100 need not be performed in the order shown, and one or more steps of the method 1100 may be omitted during the execution of the method 1100. In some embodiments, one or more steps of the method 1100 may be performed by the processing module 250 or some other component of the AR device 200.

[0056] At step 1102, an input image of a user's eye (e.g., input image I(x, y, c)) is received. In some embodiments, the input image is received from a camera (e.g., off-axis camera 260). The camera can be mounted to an optical device and / or can be a component of an optical device. In some embodiments, method 1100 includes the step of capturing the input image of the user's eye using the camera. In some embodiments, method 1100 includes the step of sending the input image from the camera to a processing module (e.g., processing module 250).

[0057] At step 1104, an input image of the eye is provided to a neural network (e.g., neural network 256). In some embodiments, the input image is provided to a processor that implements the neural network. The processor may be a special-purpose processor (e.g., a neural network processor) having an architecture that allows certain operations commonly performed by the neural network (e.g., convolution, matrix multiplication) to be performed faster than using a general-purpose processor. For example, the special-purpose processor may include a systolic array with multiple processing elements for performing various arithmetic operations on different pixels of the input image simultaneously or synchronously.

[0058] At step 1106, network output data (e.g., network output data 264) is generated using the neural network. The network output data may include data corresponding to the overall output of the neural network and the outputs of intermediate layers of the neural network. For example, the network output data may include certain data derived from the overall output of the neural network (e.g., eye segmentation data 276) and certain data derived from the outputs of intermediate layers of the neural network (e.g., glint and pupil indicators 282 and cornea center data 280). Additionally or alternatively, the network output data may include certain data derived from the outputs of different intermediate layers of the neural network and one or more additional layers of processing that do not involve the overall output of the neural network (e.g., glint detection data 278 and 2D pupil center data 274).

[0059] At step 1108, a 3D pupil center (eg, 3D pupil center 292) is calculated based on the network output data. In some embodiments, the 3D pupil center is calculated based on the 2D pupil data and the eye segmentation data.

[0060] At step 1110, an optical axis associated with the user's eye (eg, optical axis 310) is calculated based on the network output data. In some embodiments, the optical axis is calculated based on the 3D pupil center and certain data of the network output data (eg, cornea center data 280).

[0061] At step 1112, a gaze vector corresponding to the eye (e.g., gaze vector 238) is calculated based on the network output data. In some embodiments, only certain components of the network output data (e.g., 2D pupil center data 274, eye segmentation data 276, and cornea center data 280) are used to calculate the gaze vector, while other components of the network output data (e.g., flash detection data 278) are not used in the calculation. In some embodiments, calculating the gaze vector may include one or more post-processing steps. For example, a 3D pupil center (e.g., 3D pupil center 292) may be first calculated based on one or more components of the network output data (e.g., 2D pupil center data 274 and eye segmentation data 276). Second, an optical axis (e.g., optical axis 310) may be calculated based on the 3D pupil center and additional components of the network output data (e.g., cornea center data 280). Next, a gaze vector may be calculated based on the optical axis and a calibration angle corresponding to the user.

[0062] Fig.12 A method 1200 for training a neural network is shown. The steps of method 1200 do not have to be performed in the order shown, and one or more steps of method 1200 may be omitted during the execution of method 1200. In some embodiments, one or more steps of method 1200 may be performed by processing module 250 or some other component of AR device 200.

[0063] At step 1202, a plurality of training input images (e.g., input images I(x, y, c)) are received. The plurality of training input images may be received from a camera (e.g., off-axis camera 260), or may be artificially generated or retrieved for training purposes. Each of the plurality of training images may be an image of an eye. Step 1202 may be similar to step 1102.

[0064] Steps 1204 to 1212 may be performed for each training input image in a plurality of training input images. At step 1204, the training input image is provided to a neural network (e.g., neural network 256). Step 1204 may be similar to step 1104.

[0065] At step 1206 , training network output data (eg, network output data 264 ) is generated using the neural network. Step 1206 may be similar to step 1106 .

[0066] At step 1208, GT data (e.g., GT data 268) is retrieved from the user input device. The GT data may include one or more components corresponding to one or more components of the training network output data (e.g., 2D pupil center GT data 283, eye segmentation GT data 284, flash detection GT data 286).

[0067] At step 1210, error data (e.g., error data 270) is calculated based on the difference between the training network output data and the GT data. The error data may include one or more components (e.g., first error data 270A, second error data 270B, third error data 270C, fourth error data 270D) corresponding to one or more components of the GT data and / or the training network output data.

[0068] At step 1212, the neural network is modified based on the error data. In some embodiments, the magnitude of the modification to the neural network is proportional to the magnitude of the error data, such that a larger difference between the training network output data and the GT data may correspond to a larger modification to the neural network. In some embodiments, the neural network may be trained using a back-propagation algorithm that calculates one or more weight updates to the weights of the neural network.

[0069] Fig.13 1 shows a simplified computer system 1300 according to the embodiments described herein. Fig.13 The computer system 1300 shown in FIG. 1 may be incorporated into a device such as the AR device 200 described herein. Fig.13 A schematic diagram of one embodiment of a computer system 1300 is provided that can perform some or all of the steps of the methods provided by various embodiments. It should be noted that Fig.13 It is intended only to provide a general illustration of the various components, any or all of which may be utilized as appropriate. Fig.13 It is broadly shown how various system elements may be implemented in a relatively discrete or relatively more integrated manner.

[0070] A computer system 1300 is shown, which includes hardware elements that can be electrically coupled via a bus 1305 or can communicate in other appropriate ways. The hardware elements may include one or more processors 1310, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors, such as digital signal processing chips, graphics acceleration processors, etc.; one or more input devices 1315, which may include but are not limited to a mouse, a keyboard, a camera, etc.; and one or more output devices 1320, which may include but are not limited to a display device, a printer, etc.

[0071] The computer system 1300 may further include and / or communicate with one or more non-volatile storage devices 1325, which may include, but are not limited to, local and / or network accessible storage devices, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices (such as random access memory ("RAM") and / or read-only memory ("ROM"), which may be programmable, updateable, etc.). Such storage devices may be configured to implement any appropriate data storage, including, but not limited to, various file systems, database structures, etc.

[0072] The computer system 1300 may also include a communication subsystem 1330, which may include but is not limited to a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a Bluetooth TM Devices, 802.11 devices, WiFi devices, WiMax devices, chipsets for cellular communication facilities, etc., etc. The communication subsystem 1330 may include one or more input and / or output communication interfaces to allow data to be exchanged with a network such as the network described below: the network is to name an example, other computer systems, televisions, and / or any other devices described herein. Depending on the desired functionality and / or other implementations, a portable electronic device or similar device can transmit images and / or other information via the communication subsystem 1330. In other embodiments, a portable electronic device (e.g., a first electronic device) can be incorporated into the computer system 1300, such as an electronic device as an input device 1315. In some embodiments, as described above, the computer system 1300 will further include a working memory 1335, which can include RAM or ROM devices.

[0073] The computer system 1300 may also include software elements, shown as currently located in the working memory 1335, including an operating system 1340, device drivers, executable libraries and / or other code, such as one or more applications 1345, which, as described herein, may include computer programs provided by various embodiments, and / or may be designed to implement methods and / or configure systems provided by other embodiments. Merely by way of example, one or more processes described with respect to the above methods may be implemented as codes and / or instructions executable by a computer and / or a processor within a computer, and thus, in one aspect, such codes and / or instructions may be used to configure and / or adapt a general purpose computer or other device to perform one or more operations according to the described methods.

[0074] The set of these instructions and / or codes may be stored on a non-transitory computer-readable storage medium, such as the storage device 1325 described above. In some cases, the storage medium may be incorporated into a computer system, such as the computer system 1300. In other embodiments, the storage medium may be separate from the computer system (e.g., a removable medium such as an optical disk), and / or provided in an installation package, such that the storage medium may be used to program, configure, and / or adapt a general-purpose computer and the instructions / code stored thereon. The instructions may take the form of executable code that can be executed by the computer system 1300, and / or may take the form of source code and / or installable code that then takes the form of executable code when compiled and / or installed on the computer system 1300, for example, using various commonly available compilers, installers, compression / decompression utilities, etc.

[0075] It will be apparent to those skilled in the art that substantial variations may be made depending on specific requirements. For example, customized hardware may also be used, and / or specific elements may be implemented in hardware, software including portable software (such as applets, etc.), or both. In addition, connections to other computing devices such as network input / output devices may be employed.

[0076] As described above, in one aspect, some embodiments may employ a computer system such as computer system 1300 to perform methods according to various embodiments of the present technology. According to one set of embodiments, some or all of the processes of such methods are performed by computer system 1300 in response to processor 1310 executing one or more sequences of one or more instructions, which may be incorporated into operating system 1340 and / or other code contained in working memory 1335, such as application 1345. Such instructions may be read into working memory 1335 from another computer-readable medium, such as one or more storage devices 1325. By way of example, execution of the sequences of instructions contained in working memory 1335 may cause processor 1310 to perform one or more processes of the methods described herein. Additionally or alternatively, portions of the methods described herein may be performed by dedicated hardware.

[0077] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in a specific manner. In embodiments implemented using computer system 1300, various computer-readable media may be involved in providing instructions / code to processor 1310 for execution and / or may be used to store and / or carry such instructions / code. In many embodiments, computer-readable media are physical and / or tangible storage media. Such media may take the form of non-volatile media or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device 1325. Volatile media include, but are not limited to, dynamic memory, such as working memory 1335.

[0078] Common forms of physical and / or tangible computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cassette or any other medium from which a computer can read instructions and / or code.

[0079] Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to processor 1310 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium for receipt and / or execution by computer system 1300.

[0080] The communication subsystem 1330 and / or its components will typically receive the signal, and the bus 1305 may then carry the signal and / or the data, instructions, etc. carried by the signal to the working memory 1335, from which the processor 1310 retrieves and executes the instructions. The instructions received by the working memory 1335 may optionally be stored on the non-transitory storage device 1325 before or after execution by the processor 1310.

[0081] The methods, systems, and devices discussed above are examples. Various configurations may omit, replace, or add various processes or components as appropriate. For example, in alternative configurations, the method may be performed in an order different from the order described, and / or various stages may be added, omitted, and / or combined. In addition, features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. In addition, technology is developing, and therefore, many elements are examples and do not limit the scope of the present disclosure or claims.

[0082] Specific details are given in the specification to provide a thorough understanding of the exemplary configurations including the embodiments. However, the configurations can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures and techniques have been shown without unnecessary details to avoid obscuring the configurations. This description only provides example configurations and does not limit the scope, applicability or configuration of the claims. On the contrary, the previous description of the configuration will provide an enabling description for implementing the described technology to those skilled in the art. Various changes may be made to the functions and arrangements of the elements without departing from the spirit or scope of the present disclosure.

[0083] In addition, the configuration can be described as a process, which is described as a schematic flow chart or block diagram. Although each operation can describe the operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. The process may have additional steps not included in the figure. In addition, the example of the method can be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segment for performing the necessary tasks can be stored in a non-transitory computer-readable medium such as a storage medium. The processor can perform the described tasks.

[0084] Several example configurations have been described, and various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the present disclosure. For example, the above elements may be components of a larger system, where other rules may take precedence over or otherwise modify the application of the technology. Similarly, many steps may be taken before, during, or after considering the above elements. Therefore, the above description does not limit the scope of the claims.

[0085] As used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Thus, for example, reference to "a user" includes a plurality of such users and reference to "a processor" includes reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.

[0086] Furthermore, when used in this specification and the appended claims, the words "comprising", "including", "containing", "included", and "including" are intended to specify the presence of stated features, integers, components or steps, but they do not exclude the presence or addition of one or more other features, integers, components, steps, actions or groups.

[0087] It should also be understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes thereto will be suggested to those skilled in the art and are to be included within the spirit and purview of the present application and the scope of the appended claims.

Claims

1. A method for estimating a gaze vector for a user of an optical device, the method comprising: receiving an input image of the user's eye from a camera mounted to the optical device; providing the input image of the eye of the user to a neural network having a plurality of layers, the neural network having been previously trained using a plurality of training input images; generating network output data using the neural network, the network output data comprising two-dimensional pupil data and cornea center data, wherein a first subset of the plurality of levels is used to generate the two-dimensional pupil data and a second subset of the plurality of levels is used to generate the cornea center data, wherein the first subset and the second subset share at least one common layer, and wherein each of the first subset and the second subset includes at least one unique layer; and Based on the network output data, the gaze vector corresponding to the eye of the user is calculated.

2. The method according to claim 1, wherein: The network output data further includes: Eye segmentation data.

3. The method according to claim 1 or 2, wherein: The two-dimensional pupil data includes a two-dimensional pupil center.

4. The method according to claim 1 or 2, wherein: The eye segmentation data includes a segmentation of the eye of the user into a plurality of regions including one or more of a background region, a sclera region, a pupil region, or an iris region.

5. The method according to claim 1 or 2, wherein: The cornea center data includes a two-dimensional or three-dimensional cornea center.

6. The method according to claim 2, further comprising: Based on the two-dimensional pupil data and the eye segmentation data, a three-dimensional pupil center is calculated.

7. The method according to claim 6, further comprising: Based on the three-dimensional pupil center and the cornea center data, an optical axis associated with the eye of the user is calculated.

8. The method according to claim 7, further comprising: A gaze vector associated with the eye of the user is calculated based on the optical axis and a calibration angle corresponding to the user.

9. The method according to claim 1, wherein: The neural network was previously trained by: For each training input image in the plurality of training input images: providing the training input image to the neural network; Using the neural network to generate training network output data; Receive real data; Calculating error data based on the difference between the training network output data and the true data; as well as The neural network is modified based on the error data.

10. The method according to claim 9, wherein: The training network output data includes one or more of the following: Training 2D pupil center data; Train the eye to segment the data; training flash detection data; or Training cornea center data.

11. The method according to claim 10, wherein: The real data includes one or more of the following: Two-dimensional pupil center real data; Segment the ground truth data by eye; or Flash detection real data.

12. The method according to claim 11, wherein: The eye segmentation real data includes a manually created segmentation of the eye of the user into a plurality of training regions including one or more of a training background region, a training sclera region, a training pupil region, and a training iris region.

13. The method according to claim 11, wherein: The flash detection truth data includes manually created identifications of flash locations.

14. The method according to claim 11, wherein: The error data includes one or more of the following: The difference between the training two-dimensional pupil center data and the two-dimensional pupil center real data; a difference between the training eye segmentation data and the eye segmentation true data; a difference between the training flash detection data and the flash detection real data; or An error is determined based on the training cornea center data and one or more geometric constraints.

15. An optical device comprising: camera; as well as A processor is communicatively coupled to the camera and configured to perform the following operations, including: receiving an input image of a user's eye from the camera; providing the input image of the eye of the user to a neural network having a plurality of layers, the neural network having been previously trained using a plurality of training input images; generating network output data using the neural network, the network output data comprising two-dimensional pupil data and cornea center data, wherein a first subset of the plurality of levels is used to generate the two-dimensional pupil data and a second subset of the plurality of levels is used to generate the cornea center data, wherein the first subset and the second subset share at least one common layer, and wherein each of the first subset and the second subset includes at least one unique layer; and Based on the network output data, a gaze vector corresponding to the eye of the user is calculated.

16. The optical device according to claim 15, further comprising: A plurality of infrared (IR) light emitting diodes (LEDs).

17. The optical device according to claim 15, wherein: The network output data further includes: Eye segmentation data.

18. The optical device according to claim 17, wherein: The two-dimensional pupil data includes a two-dimensional pupil center.

19. The optical device according to claim 17, wherein: The eye segmentation data includes a segmentation of the eye of the user into a plurality of regions including one or more of a background region, a sclera region, a pupil region, or an iris region.

20. The optical device according to claim 15, wherein: The neural network was previously trained by: For each training input image in multiple training input images: providing the training input image to the neural network; Using the neural network to generate training network output data; Receive real data; Calculating error data based on the difference between the training network output data and the true data; as well as The neural network is modified based on the error data.

Images