Training of neural networks for 3D gaze prediction

Abstract

The present application describes a technique for generating 3D gaze predictions based on a deep learning system. In one example, the deep learning system includes a neural network. The neural network is trained using training images, which are generated by a camera and show the user's eyes when looking at stimulus points. Some of the stimulus points are in the plane of the camera. The remaining stimulus points are not in the plane of the camera. The training includes inputting a first training image associated with the stimulus points in the camera plane, and inputting a second training image associated with the stimulus points outside the camera plane. The training minimizes a loss function of the neural network based on the distance between at least one of the stimulus points and the gaze line.

Description

Technical Field

[0001] The present application relates to gaze detection systems and methods. In one example, the systems and methods rely on a deep learning system, such as a neural network, to detect three-dimensional (3D) gaze. Background Art

[0002] Interaction with computing devices is a very important activity in today's world. Computing devices, such as personal computers, tablet computers, and smart phones, are ubiquitous in daily life. In addition, wearable computing devices, such as wearable headsets (e.g., virtual reality headsets and augmented reality headsets), are becoming more and more popular. Systems and methods for interacting with these devices define how these devices are used and what these devices are used for.

[0003] The development of eye tracking technology has made it possible to use a person's gaze information to interact with a computing device. In other words, the location of the user's gaze on the display. This information can be used alone for interaction or in combination with contact-based interaction techniques (e.g., using a user input device such as a keyboard, mouse, touch screen or other input / output interface) for interaction.

[0004] Previously proposed interaction techniques using gaze information can be found in U.S. Patent 6,204,828, U.S. Patent Application Publication 20130169560, U.S. Patent 7,113,170, U.S. Patent Application Publication 20140247232, and U.S. Patent 9,619,020. The entire contents of these patents and applications are incorporated herein by reference.

[0005] In general, gaze-based interaction techniques rely on detecting the user's gaze at a gaze point. Existing systems and methods can accurately detect two-dimensional (2D) gaze. Recently, neural networks have been used to detect the 2D gaze.

[0006] Attempts have been made to extend existing techniques that rely on neural networks to three-dimensional (3D) gaze. However, the accuracy of the predictions is not as good as that for 2D gaze. Without accurate 3D gaze tracking, support for stereoscopic displays and 3D applications is significantly limited. Furthermore, even in the 2D domain, neural networks are typically trained for specific camera and screen configurations (e.g., image resolution, focal length, distance from the screen of the computing device, screen size, etc.). Therefore, whenever the configuration changes (e.g., different image resolution, different screen size, etc.), the neural network is no longer able to predict 2D gaze with acceptable accuracy. Retraining the neural network for the new configuration will become necessary. Summary of the invention

[0007] Embodiments of the present disclosure relate to three-dimensional (3D) gaze detection based on a deep learning system. In one example, a computer system accesses training images, the training images including a first set of training images and a second set of training images. The first set of training images shows a user's eyes associated with a gaze point within a camera plane. The second set of training images shows a user's eyes associated with a gaze point outside the camera plane. The computer system trains a neural network based on the training images. The training includes inputting a first training image into the neural network. The first training image belongs to the first set of training images. The training also includes inputting a second training image into the neural network. The second training image belongs to the second set of training images. The training also includes minimizing a loss function of the neural network based on a distance between a gaze point and a gaze line. The gaze point is associated with one of the first training image or the second training image. For the user's eyes shown in the one of the first training image or the second training image, a gaze line is predicted by the neural network.

[0008] Mention of these illustrative embodiments is not intended to limit or define the present disclosure, but rather to provide examples to aid in understanding the present disclosure.Additional embodiments are set forth in the Detailed Description section, and further description is provided there. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A further understanding of the nature and advantages of the various embodiments may be achieved by reference to the following drawings. In the drawings, similar components or features may have similar reference numerals. In addition, various components of the same type may be distinguished by following the reference numeral with a broken line and a second reference numeral to distinguish between similar components. Whenever a first reference numeral is used in the specification, such description may apply to any similar component having the same first reference numeral, regardless of the second reference numeral.

[0010] Figure 1 An eye tracking system according to one embodiment is shown.

[0011] Figure 2 An example of an image of an eye captured by an image sensor is shown according to one embodiment.

[0012] Figure 3 is a block diagram illustrating a special purpose computer system according to one embodiment.

[0013] Figure 4 An example of a wearable computing device implementing components of an eye tracking system is shown, according to one embodiment.

[0014] Figure 5 An example computing environment for predicting 3D gaze based on a deep learning system is shown, according to one embodiment.

[0015] Figure 6Example components of a deep learning system for predicting 3D gaze are shown, according to one embodiment.

[0016] Figure 7 An example network architecture for a neural network is shown according to one embodiment.

[0017] Figure 8 Example image normalization is shown according to one embodiment.

[0018] Fig. 9 An example 3D gaze prediction is shown according to one embodiment.

[0019] Fig.10 Predictions of 2D gaze vectors and distance corrections generated by a neural network are shown according to one embodiment.

[0020] Fig.11 An example process of predicting 3D gaze based on a deep learning system according to one embodiment is shown.

[0021] Fig.12 A strategy for using training images associated with a diversity of gaze point locations relative to the 2D plane of the camera that generated the training images is shown according to one embodiment.

[0022] Fig.13 An example process for training a neural network according to one embodiment is shown.

[0023] Fig.14 An example process for training a neural network using embedded calibration parameters according to one embodiment is shown. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure relate to three-dimensional (3D) gaze detection based on a deep learning system. In one example, a neural network is used. The neural network can be used independently of the camera and screen configuration. In other words, regardless of the specific camera, screen, and combination of camera and screen configurations, the neural network can be correctly trained to predict gaze information to support 3D gaze detection.

[0025] More specifically, the camera captures a two-dimensional (2D) image of a user who is looking at a point in 3D space. A rough distance between the camera and the user's eyes is estimated from the 2D image. The 2D image is normalized to produce a distorted image centered on the user's eyes at a high resolution, and a distorted image around the user's face at a low resolution. These distorted images are input to a neural network, which predicts a 2D gaze vector for each eye and a distance correction for the rough distance. The position of the eye in 3D space is estimated based on the rough distance and the distance correction and the position of the camera in 3D space. Based on a mapping function between 2D space and 3D space, each 2D gaze vector is mapped to a 3D gaze direction. The 3D position of the eye and its 3D gaze direction indicate the 3D gaze associated with the eye.

[0026] Because normalized 2D images are used and because the neural network predicts distance corrections and 2D gaze vectors, the neural network's predictions are independent of camera and screen configurations. Thus, the same trained neural network can be used in different eye tracking systems, including those integrated with different types of smartphones, tablets, laptops, wearable headsets (e.g., virtual reality and augmented reality headsets), as well as standalone eye tracking systems. Furthermore, because 3D gaze is detected, stereoscopic displays and 3D applications can be supported.

[0027] The training of the neural network generally relies on a plurality of training images that are different in terms of the position of the gaze point relative to the 2D plane of the camera used to capture the training image (e.g., for each camera, the X, Y plane perpendicular to the line of sight of the camera). Specifically, some of the training images show the user's eyes gazing at a gaze point located within the 2D plane of the camera, while other training images show the user's eyes gazing at a gaze point outside the 2D plane. During training, the neural network finds the gaze angle between the user's eye-camera and the user's eye-gaze point, as well as the distance from the eye to the camera. Because diversity is used, the neural network is able to correctly find the aforementioned gaze angle and distance. From the gaze angle, a gaze line (also called a gaze ray) is predicted and the distance between the gaze point and the gaze line (the distance from the gaze point to the gaze line) is calculated. The loss function of the neural network involves the distance from the gaze point to the gaze line. During training, the parameters of the neural network (e.g., the weights of the connections between nodes at different layers) are updated to minimize the loss function by minimizing the difference between the true distance from the gaze point to the gaze line and the predicted distance from the gaze point to the gaze line. For simplicity, from here on, images refer to 2D images unless otherwise specified.

[0028] These and other features of training and using a neural network for 3D gaze prediction independent of camera and screen configurations are further described below. This document next describes various components and configurations of an eye tracking system to provide a better understanding of 3D gaze prediction techniques.

[0029] Additionally, during training, the neural network learns "n" calibration parameters for the people shown in the training images. The designer of the neural network only needs to specify the value "n" and does not actually specify the calibration parameters. These parameters are inputs to the various layers of the neural network and become part of the training. Once training is complete, the end user operating the tracking system typically then performs a calibration process. As part of the calibration process, a calibration image (e.g., where the end user is requested to look at certain specific points) is input to the neural network, which then generates "n" calibration parameters for that user. These parameters are used for gaze predictions for the end user.

[0030] Figure 1 An eye tracking system 100 (also referred to as a gaze tracking system) according to one embodiment is shown. System 100 includes: illuminators 111, 112 for illuminating the eyes of a user; and image sensor 113 for capturing images of the eyes of the user. Illuminators 111, 112 may be, for example, light emitting diodes that emit light in an infrared band or a near-infrared band. Image sensor 113 may be, for example, a camera, such as a complementary metal oxide semiconductor (CMOS) camera or a charge coupled device (CCD) camera. The camera is not limited to an infrared camera or a depth camera or a light field camera. The shutter mechanism of the image sensor may be a rolling shutter or a global shutter.

[0031] The first illuminator 111 is arranged coaxially with the image sensor 113 (or close to the image sensor 113) so that the image sensor 113 can capture a lighted pupil image of the user's eye. Due to the coaxial arrangement of the first illuminator 111 and the image sensor 113, light reflected from the retina of the eye passes through the pupil back toward the image sensor 113, so that the pupil appears brighter than the iris surrounding the pupil in the image of the eye illuminated by the first illuminator 111. The second illuminator 112 is arranged non-coaxially with the image sensor 113 (or farther away from the image sensor 113) to capture a dim pupil image. Due to the non-coaxial arrangement of the second illuminator 112 and the image sensor 113, light reflected from the retina of the eye does not reach the image sensor 113, and the pupil appears darker than the iris surrounding the pupil in the image of the eye illuminated by the second illuminator 112. The illuminators 111, 112 can, for example, illuminate the eye in turn so that each first image is a lighted pupil image and each second image is a dim pupil image.

[0032] The eye tracking system 100 also includes circuitry 120 (e.g., including one or more processors) for processing images captured by the image sensor 113. The circuitry 120 may be connected to the image sensor 113 and the illuminators 111, 112, for example, via a wired connection or a wireless connection. In another example, the circuitry 120 in the form of one or more processors may be disposed in one or more stacked layers below the photosensitive surface of the image sensor 113.

[0033] Figure 2 An example of an image of an eye 200 captured by an image sensor 113 is shown. The circuit 120 may, for example, employ image processing (e.g., digital image processing) to extract features in the image. The circuit 120 may, for example, employ pupil center corneal reflection (PCCR) eye tracking to determine where the eye 200 is looking. In PCCR eye tracking, the processor 120 estimates the location of the center of the pupil 210 and the location of the center of the glint 220 of the eye 200. The glint 220 is caused by the reflection of light from one of the illuminators 111, 112. The processor 120 uses the glint 220 to calculate the user's position in space and uses the pupil 210 to calculate the position to which the user's eye 200 is directed. Since there is usually an offset between the optical center of the eye 200 and the fovea, the processor 120 performs calibration for the foveal offset to be able to determine where the user is looking. The gaze directions obtained from the left eye and the right eye can then be combined to form a combined estimated gaze direction (or viewing direction). As will be described below, many different factors may influence how the left eye's gaze direction and the right eye's gaze direction should be weighted relative to each other when forming this combination.

[0034] In reference Figure 1 In the described embodiment, illuminators 111, 112 are arranged within an eye tracking module 110, which is arranged below the display viewed by the user. This arrangement is by way of example only. It will be appreciated that any greater or lesser number of illuminators and image sensors may be employed for eye tracking, and that these illuminators and image sensors may be distributed in many different ways relative to the display viewed by the user. It will be appreciated that the eye tracking scheme described in the present disclosure may, for example, be employed for remote eye tracking (e.g., in a personal computer, in a smartphone, or integrated in a vehicle), or for wearable eye tracking (e.g., in virtual reality glasses or augmented reality glasses).

[0035] Figure 3 is a block diagram illustrating a special purpose computer system 300 in which embodiments of the present disclosure may be implemented. This example illustrates a special purpose computer system 300 that may be used, for example, in whole, in part, or with various modifications to provide the functionality of the various components described herein.

[0036] The special purpose computer system 300 is illustrated as including hardware elements that may be electrically coupled via a bus 390. These hardware elements may include one or more central processing units 310, one or more input devices 320 (e.g., a mouse, keyboard, eye tracking device, etc.), and one or more output devices 330 (e.g., a display device, a printer, etc.). The special purpose computer system 300 may also include one or more storage devices 340. By way of example, the storage device 340 may be a disk drive, an optical storage device, a solid-state storage device (e.g., a random access memory (RAM) and / or a read-only memory (ROM)), which may be programmable, flash-updatable, and / or the like.

[0037] The special purpose computer system 300 may additionally include a computer readable storage medium reader 350, a communication system 360 (e.g., a modem, a network card (wireless or wired), an infrared communication device, a Bluetooth TM Device, cellular communication device, etc.) and working memory 380, which may include RAM and ROM devices as described above. In some embodiments, the special-purpose computer system 300 may also include a processing acceleration unit 370, which may include a digital signal processor, a special-purpose processor and / or the like.

[0038] Figure 4 Shown to achieve the combination Figure 1-2 The wearable computing device 400 may include some or all of the above components of the eye tracking system described above. The wearable computing device 400 may be a virtual reality (VR) headset or an augmented reality (AR) headset that can be worn by a user. As shown, the wearable computing device 400 includes a set of lenses 410 (e.g., Fresnel lenses), a set of cameras 420, a set of heat reflectors 430 (e.g., Figure 12-14 As further shown, the set of hot mirrors includes two hot mirrors for each eye in various embodiments) and a set of displays 440. The camera 420 may include Figure 1 image sensor 113. Although Figure 4 Although not shown in the figure, the wearable computing device 400 may also include a set of illuminators and processing circuits. These and other components may be integrated into the housing 450 of the wearable computing device 400. Thus, once the user mounts the wearable computing device 400 on his / her head, the set of lenses 410 will be relatively close to the user's eyes, and the set of displays will be relatively far from the user's eyes, and the remaining components may be located between the lenses and the displays. The arrangement of these components allows the user's gaze point 460 to be detected in a three-dimensional virtual or real space.

[0039] The present invention further describes the use of a deep learning system for 3D gaze prediction. For ease of explanation, the system is described in conjunction with a camera, a screen, and a user's eyes (e.g., a camera captures an image, and some or all of the captured image shows the user's eyes). The deep learning system can be used with any camera and screen configuration that uses visible light, passive infrared light, active bright pupil (BP) infrared light, etc. for eye tracking. However, embodiments of the present disclosure are not limited thereto.

[0040] For example, these embodiments are equally applicable to eye tracking systems that use one camera for each user's eye, such as in the context of a virtual reality headset or an augmented reality headset. Changes to how to implement a deep learning system for single-camera tracking per eye will be apparent to those skilled in the art in light of the present disclosure. For example, instead of inputting two distorted images, each focused on one of the user's eyes, only one distorted image of the user's eye associated with the camera may be used. In addition, the distorted image around the user's face may not be input. During training, the neural network learns to predict a distance correction from the distorted image, rather than predicting such a correction based on the distorted image around the user's eye and the user's face. The system can then output a 2D gaze vector for the user's eye associated with the camera and a distance correction for the rough camera-to-eye distance. In addition, in the case of a VR or AR device, the rough distance may be predefined and does not need to be estimated based on the image produced by the camera.

[0041] Figure 5 An exemplary computing environment for predicting 3D gaze based on a deep learning system according to one embodiment is shown. In general, 2D gaze information represents an X, Y gaze position on a 2D plane. In contrast, 3D gaze information represents not only an X, Y gaze position, but also a Z gaze position. In one example, a 3D gaze can be characterized by an eye position as an origin in a 3D space and a direction of the 3D gaze from the origin.

[0042] like Figure 5As shown, a user 510 operates a computing device 520 that tracks a 3D gaze 512 of the user 510. To this end, the computing device 520 in one example communicates with a server computer 530 that hosts (or hosts) a deep learning system 532. The computing device 520 sends a 2D image 550 to the server computer 530 over a data network (not shown), the 2D image 550 showing the user's eyes when the user 510 is looking. The server computer 530 inputs the 2D image 550 to the deep learning system 532, which in response predicts a 3D gaze 512. The server computer 530 sends information 560 about the 3D gaze 512 (e.g., 3D eye position and 3D gaze direction) back to the computing device 520 over the data network. The computing device 520 uses the information 560 to provide a 3D gaze-based computing service to the user 510.

[0043] although Figure 5 A computer server 530 hosting a deep learning system 532 is shown, but embodiments of the present disclosure are not limited thereto. For example, a computing device 520 may download the code of the deep learning system 532 and host an instantiation of the deep learning system 532. In this way, the computing device 520 locally predicts the 3D gaze 512 based on the instantiation and does not need to send the 2D image 550 to the server computer 530. In this example, the server computer 530 (or some other computer system connected thereto via a data network) may train the deep learning system 532 and provide an interface (e.g., a network interface) to download the code of the deep learning system 530 to the computing devices, thereby hosting the instantiation of the deep learning system 530 on these computing devices.

[0044] In one example, computing device 520 includes camera 522, screen 524, and 3D gaze application 526. Camera 522 generates 2D image 550, which is a 2D representation 540 of a user's face. The 2D image 550 shows the user's eyes when gazing in 3D space. A 3D coordinate system 528 may be defined in association with camera 522. For example, camera 522 is at the origin of the 3D coordinate system 528. The X and Y planes may be planes perpendicular to the line of sight of camera 522. In contrast, 2D image 550 has a 2D plane that may be defined around 2D coordinate system 542, which is in the same position as 2D representation 540 of the user's face. Camera 522 is associated with a mapping between 2D space and 3D space (e.g., between two coordinate systems 542, 528). In one example, the mapping includes a back-projection matrix for the camera 522 and is stored locally on the computing device 520 (eg, in a storage location associated with the 3D gaze application 526 ).

[0045] Screen 524 can, but need not, be located within the X, Y plane of camera 522 (if not, the relative position between the two is determined based on the configuration of computing device 520). 3D gaze application 526 can process 2D image 550 for input to deep learning system 530 (whether located remotely or locally on computing device 520), and can process information 560 about 3D gaze to support stereoscopic display (if screen 524 also supports it) and 3D applications (e.g., 3D control and manipulation of objects displayed on screen 524 based on information 560).

[0046] Figure 6 Exemplary components of a deep learning system 600 for predicting 3D gaze are shown according to one embodiment. As shown, the deep learning system includes an eye detector 620, a distance estimator 630, an image generator 640, a neural network 650, and a 3D gaze generator 660. Some or all of these components may be implemented as dedicated hardware and / or software modules (e.g., specific computer-readable instructions) hosted on dedicated or general-purpose processing hardware.

[0047] As shown, a 2D image 610 is input to an eye detector 620. For example, the 2D image 610 is generated by a camera. In response, the eye detector 620 detects the user's eyes in the 2D image 610 and outputs information about the position 622 of the eyes in the image (e.g., the position of the center of the pupil within the 2D plane of the 2D image 610). In one example, the eye detector 620 is implemented as a machine learning algorithm that is trained to perform eye detection. Many machine learning algorithms are possible and are well known to those skilled in the art.

[0048] The eye positions 622 and the 2D image 610 are input to a distance estimator 630. In response, the distance estimator 630 generates an estimated distance 632, e.g., a rough distance. To this end, the distance estimator 630 projects the eyes detected in the 2D image 610 into a 3D coordinate system centered on the camera. This projection uses the camera's 2D-3D space mapping. The distance estimator 630 searches for eye projections in the 3D space at which the interocular distance (ID) is approximately the average interocular distance (ID) for a person (e.g., 63 mm). The distance between the camera and each of these eye projections may be referred to as a projection distance. The rough distance is set based on the average interocular distance (ID) for a person and the projection distance.

[0049] To illustrate, let d rough represents the coarse distance, and K represents the intrinsic camera matrix, and e left and e rightrepresents the eyes detected in the 2D image 610. K is the component of the camera's 2D-3D space mapping. The projected left eye e′ left，proj and right eye e′ right，proj Calculated as e′ left，proj =K -1 e left and e′ right，proj =K -1 e right , and represents the eye vector projected from the eye detected in the 2D image 610 into the 3D coordinate system centered on the camera, as Figure 8 These projected eyes are normalized to e left，proj =e′ left，proj / ||e′ left，proj || and e right，proj =e′ right，proj / ||e′ right，proj ||. The rough distance is calculated as d rough =ID / ||e left，proj -e right，proj ||.

[0050] The image generator 640 receives the 2D image 610 and in response generates a distorted image 642 around the user's eyes and a distorted image 644 around the user's face. In one example, the image generator 640 accesses a first predefined distance s in pixels of the user's eye distorted image 642 and a second predefined distance s in pixels of the user's face distorted image 644 from a local memory. These predefined distances s are different inter-eye distances s in pixels so that the user's eye distorted image 642 and the user's face distorted image 644 are at different projected positions in the 3D space and have different resolutions (by using a first predefined distance s that is smaller than the second predefined distance s, the resolution of the user's eye distorted image 642 is higher than the resolution of the user's face distorted image 644). The image generator 640 uses these predefined distances s to generate the distorted images so that they are normalized relative to the camera. Specifically, each warped image represents a projection of the 2D image 610 into a 3D coordinate system centered on the camera, a rotation of the 2D image 610 about the X-axis (so that any head tilt is rotated to a horizontal position), a scaling of the 2D image 610 (based on a predefined distance s), and a warp (so that the user's eyes and the user's face are centered in each warped image and are not geometrically skewed). In other words, each warped image represents a normalized image that can be input to the neural network 650, wherein the normalization removes the dependency of the input on the camera configuration (e.g., image resolution, focal length, distance to the screen, camera type (e.g., pinhole camera and non-pinhole camera), etc.).

[0051] To illustrate, the image generator 640 generates a rotation matrix R that rotates a point from a real 3D space (e.g., a 3D coordinate system centered on the camera) to a normalized 3D space (e.g., a 3D coordinate system also centered on the camera but rotated relative to the real 3D space so that the vector between the user's eyes in the 2D image 610 becomes horizontal), as shown in FIG. Figure 8 As further shown. The image generator 640 also generates a scaling matrix M based on the predefined distance s (generating a matrix M for the user's eyes and another matrix M for the user's face). For example, M is generated as a diagonal matrix, where M=dig([1, 1, f]), where f is a focal length selected to make the interocular distance between the user's eyes equal to the predefined distance s. The transformation matrix T is defined based on the intrinsic matrix K, the rotation matrix R, and the scaling matrix M, and is used to normalize the 2D image 610 into a normalized image (based on the M matrix for the user's eyes and for the user's face). For example, the transformation matrix T is represented as T=MRK -1, and is applied to the 2D image 610 to generate a first normalized image for the user's eye at a first predefined distance s and a second normalized image for the user's face at a second predefined distance s. Each of these normalized images is a rotation and projection of the 2D image 610 from the real 3D space to the normalized 3D space, wherein the rotated and projected image is at a predefined distance s within the normalized 3D space. The image generator 640 generates a user eye distorted image 642 by distorting the normalized user eye image using bilinear interpolation and cutting out a W×H area centered on one eye of the user. Similarly, the image generator 640 generates a user face distorted image 644 by distorting the normalized user face image using bilinear interpolation and cutting out a W×H area centered on the midpoint between the user's eyes.

[0052] In addition, one of the plurality of eye distorted images 642 is mirrored so that the resulting mirrored image is aligned with the other (non-mirrored) eye distorted image. For example, the distorted image around the left eye is mirrored. As a result, the inner canthus of the left eye is aligned with the inner canthus of the right eye, and the outer canthus of the left eye is aligned with the outer canthus of the right eye, as shown in FIG. Fig.10 This is shown in Figure 1. With this alignment, the eye distorted images input to the neural network are identical in orientation, thereby simplifying the architecture and training of the neural network.

[0053] The eye distortion image 642 and the face distortion image 644 are input to a neural network 650. In one example, the 2D image 610 is not input to the neural network 650. In response, the neural network 650 outputs a distance correction 652 and a 2D gaze vector 654 (one for each eye). The distance correction 652 is a multiplicative correction factor that can be multiplied by the estimated distance 632 to correct such estimated distance and produce a corrected distance. Each 2D gaze vector 654 has a gaze origin (e.g., the center of the pupil or glint) and a gaze direction within the corresponding eye cropped image (the origin and gaze direction are within the 2D plane of the image and can be traced back to real 3D space).

[0054] In one example, the neural network 650 is a convolutional neural network that includes multiple sub-networks (e.g., parallel branches along the neural network 650). These sub-networks (and equivalently, the entire convolutional neural network) can be trained collaboratively. Figure 7 An example of a network architecture is shown in FIG. Fig.12 and Fig.13An example of training is described. Each distorted eye image 642 is input to a subnetwork. These images 642 can be input in concert (e.g., two images are input in parallel) or independently of each other (e.g., one image is input at a time, where the first subnetwork will predict the gaze direction from the image). In response, the subnetwork generates a 2D gaze vector 654 for each eye (e.g., corresponding to the user's eyes shown in the input image). Each 2D gaze vector can be represented as a 2D gaze origin o 2D (e.g., the 2D position of the user's eyes in the image plane) and the 2D gaze direction d 2D The face warped image 644 is also input to the subnetwork, which responds by giving the distance correction c 652. Thus, the output from the neural network consists of five parts: the 2D gaze origin o for each eye 2D and 2D gaze direction d 2D , and the distance correction c.

[0055] The distance correction 652, the estimated distance 632, and the 2D gaze vector 654 are input to a 3D gaze generator 660. In response, the 3D gaze generator 660 generates and outputs a 3D gaze 662. In one example, the 3D gaze 662 includes the 3D gaze direction of each eye of the user (which can be expressed in 3D real space) and the 3D position of the user's eyes (which can also be expressed in 3D real space).

[0056] For illustration, the corrected distance d corr is generated as d corr =d rough ×c. Referring to one of the eyes and its 2D gaze vector, the 3D gaze generator 660 calculates its 3D gaze based on the corrected distance and the 2D-3D space mapping (eg, correlation matrix). For example, the 3D gaze origin o in the normalized 3D space 3D，N is calculated as o′ 3D，N =M -1 o 2L And o 3D，N =o′ 3D，N / ||o′ 3D，N ||d corr To generate the 3D gaze direction in the normalized 3D space, first generate the normalized basis vectors (X, Y, Z), where Z′=M -1 o 2D , Y′=[0,1,0] T ×Z′ and X′=Y′×Z′, and X=X′ / ‖X′||, Y=Y′ / ||Y′|| and Z=Z′ / ||Z′||. The normalized 3D gaze direction d 3D，N Produced as and d 3D，N =d′3D，N / ||d′ 3D，N ||. The 3D gaze origin and the 3D gaze direction are mapped from the normalized 3D space to the real 3D space based on the rotation matrix R. For example, the 3D gaze origin O in the real 3D space 3D is calculated as o 3D =R -1 o 3D，N Similarly, the 3D gaze direction d in the real 3D space 3D is calculated as d 3D -R -1 d 3D，N .

[0057] Figure 7 An example network architecture of a neural network according to one embodiment is shown. Figure 7 As shown, three images 710-714 are input to the network: a face warped image 710 at low resolution, a left eye warped image 712 at high resolution, and a right eye warped image 714 at high resolution (where "low" and "high" are relative terms, e.g., "low resolution" means lower than "high resolution"). The eye warped images 712 and 714 are centered on the eye detection xe left orright , scaled to an interocular distance of s=320 pixels and cropped to 224×112 pixels. The right eye image 714 is mirrored 716 by modifying the rotation matrix R. This provides a consistent appearance for the network. The face warped image 710 is centered at the midpoint between the eye detections, scaled to equal 84 pixels and cropped to 224×56 pixels.

[0058] The network includes different convolutional neural networks (CNNs) 720-726 for eyes and face (illustrated as including CNN 722 for the left eye and CNN 724 for the right eye, which have equal weights and include CNN 720 for the face). Both are convolutional parts of ResNet-18, which are similar to those described in "Deep residual learning for image recognition" published by K. He, X. Zhang, S. Ren, and J. Sun in CoRR, abs / 1512.03385, 2015, the content of which is incorporated herein by reference. The outputs of all CNNs 720-726 for eyes and face are cascaded into a cascade layer 730 and fed to a fully connected module 740, which predicts the distance correction c.

[0059] The outputs from each eye CNN 722 and CNN 724 are concatenated together with a set of "n" individual calibration parameters and distance correction c. The concatenation for the left and right eyes is achieved through concatenation layers 732 and 734, respectively. The combined feature vectors from each cascade are fed to fully connected modules (illustrated as fully connected modules 742 for the left eye and fully connected modules 744 for the right eye).

[0060] The fully connected modules 740-744 can be described as: FC(3072)-BN-ReLU-DO(0.5)-FC(3072)-BN-ReLU-DO(0.5)-FC({4,1}). The final output is either the 2D gaze origin and the 2D gaze direction, or the distance correction c.

[0061] There is no need to provide "n" correction parameters to the distance estimation component (eg, cascade layer) because it is generally difficult to detect distance errors from calibration data acquired at one distance (which is typically available).

[0062] The network was trained to minimize the average minimum misalignment distance between the predicted fixation line and the ground-truth stimulus point (as shown in the following figures). Fixation origins that fell outside the eye image and distance corrections exceeding 40% in any direction were also penalized.

[0063] Figure 8 An exemplary image normalization according to one embodiment is shown. As shown, the real 3D space 810 is centered on the camera that generates the real image 830. The 3D space 810 can be defined as a 3D coordinate system. The normalized 3D space 820 can also be centered on the camera and generated from the real 3D space 810 based on the rotation matrix R. For example, the normalized 3D space 820 corresponds to the rotation of the real 3D space 810 along one of the axes (e.g., the Y axis) of the real 3D space 810. The center between the user's eyes forms a straight line. The straight line is parallel to an axis (e.g., the X axis) in the normalized 3D space 820. The normalized image 840 (illustrated as corresponding to the left eye distorted image) is projected from the real image 830 into the normalized 3D space (illustrated as being a certain distance away from and parallel to the real image 830). In the case where the normalized image 840 corresponds to the position of the average human ID of 63mm for the interocular distance between the two eyes, the distance between the camera and the center of the eyes in the normalized image 840 is a rough distance.

[0064] Fig. 9 An exemplary 3D gaze prediction according to one embodiment is shown. As shown, using Figure 8 The normalized 3D space 820, although this prediction can also be achieved by using a combination of Figure 6The inverse of the rotation matrix R described is mapped to the real 3D space 810. The 3D gaze origin 910 is derived from the 2D gaze origin 920 based on the inverse of the scaling matrix M and the corrected distance 930. The 2D gaze origin 920 and the 2D gaze direction 940 represent the 2D gaze vector within the plane of the normalized image 840. The 3D gaze direction 950 is based on the combination of Figure 6 The inverse of the scaling matrix M described is derived from the 2D gaze origin 920 and the 2D gaze direction 940 .

[0065] During training, a 3D gaze line (or 3D gaze ray) is projected from a 3D gaze origin 910 along a 3D gaze direction 950. A gazed stimulus point 960 is known (e.g., a known gaze point). A distance 970 (e.g., the shortest distance) between the stimulus point 960 and the 3D gaze line is a distance parameter of the loss function. The loss function is minimized by minimizing this distance 970 (e.g., if the neural network correctly predicts the 3D gaze, the distance 970 may be zero and the stimulus point 960 will fall on the 3D gaze line).

[0066] Fig.10 Predictions of 2D gaze vectors and distance corrections generated by a neural network are shown, according to one embodiment. Fig.10 Three images 1010, 1020, 1030 are shown as input to a neural network. Images 1010, 1020 are eye-distorted images, while image 1030 is a face-distorted image. In addition, image 1020 (e.g., of the left eye) is mirrored so that the two images 1010, 1020 are aligned (e.g., the inner canthus of the left eye is aligned with the inner canthus of the right eye, and the outer canthus of the left eye is aligned with the outer canthus of the right eye).

[0067] The predicted 2D gaze vector is overlaid on images 1010, 1020. As shown in the overlaid image of image 1010 (and similarly, as shown in the overlaid image of image 1020), the 2D gaze vector has a gaze origin 1012 (e.g., a flashing spot on the right eye) and a gaze direction 1014. The estimated and corrected distance 1032 is overlaid in image 1030. As shown, the estimated distance is 653 mm and the corrected distance is 475 mm, which represents a predicted distance correction of approximately 72%.

[0068] Fig.11An exemplary process for predicting 3D gaze based on a deep learning system, such as deep learning system 600, according to one embodiment is shown. An eye tracking system is described as performing the operations of the exemplary process. In one example, the eye tracking system hosts the deep learning system. In another example, a remote computer system hosts the deep learning system, and the eye tracking system interacts with the remote computer system over a data network to provide 2D images and receive 3D gaze predictions. In yet another example, the deep learning system is distributed between the eye tracking system and the remote computer system (e.g., the remote computer system may host the neural network, and the eye tracking system may host the rest of the deep learning system).

[0069] Instructions for performing the operations of the example processes may be stored as computer-readable instructions on a non-transitory computer-readable medium of the eye tracking system. Once stored, the instructions represent a programmable module that includes code or data that can be executed by (one or more) processors of the eye tracking system. Execution of these instructions configures the eye tracking system to perform the specific operations shown in the figures and described herein. Each programmable module is combined with a processor to represent a means for performing (one or more) corresponding operations. Although these operations are shown in a particular order, it should be understood that no particular order is required and one or more operations may be omitted, skipped and / or rearranged.

[0070] The exemplary process begins at operation 1102, where an eye tracking system receives an image (e.g., a 2D image). In one example, the image is generated by a camera associated with the eye tracking system (e.g., a camera integrally formed with the system or interfaced with the system). The image shows the user's eyes (e.g., when the camera is associated with both eyes of the user, the image shows both eyes; when the camera is associated with one eye of the user, such as in the case of a VR headset, the image shows only that eye of the user).

[0071] At operation 1104, the eye tracking system generates a distorted image around the user's eyes from the image. For example, the eye tracking system detects the user's eyes in the image and estimates a rough distance between the camera and the user's eyes detected in the image. The eye tracking system also projects the image into a 3D space based on the camera's intrinsic matrix K, rotates the projected image based on a rotation matrix R, and scales the rotated image based on a scaling matrix M. The scaling matrix M is generated based on a predefined distance s in pixels and an average interocular distance of a person. The eye tracking system then distorts the scaled image using bilinear interpolation and crops the distorted image around the user's eyes based on a predefined area (e.g., an area with a predefined width and height). The image has a first image resolution that depends on the predefined distance s.

[0072] In operation 1106, the eye tracking system generates a second warped image around the second user's eye from the image. If the image shows the second user's eye, then this operation is performed, otherwise the operation is skipped. Operation 1106 is similar to operation 1104. The second warped image shows the second user's eye and has the first image resolution.

[0073] At operation 1108, the eye tracking system generates a third warped image around the user from the image. This operation is performed if the image shows a second user eye, otherwise it is skipped. Operation 1106 is similar to operation 1104, where scaling is performed based on a second predefined distance s (and thus a second scaling matrix M) to obtain a second image resolution. The third warped image shows the user's face or at least two eyes of the user and has a second image resolution that is lower than the first image resolution.

[0074] At operation 1110, the eye tracking system inputs the distorted image of the user's eye to the neural network. If two other distorted images are generated (e.g., a distorted image of a second user's eye and a distorted image of the user's face), they are also input to the neural network. The neural network has been trained and predicts: (i) a distance correction c, and (ii) a 2D gaze origin and a 2D gaze direction for each eye in the associated distorted eye image. In general, the distance correction c is used to correct the estimated coarse distance and is predicted based on the distorted image and (if any) the second and third distorted images. The 2D gaze origin and 2D gaze direction of the user's eye (again, the second user's eye) are generated from a distorted image that is different from the second and third distorted images (again, from a second distorted image that is different from the distorted image and the third distorted image).

[0075] In one example, the neural network is also trained based on "n" calibration parameters embedded in the neural network. During calibration of the eye tracking system, the system generates multiple calibration images by instructing the user to gaze at a number of known gaze points. These calibration images are also normalized and distorted and input into the neural network. The "n" calibration parameters are adjusted to minimize the loss function of the neural network. The minimization uses the known gaze points as the ground truth. Once the images are received by the eye tracking system at operation 1102 and processed at operations 1106-1108, the predictions of the neural network at operation 1110 use the "n" calibration parameters.

[0076] The eye tracking system receives the distance correction c from the neural network at operation 1112. The eye tracking system also receives the 2D gaze origin and 2D gaze direction of the user's eye and, if applicable, the 2D gaze origin and 2D gaze direction of the second user's eye from the neural network.

[0077] At operation 1114, the eye tracking system generates a corrected distance between the user's eye and the camera by at least updating the estimated coarse distance based on the distance correction c. For example, the distance correction c is a correction factor, and the eye tracking system multiplies the estimated coarse distance by the factor to generate the corrected distance.

[0078] In operation 1116, the eye tracking system generates 3D gaze information of the user's eye from the corrected distance and its 2D gaze origin and 2D gaze direction. The 3D gaze information includes the position of the user's eye in 3D space (e.g., the 3D gaze origin) and the 3D gaze direction starting from the user's eye. In one example, the 3D gaze origin and the 3D gaze direction are derived from the corrected distance, the 2D gaze origin, and the 2D gaze direction based on the scaling matrix M and the rotation matrix R (or their inverse matrices). This operation can also be repeated to generate the 3D gaze origin and 3D gaze direction of the second user's eye based on the corrected distance and its 2D gaze origin and 2D gaze direction. In this case, the 3D gaze information includes the 3D gaze origin and 3D gaze direction of both eyes of the user.

[0079] At operation 1120, the eye tracking system provides 3D gaze information generated for the user's eye and, where appropriate, for the second user's eye to a 3D gaze application. The 3D gaze application uses the 3D gaze information to support stereoscopic display and 3D applications (3D control and manipulation of objects displayed on the screen).

[0080] A neural network of the deep learning system is trained based on the training images to predict a distance correction and a 2D gaze vector (e.g., a 2D gaze origin and a 2D gaze direction) for each user's eye. In general, training is repeated for these training images by backpropagation (e.g., backpropagation using gradient descent) to minimize a loss function and update the parameters of the neural network accordingly. Since the neural network should predict two outputs (a distance correction and a 2D gaze vector), it is not sufficient to have training images showing only the user's eyes looking at a gaze point within a 2D plane associated with a camera that captured the training images. Instead, for proper training, a diversity of the positions of the gaze points relative to the 2D plane is necessary. In combination with Fig.12 Further describe the problems and solutions of this training.

[0081] Fig.12 A strategy for using training images according to one embodiment is shown, which are associated with a diversity of gaze point locations relative to the 2D plane of the camera that generated the training images. Fig.12A single user eye is discussed. However, training images showing two user eyes are equally applicable. The user eye captured in the training image is also referred to herein as the "displayed user eye". The camera used to generate the training image is also referred to herein as the "training camera". As shown, camera 1210 generates training images showing user eyes. Each training image is a 2D image captured when the user's eye is looking at a gaze point.

[0082] For example, the first training image shows the user when the user is looking at gaze point 1220. If the first training input is the input sent to the neural network during neural network training, the neural network may incorrectly find the gaze direction of the user's eyes to gaze point 1220 and the distance from the user's eyes to camera 1210. Fig.12 As shown, there can be multiple solutions for gaze direction and distance ( Fig.12 1230) is a first solution. The neural network may predict that the user's eyes (illustrated in the solution as element 1230) are looking at a first angle "a1" 1232 relative to the gaze point 1220 and are separated from the camera 1210 by a first distance "d1" 1234. In a second solution, the neural network may predict that the user's eyes (illustrated in the solution as element 1240) are looking at a second angle "a2" 1242 relative to the gaze point 1220 and are separated from the camera 1210 by a second distance "d2" 1244. Both solutions are possible, but only one is correct. Therefore, the neural network may predict an incorrect solution, and thus the training may be incorrect and cause the trained neural network to produce inaccurate predictions.

[0083] To avoid this possibility, a strategy of gaze point position diversity is used. Under this strategy, a second training image of the user's eyes is additionally used. In this second training image, either the gaze angle or the distance from the user's eyes to the camera is changed. Therefore, when two training images are used, a set of possible solutions is reduced to a single solution (e.g., either the first solution in the above example or the second solution in the above example).

[0084] Many different methods can achieve this diversity. In general, the training images can include two sets of training images. The first set of training images are training images of the user's eyes looking at a gaze point within the 2D plane of the camera. The second set of training images are training images of the user's eyes looking at a gaze point outside the 2D plane of the camera. In this way, when various training images are input to the neural network, the neural network learns to find the correct solution.

[0085] Fig.12A specific method is shown. As shown in the figure, when the user's eyes are fixed on the second gaze point 1250 along the same gaze angle as in the first training image, a second training image of the user's eyes is generated. In this way, the only possible solution is the first solution (for example, the first angle "a1" 1232 and the first distance "d1" 1234). Another method is to use the same gaze point 1220 for the second training image, but change the distance between the user's eyes and the camera 1210 (for example, by moving the camera to make it closer to or farther away from the user's eyes while the user's eyes remain fixed on the gaze point 1220). Yet another method is to maintain the same user eye-to-camera distance, introduce a second gaze point (not necessarily at the same distance as the user's eyes), and then change the distance between the user's eyes and the camera 1210. Fig.12 The method of the embodiment of the present invention is to generate a second training image when the user's eye is fixed on the second fixation point (at the same fixation angle in the specific method shown) and generate a second training image when the user's eye is fixed on the second fixation point. Of course, any combination of these different methods can be used to generate multiple training images for different user eyes, different fixation angles, and different user eye-to-camera distances. The training images can be generated using the same camera (or the same camera configuration for multiple cameras) but are not necessarily required, because the training is not dependent on a specific camera or screen configuration.

[0086] Once the training images are generated, they are input to the neural network for training. Specifically, the neural network predicts the gaze angle and the distance from the user's eyes to the camera from these training images. A loss function can be defined based on the gaze angle and the distance from the user's eyes to the camera. For clarity, the first training image described above is considered here (e.g., the first training image generated when the user's eyes are looking at the gaze point 1220). Based on this training image (and the second training image), the neural network predicts a first solution (e.g., a first angle "a1" 1232 and a first distance "d1" 1234). Let us assume that the first angle "a1" 1232 and the first distance "d1" 1234 are the actual gaze angle and distance, which are known in training (e.g., they are the basic real measurements that can be stored in the training tags associated with the first training image). Let us also assume that the prediction is not completely accurate (e.g., the predicted angle deviates from the first angle "a1" 1232, and the predicted distance deviates from the first distance "d1" 1234). The loss function includes a distance term and an angle term. The distance term is the difference between the predicted distance and the ground truth distance (e.g., the actual first distance "d1" 1234). The angle term is the difference between the predicted gaze angle and the ground truth gaze angle (e.g., the actual first angle "a1" 1232). The purpose of training is to update the parameters of the neural network to minimize its loss function, wherein minimizing the loss function includes minimizing the angle term and the distance term so that the predicted angle is as close as possible to the ground truth angle and the predicted distance is as close as possible to the ground truth distance.

[0087] In one example, a single distance term may be used instead of the angle term and the distance term. For example, a predicted gaze line is generated at a predicted distance from the camera and has a predicted gaze angle. The distance (e.g., the shortest distance) between the gaze point 1220 and the predicted gaze line is measured (this is in Fig. 9 970). If the prediction is completely accurate, then the distance from the fixation point to the predicted fixation line will be zero. However, in other cases, the loss function is minimized (and the neural network parameters are updated) by minimizing the distance from the fixation point to the predicted fixation line so that the predicted fixation line is as close to the fixation point 1220 as possible.

[0088] In this example, the loss function may also include a penalty term. More specifically, when the origin of the predicted gaze line falls outside the corresponding training image (referring back to the previous example, if "d1" 1234 is large enough that the eye 1230 falls outside the first training image), a penalty is added to the loss function (e.g., the penalty has a predefined value). Similarly, if the predicted distance correction exceeds a certain threshold in either direction (e.g., more than 40% in either direction), the same penalty or another penalty is added to the loss function.

[0089] Fig.13 An example process for training a neural network according to one embodiment is shown. Fig.14 Additional example flows for training are also shown, where such training uses embedded calibration parameters according to one embodiment. A computer system is described as performing the operations of the example flows. In one example, the computer system executes code to train and store a neural network. Once training is complete, the computer system can receive images from an eye tracking system and use the neural network to respond with 3D gaze predictions. Additionally or alternatively, the eye tracking system can download the code for the neural network from the computer system.

[0090] Instructions for performing the operations of the illustrated process may be stored as computer-readable instructions on a non-transitory computer-readable medium of a computer system. Once stored, the instructions represent a programmable module that includes code or data that can be executed by (one or more) processors of the computer system. The execution of these instructions configures the computer system to perform the specific operations shown in the accompanying drawings and described herein. Each programmable module is combined with a processor to represent a means for performing (one or more) corresponding operations. Although these operations are set forth in a particular order, it should be understood that this particular order is not required, and one or more operations may be omitted, skipped, and / or rearranged.

[0091] Fig.13The exemplary process of FIG. 13 begins at operation 1302, where a computer system accesses training images, the training images comprising a first set of training images and a second set of training images. Some or all of the first set of training images show the user's eyes associated with a gaze point within a camera plane. Some or all of the second set of training images show the user's eyes associated with a gaze point outside the camera plane. In this way, a diversity of training images is obtained, the training images showing the user's eyes looking at a gaze point within and outside the plane.

[0092] At operation 1304, the computer system trains the neural network based on the training images. In general, training includes updating the parameters of the neural network (e.g., the weights of the connections between the nodes of each layer of the neural network) to minimize the loss function of the neural network. This loss function can be used in conjunction with Fig.12 The angle terms and / or distance terms explained. In one example, operation 1304 includes two additional operations 1306 , 1308 .

[0093] At operation 1306, the computer system inputs a first training image and a second training image from the training images to the neural network. The first training image belongs to a first set of training images. The second training image belongs to a second set of training images. In one specific example, the first training image shows the user's eyes when the user's eyes are fixed on a fixation point at a certain fixation angle, and the second training image shows the user's eyes when the user's eyes are fixed on another fixation point at the certain fixation angle. In another specific example, the first training image and the second training image show the user's eyes when the user's eyes are fixed on the fixation point at a certain fixation angle. In this example, the first training image corresponds to a first distance between the camera and the user's eyes, and the second training image corresponds to a second distance between the camera and the user's eyes. In yet another specific example, the first training image shows the user's eyes when the user's eyes are fixed on the fixation point at a certain fixation angle, and the second training image shows the user's eyes when the user's eyes are fixed on another fixation point at a different fixation angle. In these examples, the first and second training images may be input in pairs.

[0094] As combined Fig.13 As used herein, the training images refer to a set of training images, which include a distorted image showing one eye of a person, and optionally a distorted image showing the other eye of a person and a distorted image of a person's face. In other words, the images used in the training are the same type of images that will be used at the end of the training (such as combined Fig. 9 ), and can be generated based on the same operations such as projection, rotation, scaling and shearing.

[0095] At operation 1308, the computer system minimizes a loss function of the neural network based on the distance between the gaze point and the gaze line. The gaze point is associated with one of the first training image or the second training image. The gaze line of the user's eye is predicted by the neural network from the one of the first training image or the second training image. Operations 1306-1308 are repeated for many, if not all, training images to complete training of the neural network.

[0096] Steering Fig.14 , the training of the neural network may also include calibration parameters, so that when the neural network predicts gaze information (as described above), the prediction is based on these parameters. The designer of the neural network and / or the eye tracking system does not need to specify the calibration parameters. Instead, the calibration parameters are built-in and the designer only needs to specify the number of these parameters. During the training process, the calibration parameters are initialized and then updated together with the parameters of the neural network (e.g., weights between connections in the network) based on minimizing the loss function of the neural network. In this way, the parameters of the neural network are also updated during training based on the calibration parameters. During the training of the neural network, Fig.14 Some operations in the process can be combined Fig.13 The operations are implemented and used. Once the training is completed, Fig.14 The remaining operations can be combined with Fig.11 The operation of is implemented and used. In this case, for a given set of calibration images, the user's calibration parameters are set based on the minimization of the loss function. These calibration parameters are then input to the neural network to support 3D gaze prediction.

[0097] although Fig.14 Learning calibration parameters is described in the context of a neural network that predicts gaze information, but embodiments are not limited thereto. In general, Fig.14 The same applies to neural networks that are associated with a system and trained for a task (e.g., predicting a specific outcome), where the correct operation of the system involves calibration. In addition, for the sake of clarity, Fig.14 The exemplary process of is described in conjunction with one training image, one calibration image, and one image. However, the exemplary process is also applicable to multiple training images, multiple calibration images, and multiple images.

[0098] As shown in the figure, Fig.14The exemplary process of begins at operation 1402, where a computer system accesses a training image associated with a person. The training image shows the face and / or (one or both) eyes of the person. In general, the person has an index number "i" and the training image can be indexed by the index number "i" (e.g., the label or metadata of the training image includes the index number "i"). The training image can be obtained from a data storage and retrieved from it as part of the training process. The value "n" is an integer greater than zero specified by the designer of the neural network. In one example, the value "n" is between 2 and 10. In a specific example, the value "n" is 3.

[0099] At operation 1404, the computer system initializes "n" calibration parameters for the person. For example, the values ​​of the calibration parameters are set to zero or some other first value.

[0100] At operation 1406, the computer system inputs the training image and the "n" calibration parameters to the neural network. For example, the training image is input to the associated Res18 CNN of the neural network, and the "n" calibration parameters are input to the cascaded layers associated with the fully connected modules of the neural network, where these modules are responsible for predicting the 2D gaze origin and the 2D gaze direction.

[0101] At operation 1408, the computer system updates the "n" calibration parameters and parameters of the neural network (referred to herein as network parameters and including, for example, weights of connections between nodes of the neural network) based on minimizing a loss function of the neural network. The loss function is minimized based on the training images and the "n" calibration parameters by back propagation. Thus, a first value (e.g., zero) of the "n" calibration parameters is updated to a second value.

[0102] Operations 1402-1408 may be implemented by a computer system as part of training a neural network. Once trained, the neural network may be used by a user of the eye tracking system. When using an eye tracking system, the system may be calibrated by calibrating the neural network specifically for the user.

[0103] At operation 1410, the computer system initializes the "n" calibration parameters for the user. This operation is similar to operation 1404. In one example, the initialization can be - but does not necessarily need to be - to the same first value (e.g., zero). In another example, the initialization is to the updated value (e.g., second value) of the "n" calibration parameters, which is determined at operation 1408.

[0104] At operation 1412, the computer system generates a calibration image of the user. For example, the computer system instructs the user to gaze at a known gaze point and generates a calibration image accordingly. The calibration image shows one eye of the user (or both eyes or face of the user, where applicable) based on image data generated by a camera associated with the eye tracking system.

[0105] At operation 1414, the computer system inputs the calibration image and calibration parameters to the neural network. This operation is similar to operation 1406.

[0106] In operation 1416, the computer system updates the "n" calibration parameters for the user without updating the network parameters set during training. The "n" calibration parameters are updated at least by minimizing the loss function based on the calibration image and the "n" calibration parameters, wherein the minimization does not change the network parameters. Here, the calibration image has known calibration points. The known calibration points are used as the basis truth of the user's gaze in the minimization of the loss function. The computer system repeatedly updates the "n" calibration parameters by back propagation while keeping the network parameters unchanged until the loss function is minimized. Therefore, the value (e.g., the first value) of the initialization of the "n" calibration parameters performed in operation 1410 is updated to a third value. In general, the third value is different from the second value of the "n" calibration parameters determined in operation 1408.

[0107] Operations 1410-1416 may be implemented by a computer system as part of calibrating a neural network. Once calibration is complete, the neural network may be used to generate 3D gaze information for the user.

[0108] At operation 1418, the computer system inputs the image of the user and the updated "n" calibration parameters into the neural network. Based on the additional image data generated by the camera, the image shows one eye of the user (or both eyes or face of the user, if appropriate). This operation is similar to operation 1406.

[0109] At operation 1420, the computer system receives a prediction based on the image and "n" calibration parameters from the neural network. The prediction includes a distance correction, a 2D gaze origin of the user's eyes in the image, and a 2D gaze direction of the user's eyes in the image.

[0110] The disclosure has been described in some detail for purposes of clarity and understanding. It will, however, be understood that certain changes and modifications may be practiced within the scope of the appended claims.

[0111] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the above description of the exemplary embodiments will provide a feasibility description for implementing one or more exemplary embodiments for those skilled in the art. It is to be understood that various changes may be made to the function and configuration of each element without departing from the spirit and scope of the present disclosure as described herein.

[0112] For example, any detail discussed for one embodiment may or may not appear in all contemplated versions of that embodiment. Likewise, any detail discussed for one embodiment may or may not appear in all contemplated versions of other embodiments described herein. Finally, the absence of discussion of any detail about an embodiment herein should implicitly indicate that the detail may or may not appear in any version of any embodiment discussed herein.

[0113] Specific details are given in the above description to provide a thorough understanding of the embodiments. However, those skilled in the art will appreciate that the embodiments can be implemented without these specific details. For example, in order not to obscure the embodiments due to unnecessary details, the circuits, systems, networks, processes and other elements in the present disclosure may be represented as components in the form of block diagrams. In other cases, in order to avoid obscuring the embodiments, known circuits, processes, algorithms, structures and techniques are shown with unnecessary details omitted.

[0114] It should be noted that each embodiment can be described as a process, which is described as a flow chart, a flow chart, a data flow chart, a structure diagram or a block diagram. Although the flow chart may describe the operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. When the operation of the process ends, the process can terminate, but the process can have additional steps that are not discussed and not included in the accompanying drawings. In addition, the operations in any specifically described process are not all required to appear in all embodiments. The process can correspond to a method, a function, a process, a subroutine or a subprogram, etc. When the process corresponds to a function, the end of the process corresponds to returning the function to the calling function or the main function.

[0115] The term "machine-readable medium" includes, but is not limited to, temporary and non-temporary, portable or fixed storage devices, optical storage devices, wireless channels, and various other media capable of storing, containing, or carrying one or more instructions and / or data. A code segment or machine executable instruction may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, class, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0116] In addition, embodiments of the present disclosure may be implemented at least in part manually or automatically. Manual or automatic implementations may be performed, or at least assisted, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks may be stored in a machine-readable medium. One or more processors may perform the necessary tasks.

[0117] As used herein, the phrase "a first thing is based on a second thing" and the like may mean that the first thing is based only on the second thing, or that the first thing is based on the second thing and one or more additional things.

Claims

1. A non-transitory computer-readable storage medium storing instructions which, once executed on a computer system, cause the computer system to perform operations comprising: accessing training images, the training images comprising a first set of training images and a second set of training images, wherein the first set of training images shows a user's eyes associated with a gaze point in a plane of a camera, and wherein the second set of training images shows a user's eyes associated with a gaze point outside of the plane of the camera; and Training a neural network based on the training image, wherein the training comprises: Inputting a first training image into the neural network, wherein the first training image belongs to the first group of training images; Inputting a second training image into the neural network, wherein the second training image belongs to the second set of training images; A loss function of the neural network is minimized based on a distance between a gaze point and a gaze line, wherein the gaze point is associated with one of the first training image or the second training image, and wherein the gaze line is predicted by the neural network for an eye of a user shown in the one of the first training image or the second training image.

2. The non-transitory computer-readable storage medium of claim 1, wherein: Once trained, the neural network predicts, based on the warped image centered at the user's eyes: (i) a distance correction, (ii) a 2D gaze origin of the user's eyes, and (iii) a 2D gaze direction of the user's eyes.

3. The non-transitory computer-readable storage medium of claim 1, wherein: The first training image shows the user's eyes when fixating the fixation point according to a fixation angle, wherein the second training image shows the user's eyes when fixating another fixation point according to the fixation angle.

4. The non-transitory computer-readable storage medium of claim 1, wherein: The first training image and the second training image show the user's eyes when fixating the gaze point at a gaze angle, wherein the first training image corresponds to a first distance between the camera and the user's eyes, and wherein the second training image corresponds to a second distance between the camera and the user's eyes.

5. The non-transitory computer-readable storage medium of claim 1, wherein: The first training image shows the user's eyes when fixating the fixation point according to a fixation angle, wherein the second training image shows the user's eyes when fixating another fixation point according to a different fixation angle.

6. The non-transitory computer-readable storage medium of claim 1, wherein: Training the neural network further comprises: During training of the neural network, parameters of the neural network are updated based on calibration parameters.

7. The non-transitory computer-readable storage medium of claim 6, wherein: Once training is complete, the updated values ​​of the calibration parameters are input to the neural network.

8. A computer system comprising: processor; as well as a memory storing computer-readable instructions that, upon execution by the processor, configure the computer system to perform operations comprising: accessing training images, the training images comprising a first set of training images and a second set of training images, wherein the first set of training images shows a user's eyes associated with a gaze point in a plane of a camera, and wherein the second set of training images shows a user's eyes associated with a gaze point outside of the plane of the camera; and Training a neural network based on the training image, wherein the training comprises: Inputting a first training image into the neural network, wherein the first training image belongs to the first group of training images; Inputting a second training image into the neural network, wherein the second training image belongs to the second set of training images; A loss function of the neural network is minimized based on a distance between a gaze point and a gaze line, wherein the gaze point is associated with one of the first training image or the second training image, and wherein the gaze line is predicted by the neural network for an eye of a user shown in the one of the first training image or the second training image.

9. The computer system according to claim 8, wherein: Once trained, the neural network predicts, based on the warped image centered at the user's eyes: (i) a distance correction, (ii) a 2D gaze origin of the user's eyes, and (iii) a 2D gaze direction of the user's eyes.

10. The computer system according to claim 9, wherein: The operations further include, upon completion of the training: inputting the warped image into the neural network based on determining an estimated distance between a camera of an eye tracking system and an eye of a user, the warped image generated based on a two-dimensional (2D) image generated by the camera and showing the eye of the user; generating a corrected distance between the user's eye and the camera by at least updating the estimated distance based on the distance correction; as well as 3D gaze information of the user's eyes is generated based on the 2D gaze origin, the 2D gaze direction, and the correction distance.

11. The computer system according to claim 10, wherein: The operations further include: determining the estimated distance between the camera and the user's eyes based on the 2D image; and The position of the user's eyes in the 3D space is estimated based on the corrected distance and the position of the camera in the 3D space, wherein the 3D gaze information includes the position of the user's eyes in the 3D space.

12. The computer system according to claim 11, wherein: The operations further include: A 3D gaze direction starting from the position of the user's eyes in the 3D space is estimated based on the 2D gaze origin and the 2D gaze direction, wherein the 3D gaze information includes the 3D gaze direction.

13. The computer system according to claim 11, wherein: Generating the distorted image includes normalizing the 2D image to generate a normalized image, and cropping the normalized image around the user's eye to generate the distorted image.

14. The computer system of claim 11, wherein the 2D image further shows a second user eye and a user face, and wherein, The operations further include: generating, by the computer system, a second distorted image around the second user's eye from the 2D image; generating a third warped image around the user's face from the 2D image; and The second warped image and the third warped image are input to the neural network, wherein the neural network predicts the distance correction further based on the second warped image and the third warped image.

15. The computer system of claim 14, wherein: The operations further include: generating a second position of the second user's eye in the 3D space and a second gaze direction starting from the second position in the 3D space based on the corrected distance, The 3D gaze information includes the second position and the second gaze direction.

16. The computer system according to claim 8, wherein: The training further comprises: inputting calibration parameters into the neural network; and The network parameters of the neural network and the calibration parameters are updated based on the loss function.

17. The computer system of claim 16, wherein the operations further comprise, upon completion of training: A calibration image and calibration parameters are input to the neural network, the calibration image showing the user's eyes based on image data produced by a camera associated with an eye tracking system of the user, wherein The image data is generated during calibration of the eye tracking system for the user; and The calibration parameters are updated without updating the network parameters of the neural network, wherein the calibration parameters are updated at least by minimizing the loss function based on the calibration image and the calibration parameters.

18. The computer system of claim 17, wherein: The operations further include: The computer system generates three-dimensional (3D) gaze information of the user, wherein the three-dimensional (3D) gaze information is generated by at least the following steps: inputting an image and calibration parameters to the neural network, the image showing the user's eyes based on additional image data generated by the camera, and Receiving a prediction based on the image and the calibration parameters from the neural network, wherein the prediction includes a distance correction, a 2D gaze origin of the eye of the user in the image, and a 2D gaze direction of the eye of the user in the image.

19. A computer-implemented method comprising: accessing, by a computer system, training images, the training images comprising a first set of training images and a second set of training images, wherein the first set of training images shows a user's eyes associated with a gaze point in a plane of a camera, and wherein the second set of training images shows a user's eyes associated with a gaze point outside of the plane of the camera; and The computer system trains a neural network based on the training images, wherein the training comprises: Inputting a first training image into the neural network, wherein the first training image belongs to the first group of training images; Inputting a second training image into the neural network, wherein the second training image belongs to the second set of training images; A loss function of the neural network is minimized based on a distance between a gaze point and a gaze line, wherein the gaze point is associated with one of the first training image or the second training image, and wherein the gaze line is predicted by the neural network for an eye of a user shown in the one of the first training image or the second training image.

20. The computer-implemented method of claim 19, wherein: Once trained, the neural network predicts, based on the warped image centered at the user's eyes: (i) a distance correction, (ii) a 2D gaze origin of the user's eyes, and (iii) a 2D gaze direction of the user's eyes.

Images