Feature-free eye-tracking methods under ecologically valid conditions

Abstract

A system and method for eye gaze tracking is disclosed, comprising the steps of receiving a video of a user captured by a forward-facing camera of a device having a screen, receiving dimensions of the screen, parsing the video into a series of uniformly sized video frame images, inputting the series of uniformly sized video frame images into a pre-trained artificial neural network to extract a plurality of features from the series of uniformly sized video frame images and determining a set of internal spatial hierarchical features on each of the uniformly sized video frame images, inputting each set of hierarchical features into a fully connected layer, which generates intermediate physical estimates in centimeters of the user's gaze position on the screen relative to the device camera, determining a set of screen positions based on the intermediate physical estimates and the dimensions of the screen, and labeling the gaze positions on the screen.

Description

[Technical Field]

[0001] Related Applications This application claims the benefit of priority to U.S. Provisional Application No. 63 / 427,300, filed November 22, 2022, which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to tracking a subject's gaze, and more particularly to a system and method for tracking a tablet user's gaze by analyzing video captured from the tablet's forward-facing video camera via a computer vision-based deep learning model. [Background technology]

[0003] Gaze tracking can be achieved through facial feature extraction from a front-facing camera. In an exemplary approach, a machine learning algorithm is used to extract facial features from video frames and then estimate the user's gaze target using a deep learning model that analyzes the extracted facial features. One major drawback of this approach is that facial feature extraction methods are often prone to failure when complex artifacts are present in the captured video. Complex artifacts include face masks or occlusions. Other issues that arise during facial feature extraction include low-lighting video conditions, varying distances between the subject and the camera, background motion, and other common video impairments. Such impairments can lead to partial or complete loss of gaze tracking capabilities.

[0004] Therefore, there is a need for a method of gaze tracking that does not rely on facial feature extraction. Instead of relying on facial features, the disclosed embodiments provide a method of passing raw video footage to a complex deep convolutional neural network, which can inherently extract important features from video frames even in the presence of the aforementioned artifacts and problems, and propagate these important features throughout the network to result in an output representing the gaze target position for each video frame. Furthermore, for network training purposes, a custom ecologically valid dataset containing videos of users wearing face masks and / or standing in various lighting conditions and distances from the camera can be used to improve prediction capabilities in harsh conditions. This custom dataset results in the generation of gaze predictions for the entire given captured video, which is crucial in applications requiring continuous, uninterrupted gaze tracking.

[0005] The foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the present disclosure. The description of the background art provided herein is intended to generally provide a context for the present disclosure. Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application and is not admitted by inclusion in this section to be prior art or an indication of prior art. Summary of the Invention [Problem to be solved by the invention]

[0006] According to certain aspects of the present disclosure, a system and method for tracking a user's gaze position on a screen is disclosed. [Means for solving the problem]

[0007] In one embodiment, a method includes receiving a video of a user captured by a forward-facing camera of a device having a screen; receiving dimensions of the screen; parsing the video into a series of uniformly sized video frame images; inputting the series of uniformly sized video frame images into a pre-trained artificial neural network, thereby extracting a plurality of features from the series of uniformly sized video frame images, and determining a set of internal spatial hierarchical features on each of the uniformly sized video frame images; inputting each set of hierarchical features to a fully connected layer, which generates intermediate physical estimates of the user's gaze positions on the screen relative to the device camera; determining a series of screen positions based on the intermediate physical estimates and the dimensions of the screen; and labeling the gaze positions on the screen.

[0008] In another embodiment, a device comprises a forward-facing camera, a screen, at least one memory storing instructions, and at least one processor configured to execute the instructions to perform operations including receiving a video of a user captured by a front-facing camera of a device having a screen; receiving dimensions of the screen; parsing the video into a series of uniformly sized video frame images; inputting the series of uniformly sized video frame images into a pre-trained artificial neural network thereby extracting a plurality of features from the series of uniformly sized video frame images to determine a set of internal spatial hierarchical features on each of the uniformly sized video frame images; inputting each set of hierarchical features to a fully connected layer, which generates intermediate physical estimates of the user's gaze position on the screen relative to the device camera; determining a series of screen positions based on the intermediate physical estimates and the dimensions of the screen; and labeling the gaze positions on the screen.

[0009] In an alternative embodiment, a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method comprising: receiving a video of a user captured by a forward-facing camera of a device having a screen; receiving dimensions of the screen; parsing the video into a series of uniformly sized video frame images; inputting the series of uniformly sized video frame images into a pre-trained artificial neural network, thereby extracting a plurality of features from the series of uniformly sized video frame images, to determine a set of internal spatial hierarchical features on each of the uniformly sized video frame images; inputting each set of hierarchical features to a fully connected layer, which generates intermediate physical estimates of the user's gaze position on the screen relative to the device camera; determining a set of screen positions based on the intermediate physical estimates and the dimensions of the screen; and labeling the gaze positions on the screen.

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a flowchart illustrating an example method for tracking gaze position according to the techniques disclosed herein.

[0012] [Figure 2] FIG. 1 is a flowchart illustrating an example method for training a neural network to predict gaze position according to the techniques disclosed herein.

[0013] [Figure 3] FIG. 1 is a workflow diagram illustrating an exemplary process of eye-gaze tracking according to the techniques disclosed herein.

[0014] [Figure 4]FIG. 2 is a diagram of a computing node according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0016] The systems, devices, and methods disclosed herein are described in detail, by way of example, with reference to the Figures. The examples discussed herein are merely examples and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be construed as essential to any particular implementation of these devices, systems, or methods, unless specifically designated as essential.

[0017] Also, in any method described, whether the method is described in conjunction with a flow chart or not, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in the execution of the method does not imply that the steps must be performed in the order presented, but may instead be performed in a different order or in parallel.

[0018] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." Furthermore, the words "a" and "an" do not denote a limitation of quantity herein, but rather denote the presence of one or more of the referenced item.

[0019] Alternative methods for tracking gaze with a tablet's forward-facing camera generally rely on facial feature extraction and landmark tracking as key components of the tracking architecture. This approach can be used to reduce computational requirements and maximize model performance because it separates the tablet user's face and eyes into individual images that the model can analyze and understand with minimal external conditions, such as background noise and irrelevant image components. A major drawback of this methodology is the loss of video frames that the feature extractor model fails to process, an unacceptable trade-off for many industrial applications. Furthermore, facial feature extraction methods trained before the COVID pandemic cannot process images of individuals wearing masks because large portions of the face are occluded, making these methods unsuitable for use in various environments where mask use is more common. The embodiments disclosed below preserve the full content of the original video recording and forgo the feature extraction step at the expense of increased computational requirements and data complexity in order to enhance ecological validity and compatibility with various environmental conditions, which are crucial for many industrial applications, including cognitive screening.

[0020] One key difference between the alternative method and the method described herein is the degree of preprocessing required before passing input images to a model for training or inference purposes. The alternative method relies on a preprocessing stage that applies a facial feature detection algorithm to generate a cropped image containing the user's eyes and / or face. The cropped image is then used as input features to a gaze position predictive model. In embodiments of the present disclosure, preprocessing involves only image resizing; therefore, the method is "feature-free" in that it does not require a preprocessing step to generate features from the data, but instead leverages the inherent ability of convolutional neural networks (CNNs) to algorithmically learn the spatial hierarchy of input image data without human supervision through model training. These internally derived features are often higher-dimensional and much more complex than previous methods (i.e., manually cropped facial features), allowing for greater emphasis on parts of the image that previous methods deemphasize, such as the location of the face in the input image, when inferring the user's gaze target.

[0021] The nature of the featureless input data used and the method for extracting it are the spatial relationships between different components of the raw image. These spatial relationships are represented as numerical arrays and learned internally within the CNN. Through model training, the CNN learns which features are important for each individual image based on the target output variable (i.e., ground truth user gaze position). The internal features most relevant to a particular prediction on a particular image may differ from those used to make predictions on other images characterizing different users, body poses, environmental lighting conditions, distance from the camera, and / or mask / eyeglass use, thereby increasing the flexibility and compatibility of the model. These features differ from the model-based facial features used in previous methods because they are not fixed crops of the raw image but instead a hierarchical numerical representation of the relevance of different image components to gaze target predictions.

[0022] Therefore, the method described below includes a neural network architecture that can process video frames directly without facial feature extraction. This enables the analysis of complex video recordings containing challenging artifacts, such as face masks, low ambient lighting, long distances from the camera, and background motion, which are difficult for traditional facial feature extraction methods to handle. This is achieved by training the neural network on a custom training video dataset. The dataset is designed to maximize the model's predictive power by collecting a diverse set of data points that includes a variety of participant demographics, environmental conditions, eyeglass and face mask use, and user distance from the camera.

[0023] 1 is a flowchart illustrating an example method 100 for eye tracking using a machine learning model, according to an example embodiment of the present disclosure. For example, the example method 100 (e.g., steps 102-112) may be performed automatically by a processor or in response to a request by a user.

[0024] An exemplary method 100 for eye tracking can include one or more of the following steps: In step 102, the method can include receiving a video. The video is captured by a front-facing camera on a device, such as a tablet, and a user is gazing at the device, the user being a distance from the front-facing camera on the device. The distance between the user and the camera can vary between videos in the plurality of videos. Dimensions of each source tablet or device of the plurality of videos are also received, the dimensions including information about the size, height, and width of the source tablet's screen, and the user's gaze in each video is directed toward the screen, and in particular, toward an object on the screen.

[0025] While method 100 is described using a tablet device, it is contemplated that the video may be captured by a forward-facing camera on any suitable device, including a mobile phone, laptop computer, or wearable device. The user's gaze is directed toward a point on the device screen. The device may be any available computing device, and the distance between the camera and the user may vary widely based on the user. It is contemplated that the multiple videos may feature a wide range of subject matter, covering different user demographics and situations. Lighting and other environmental conditions may also vary widely within the received multiple videos. It is also contemplated that the user may be wearing a mask, glasses, or other face covering. The multiple videos may cover a wide range of distances to the user, user dimensions, and environmental conditions within each of the multiple videos.

[0026] At step 104, the method may include preparing each video of the plurality of videos for processing by the neural network. Each received video is parsed into a series of video frame images, and each image undergoes a rescaling transformation to arrive at a uniform format with a standard height and width that the network has been trained to accept. The uniform format ensures that each image has the same dimensions and file format; for example, each image may be a JPEG file with a size of 480 x 480 pixels. The images are chronologically sorted based on the order in which they appear in the video upon reception. Preparing the files for processing eliminates the need to adjust images for different environmental conditions or situations, including distance, and instead allows for a wide variety of conditions to exist within the video. Images are not cropped or resized in a way that distorts the images; rather, the images are converted to standard dimensions to create uniformity between images.

[0027] Additionally, each image is assigned a label corresponding to the approximate location of the user's gaze during each video frame relative to the tablet's camera position. Label coordinates are associated with the image using high-precision timing information collected by the video application using the front-facing camera, allowing for accurate assignment of gaze target locations to each individual video frame without additional computation. The approximate location can be defined by two pixel measurements representing horizontal and vertical displacement from the top-left corner of the device screen. The screen location is used to define a ground-truth gaze label for each frame. Pairs of video frames and corresponding gaze locations constitute the input to the deep learning model for training and evaluation purposes.

[0028] In step 106, the method may include inputting a series of uniform video frames into a neural network, the neural network having been trained on a custom dataset. Features are extracted from the series of uniform video frame images to determine at least one pattern among the features, and each extracted feature is associated with a labeled approximate gaze position. During this "inference phase," the trained neural network is given preprocessed video frames from an unseen video file and tasked with estimating the gaze position for each frame.

[0029] The neural network can be a convolutional neural network (CNN), another neural network variant, or any suitable machine learning model or trained algorithm. When the neural network is a CNN, the inherent ability of CNNs to algorithmically learn the spatial hierarchy of input image data without human supervision is used to process the images. These internally derived features from a sequence of images are often higher-dimensional and much more complex than features extracted by traditional methods of facial feature extraction. These features allow for greater emphasis on parts of the image not assessed by previous methods, such as the location of the face within the input image, when predicting a user's gaze target. Generally, these features are spatial relationships represented as numerical arrays between different components of the raw image. Through model training, the neural network model learns which features (or image components) may be important for each individual image based on the target output variable. These features differ from the model-based facial features used in previous methods because the features are not fixed crops of the raw image but instead are hierarchical numerical or condensed data representations of the relevance of different image components to gaze target prediction.

[0030] During model inference (step 106), the trained neural network is shown previously unseen video frame images and tasked with generating gaze position estimates or predictions for each image based on the neural network weights learned during training. The trained neural network determines one or more patterns from the data representation of input image features to generate gaze position. The patterns are established across a series of images from the same video and can include details about the user's proportions and distance from the tablet. Because the model is fed entire raw video frame images that include the user's head as well as parts of the user's body and surroundings, these factors are automatically taken into account when analyzing spatial dependencies between image components. For example, during training, the model learns that two users viewing the same part of the screen but from different distances from the camera may have different head-to-body proportions, yet for purposes of predicting gaze, the two users are equivalent. Thus, in this situation, the model generates similar estimates of gaze position.

[0031] At step 108, the method may include passing the labeled images from the neural network to a fully connected layer. The fully connected layer generates an intermediate estimate of the user's gaze position based on learning from the trained neural network. The intermediate estimate includes a set of coordinates representing the distance between the user's gaze target on the screen and the device camera. The distance is represented as two numbers representing the horizontal and vertical components of the gaze position on the screen relative to the camera in centimeters.

[0032] At step 110, the method can include determining a set of screen pixels corresponding to on-screen locations from the predicted physical gaze positions in centimeters, and adjusting the pixel locations based on a set of dimensions for each source tablet. Specifically, the generated set of gaze predictions from the trained neural network is passed through a post-processing pipeline that performs calculations based on the unique dimensions of each device to convert the generated physical gaze estimates into a set of screen pixel locations corresponding to on-screen objects observed by the user in every video frame.

[0033] In step 112, the method may include outputting a labeled set of coordinates indicating gaze position on the tablet screen. The output screen may be used to predict a user's gaze position through a brain health assessment (e.g., a cognitive or behavioral assessment) to assist administrators in diagnosing brain health disorders such as cognitive dysfunction, depression and anxiety, or neuromotor disorders. This method of eye tracking may be able to correlate gaze position and predetermined regions of interest to predict a user's brain health status. As more data is collected, eye tracking may be added to existing and new assessments to analyze the user's gaze path and potentially extract meaningful insights into the user's cognitive, behavioral, and / or neuromotor health through correlation with external assessment scores and biomarkers.

[0034] 2 is a flowchart illustrating an example method 200 for training a neural network, according to an example embodiment of the present disclosure. For example, the example method 200 (e.g., steps 202-212) may be performed automatically by a processor or in response to a request by a user.

[0035] At step 202, the method may include receiving a dataset of training videos, the training videos including various user environmental conditions and user demographics. Each video is captured using a front-facing camera on a tablet device. This dataset of videos is designed to maximize the predictive ability of the neural network model by collecting a diverse set of data points, including various participant demographics, environmental conditions, use of eyeglasses and face masks, and user distance from the camera. By using a larger training dataset that includes a wider range of conditions, for example, for complex input images featuring face masks, long distances from the camera, and / or dim lighting, the error rate is reduced by introducing these conditions collectively in the training process. Combining all images featuring different environmental conditions into one dataset results in high user and environmental variability, which is crucial for supervised deep learning applications and for achieving high accuracy during real-world deployments.

[0036] In one embodiment, the dataset contains video data captured by the device's forward-facing camera showing the faces of 962 human participants engaged with 21 different devices. The participants represent a wide range of demographics, and several participants wear glasses, face masks, or other items on their face or head to increase the variability and range of the dataset. The devices also introduce a range of dimensions to consider when determining gaze position.

[0037] At step 204, the method may include receiving additional information about each video in the dataset, including a target gaze position for each video. The target gaze position may include labeled pixels on the screen that indicate where the user was looking. The additional information received may include device dimensions and timing information for when each video started and ended.

[0038] Gaze targets can be obtained from an application running on a tablet, in which the user is asked to look at a specific portion of the tablet screen for a set period of time and then click on either the left or right side of the screen after the period has expired, based on the brief appearance of the letters "L" or "R" on the screen. This task may be repeated multiple times, with each trial requiring the user to look at a different, randomly generated portion of the screen. The user is recorded throughout the process, including detailed spatial and temporal information of the screen location shown to the user. This allows video footage captured for specific screen locations to be associated with the user during analysis.

[0039] Using information from the gaze target acquisition process, each video can be parsed into individual frames, with each frame associated with a specific gaze target position on the screen, in step 206. The gaze targets, along with device metadata, are used to train a neural network.

[0040] At step 208, the method may include generating a trained neural network model by processing the dataset to predict gaze positions for each video frame image. The neural network may be a CNN or other suitable learning model. During training, the neural network extracts features for each input image in the form of numerical spatial hierarchical dependencies, identifies patterns among the extracted features from different images, and associates the features with input target gaze positions for each video.

[0041] Effects resulting from different distances from the screen to the user are implicitly taken into account by the model's training. Because the model is fed the entire raw image, which includes not only the user's head but also parts of the user's body and surroundings, these elements are automatically taken into account by the trained neural network when analyzing spatial dependencies between image components. Thus, the model does not require explicit measurement of the distance between the camera and the person or require that the distance between the camera and the person be maintained. During training, the model learns that two users looking at the same part of the screen but at different distances from the camera may have different head-to-body ratios, but are equivalent for the purposes of predicting gaze.

[0042] At step 210, the method may include iteratively reducing the difference between the predicted gaze position and the target gaze position until a stable set of neural network weights is determined, which are then used as parameters in a trained neural network to output a stable and accurate predicted gaze position.

[0043] At step 212, the method may include outputting the trained neural network model. The trained neural network model may be configured to provide outputs into a fully connected layer so that additional computations and adjustments to the output estimates may be performed.

[0044] FIG. 3 is a workflow illustrating an exemplary model 300 used for eye tracking according to the techniques disclosed herein. For example, the exemplary model 300 may be executed automatically by a processor or in response to a request by a user. The model may include a pre-processing pipeline 302 into which an input video enters having X and Y coordinate labels indicating a location on a video screen in pixels. The location indicates a target gaze of a target user in the input video. In one embodiment, the input video may have pixel dimensions of 480×640. This dimension may be a result of the size of the source camera. The pre-processing pipeline 302 parses the input video into a series of video frame images and converts each image to a standard dimension for processing by the neural network 304. For example, the standard image pixel dimensions of the series of video frame images may be 480×480. The pre-processing pipeline 302 also outputs a label, the X and Y distance in centimeters, between the gaze location on the screen and the device camera.

[0045] The output sequence of uniform images is then input to a neural network 304, e.g., a deep CNN. The neural network 304 is given preprocessed video frames from an unseen video file, as described above in step 106 of exemplary method 100, and is tasked with estimating the gaze position for each frame. The neural network 304 then outputs the predicted gaze positions to a fully connected layer 306. The fully connected layer 306 generates an intermediate estimate, in centimeters, of the user's gaze position on the screen relative to the device camera. This intermediate estimate may be based on previously input distance coordinates associated with the input video.

[0046] In the final post-processing pipeline 308, the estimates from the fully connected layer 306 are adjusted to reflect the dimensions of the tablet or device. Specifically, the screen dimensions are used to accurately convert the predicted gaze position from centimeters relative to the device camera to pixels on the tablet's screen. The conversion is achieved by shifting the gaze prediction reference point from the device camera to the top left corner of the device screen, and then using knowledge of the screen's specific pixel density to estimate the gaze prediction pixel position from its physical counterpart.

[0047] 4 is a schematic diagram of an exemplary computing node. Computing node 10 is merely one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments described herein. Nevertheless, computing node 10 may implement and / or perform any of the functions described above.

[0048] Computing node 10 includes computer system / server 12, which is usable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed computing environments that include any of the above systems or devices.

[0049] Computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system / server 12 may be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0050] 4, computer system / server 12 within computing node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 coupling various system components, including system memory 28, to processor 16.

[0051] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe), and an Advanced Microcontroller Bus Architecture (AMBA).

[0052] Computer system / server 12 typically includes a variety of computer system-readable media, which may be any available media that can be accessed by computer system / server 12, including both volatile and nonvolatile media, removable and non-removable media.

[0053] The system memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The algorithm computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 18 by one or more data media interfaces. As further shown and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present disclosure.

[0054] A program / utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, as well as an operating system, one or more application programs, other program modules, and program data, by way of example and not limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The program modules 42 generally perform the functions and / or methodologies of the embodiments described herein.

[0055] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, one or more devices that allow a user to interact with the computer system / server 12, and / or any device (e.g., network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computing devices. Such communication may occur via an input / output (I / O) interface 22. Additionally, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. Although not shown, it should be understood that other hardware and / or software components may be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0056] In various embodiments, a learning system is provided. In some embodiments, a feature vector is provided to the learning system. Based on the input features, the learning system generates one or more outputs. In some embodiments, the output of the learning system is a feature vector. In some embodiments, the learning system comprises an SVM. In other embodiments, the learning system comprises an artificial neural network. In some embodiments, the learning system is pre-trained using training data. In some embodiments, the training data is retrospective data. In some embodiments, the retrospective data is stored in a data store. In some embodiments, the learning system may be further trained by manual curation of previously generated outputs.

[0057] In some embodiments, the learning system is a trained classifier. In some embodiments, the trained classifier is a random decision forest. However, it will be appreciated that a variety of other classifiers are suitable for use with the present disclosure, including linear classifiers, support vector machines (SVMs), or neural networks such as recurrent neural networks (RNNs).

[0058] Suitable artificial neural networks include, but are not limited to, feedforward neural networks, radial basis function networks, self-organizing maps, learning vector quantization, recurrent neural networks, Hopfield networks, Boltzmann machines, echo state networks, long short-term memories, bidirectional recurrent neural networks, hierarchical recurrent neural networks, probabilistic neural networks, modular neural networks, associative neural networks, deep neural networks, deep belief networks, convolutional neural networks, convolutional deep belief networks, large memory storage and retrieval neural networks, deep Boltzmann machines, deep stacking networks, tensor deep stacking networks, spike and slab restricted Boltzmann machines, composite hierarchical deep models, deep coding networks, multi-layer kernel machines, or deep Q-networks.

[0059] The present disclosure may be embodied as a system, method, and / or computer program product, which may include computer-readable storage medium(s) having computer-readable program instructions for causing a processor to perform aspects of the present disclosure.

[0060] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge structures in grooves on which instructions are recorded, and any suitable combination of the above. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within the computing / processing device.

[0062] The computer-readable program instructions for carrying out the operations of the present disclosure can be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present disclosure.

[0063] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0064] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executing on the processor of the computer or other programmable data processing apparatus create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture containing instructions that perform aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0065] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to generate a computer-implemented process into a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0066] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

Claims

1. receiving a video of a user captured by a front-facing camera of a device having a screen; receiving the dimensions of the screen; parsing the video into a series of uniformly sized video frame images; inputting the series of uniform-sized video frame images into a pre-trained artificial neural network, thereby extracting a plurality of features from the series of uniform-sized video frame images to determine a set of internal spatial hierarchical features on each of the uniform-sized video frame images; inputting each set of hierarchical features into a fully connected layer, which generates an intermediate physical estimate of the user's gaze position on the screen relative to the device camera; determining a set of screen positions based on the intermediate physical estimates and the dimensions of the screen; labeling the gaze position on the screen; A method comprising:

2. the artificial neural network receiving a dataset of training videos, the training videos showing a variety of user environmental conditions, facial coverings, and user demographics; receiving a ground truth gaze position for each training video in the dataset; parsing each training video into a sequence of video frame images, each video frame image including the ground truth gaze position; training the artificial neural network by processing the data set to predict gaze positions for each video frame image; iteratively reducing the difference between the predicted gaze position and the ground truth gaze position until a stable set of neural network weights is determined; The method of claim 1 , wherein the trained trainee is pre-trained by

3. The method of claim 1 , wherein the uniform dimension video frame images have the same pixel dimensions and are the same file type.

4. The method of claim 1 , wherein the approximate location of the user's gaze is determined by the user tracking one or more objects across the device screen.

5. determining at least one pattern between the extracted features; Associating each extracted feature with a labeled predicted gaze position on each image; The method of claim 1 further comprising:

6. The method of claim 5 , wherein a pattern is determined across the series of uniform video frame images.

7. The method of claim 1 , wherein the extracted features comprise spatial relationships of components of the sequence of video frame images, represented as a numerical array.

8. The method of claim 7 , wherein the extracted features are ordered based on their relevance to the gaze target prediction.

9. The method of claim 9 , wherein image-specific features have higher relevance.

10. The method of claim 1 , wherein the artificial neural network is a convolutional neural network.

11. The method of claim 2 , wherein the different user environmental conditions include a plurality of ambient lighting settings.

12. The method of claim 2 , wherein the training video has a range of distances between the user and the camera.

13. The method of claim 1 , wherein the device is a tablet.

14. Forward-facing camera, screen, at least one memory for storing instructions; and Execute the instructions, receiving a video of a user captured by a front-facing camera of a device having a screen; receiving the dimensions of the screen; parsing the video into a series of uniformly sized video frame images; inputting the series of uniform-sized video frame images into a pre-trained artificial neural network, thereby extracting a plurality of features from the series of uniform-sized video frame images to determine a set of internal spatial hierarchical features on each of the uniform-sized video frame images; inputting each set of hierarchical features into a fully connected layer, which generates an intermediate physical estimate of the user's gaze position on the screen relative to the device camera; determining a set of image locations based on the intermediate physical estimate and the dimensions of the screen; labeling the gaze position on the screen; at least one processor configured to perform operations including A device comprising:

15. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 13.

Images