Eye tracking method and eye tracking sensor

Abstract

An eye tracking method is disclosed herein, comprising the steps of: - reading time-series data (41) from a self-mixing interferometry sensor (4); - using the time-series data (31) as an input (11) of a first machine learning model (1), wherein the first machine learning model (1) is configured for feature extraction and wherein an output (12) of the first machine learning model (1) comprises a latent feature space (13); - using a temporally stacked output (12) of the first machine learning model (1) as an input (21) of a second machine learning model (2), wherein the second machine learning model (2) is configured for feature analysis and wherein an output (22) of the second machine learning model (2) comprises predicted gaze data (23). Further, an eye tracking sensor (10) is disclosed herein.

Description

[0001] Description

[0002] EYE TRACKING METHOD AND EYE TRACKING SENSOR

[0003] An eye tracking method and an eye tracking sensor are speci fied herein .

[0004] At least one obj ect of certain embodiments is to speci fy an eye tracking method with an improved performance . A further obj ect of certain embodiments is to speci fy an eye tracking sensor with an improved performance .

[0005] According to at least one aspect , the eye tracking method comprises a step of reading time-series data from a sel fmixing interferometry sensor . For example , the sel f-mixing interferometry sensor comprises or consists of a laser, such as a laser diode , wherein the laser comprises a laser active medium arranged inside an optical resonator . In particular, the laser emits electromagnetic laser radiation during operation . The electromagnetic laser radiation is in the infrared spectral range , for example .

[0006] For example , during sel f-mixing interferometry at least a part of the electromagnetic laser radiation that is emitted by the laser during operation is at least partially reflected back to the laser by an external obj ect . Speci fically, the external obj ect is an eye , whose position, gaze direction, gaze traj ectory, gaze speed, and / or orientation is to be tracked, for example . In particular, the reflected electromagnetic radiation is at least partially coupled back into the optical resonator of the laser . For example , the reflected electromagnetic laser radiation undergoes sel fmixing interference inside the optical resonator, i . e . the reflected electromagnetic laser radiation interferes with the electromagnetic laser radiation that is generated by the laser active medium inside the optical resonator . For example , an amplitude , a phase and / or a frequency of the emitted electromagnetic radiation is altered due to the sel fmixing interference inside the optical resonator . For example , the altered amplitude , phase , and / or frequency of the emitted electromagnetic laser radiation can be detected and comprises information about the external obj ect , such as distance information, position information, movement information, and / or orientation information .

[0007] For example , the sel f-mixing interferometry sensor provides an electrical signal that comprises the time-series data . For example , the time-series data comprises information about properties of the external obj ect as a function of time . In particular, the time-series data comprises data at a series of discrete time steps , and / or it comprises data that is continuous as a function of time . For example , the timeseries data is digital or analog . Analog time series data can be converted to digital time series data using an analogdigital converter before further processing, for example .

[0008] For example , the time-series data comprises or consists of sel f-mixing interferometry sensor data points of at least 50 time steps and of at most 50000 time steps . For example , a time interval between time steps is between 0 . 1 microseconds and 100 microseconds , inclusive . For example , adj acent time steps are equidistant in time .

[0009] According to at least one further aspect , the eye tracking method comprises a step of using the time-series data as an input of a first machine learning model , wherein the first machine learning model is configured for feature extraction and wherein an output of the first machine learning model comprises or consists of a latent feature space .

[0010] In particular, the first machine learning model is a mathematical model , such as an arti ficial neural network, that is designed to learn features from the time-series data that are important for predicting gaze data, such as the gaze direction and / or the gaze speed, of the tracked eye during a training period of the machine learning model . During eye tracking, the first machine learning model outputs the learned features for previously unseen time-series data, for example .

[0011] Here and in the following " feature extraction" refers to a process where the raw input data of the first machine learning model is selected and / or trans formed into a more manageable and informative representation for eye tracking . In particular, the time-series data is selected and / or trans formed by the first machine learning model such that relevant information or attributes , i . e . the features , for eye tracking are preserved . For example , the process of feature extraction compresses the time-series data such that its important features for eye tracking are embedded in the latent feature space .

[0012] Here and in the following, the " latent feature space" is an abstract , high-dimensional representation of the time-series data that captures underlying, hidden patterns or characteristics that are relevant for eye tracking . For example , the latent feature space is created by dimensionality reduction, supervised machine learning and / or unsupervised machine learning of the first machine learning model . In particular, the latent feature space of the first machine learning model encodes important features or relationships of the time-series data that are relevant for eye tracking . These features may not directly correspond to observable or explicitly defined attributes of the tracked eye or of the time-series data . For example , the latent features space is learned during the training period . For example , during the training period a loss function is minimi zed, wherein the loss function represents a deviation between the predicted gaze data and an actual gaze data corresponding to the time-series data used for training the first machine learning model .

[0013] For example , the latent feature space comprises or consists of at least 2 and at most 5000 features , or the latent feature space comprises or consists of between 20 and 200 features , inclusive . In other words , the latent feature space has a dimension of at least 2 and at most 5000 , or it has a dimension between 20 and 200 , inclusive , for example .

[0014] According to at least one further aspect , the eye tracking method comprises a step of using a temporally stacked output of the first machine learning model as an input of a second machine learning model , wherein the second machine learning model is configured for feature analysis and wherein an output of the second machine learning model comprises or consists of predicted gaze data .

[0015] For example , the temporally stacked output of the first machine learning model comprises or consists of the extracted features , i . e . the latent feature space , of the first machine learning model concatenated over a past time window . For example , the latent feature space of the first machine learning model at a series of subsequent time steps is concatenated and forms the input of the second machine learning model . For example , the past time window comprises at least 10 and at most 1000 past time points at which the first machine learning model has generated an output . For example , the past time window has a fixed length in time and spans a time interval of at least 5 milliseconds and at most 5000 milliseconds up to the current time . For example , the past time window is a sliding time window, wherein the last time point of the past time window corresponds to the most recent time point at which the first machine learning model produced an output .

[0016] Here and in the following, " feature analysis" refers to a process where the second machine learning model predicts gaze data based on the output , i . e . the latent feature space , of the first machine learning model during the past time window .

[0017] In particular, the second machine learning model is a mathematical model , such as an arti ficial neural network, that is trained to analyze the features in the latent feature space generated by the first machine learning model in order to predict gaze data, such as the gaze direction, gaze position, gaze traj ectory and / or the gaze speed, of the tracked eye with high accuracy .

[0018] For example , the gaze data comprises or consists of the gaze direction, the gaze position, the gaze traj ectory, the gaze speed, and / or an eye-ball rotation, in particular a vertical and a hori zontal eye-ball rotation . In particular, the gaze direction or gaze position is a two-dimensional coordinate that speci fies where the tracked eye looks in the vertical , i . e . up-down, and hori zontal , i . e . left-right , directions .

[0019] For example , the gaze traj ectory corresponds to the gaze direction or gaze position as a function of time . For example , the gaze speed corresponds to an Euclidean displacement over time of the gaze position or gaze direction . Here and in the following, gaze position and gaze direction are used synonymously .

[0020] For example , the second machine learning model has a smaller number of parameters compared to the first machine learning model . For example , the number of parameters of the first machine learning model is larger than the number of parameters of the second machine learning model by a factor of at least 2 , or by a factor of at least 10 , or by a factor of at least 25 .

[0021] According to an embodiment , the eye tracking method comprises the steps of :

[0022] - reading time-series data from the sel f-mixing interferometry sensor ;

[0023] - using the time-series data as the input of the first machine learning model , wherein the first machine learning model is configured for feature extraction and wherein the output of the first machine learning model comprises the latent feature space ;

[0024] - using the temporally stacked output of the first machine learning model as the input of a second machine learning model , wherein the second machine learning model is configured for feature analysis and wherein the output of the second machine learning model comprises predicted gaze data .

[0025] The method described herein is based on the idea to use two distinct machine learning models to predict gaze data of a tracked eye that is observed by a sel f-mixing interferometry sensor . In particular, the first machine learning model is trained for robust feature extraction from the time-series data provided by the sel f-mixing interferometry sensor . For example , the feature extraction is independent of a precise eye-ball position relative to the sel f-mixing interferometry sensor and / or is independent of a user speci fic eye-ball shape or morphology . By contrast , the second machine learning model is used for feature analysis and can have a smaller number of parameters than the first machine learning model . Accordingly, the second machine learning model can be quickly re-trained i f needed in view of additional calibration parameters , such as the eye-ball position, for example . Moreover, the second machine learning model can exploit temporal dependencies between gaze data predictions , for example , which can reduce an error of the predicted gaze data and allows gaze traj ectory forecasting into the future , for example . Advantageously, by using the two machine learning models for feature extraction and feature analysis , the performance of predicting gaze data declines less quickly compared to a single machine learning model , i f the eye-ball position moves relative to the sel f-mixing interferometry sensor .

[0026] Moreover, by separating the feature extraction and feature analysis using two dedicated machine learning models as described herein, the computational demand for training the machine learning models can be reduced . For example , calibration data, such as user speci fic data or system speci fic data can be ef ficiently integrated into the second machine learning model without a need to re-train the larger first machine learning model . In particular, the training intensive first machine learning model can be pre-trained or pre-compiled, whereas the second machine learning model can be trained on-the- fly during eye tracking, for example . In particular, the first machine learning model is user independent and / or at least to some degree system independent , for example with regard to speci fics of the sel f-mixing interferometry sensor, whereas the second machine learning model can be trained or re-trained using user speci fic data or information . For example , the second machine learning model can be retrained using consumer product standard usage of fixation points or using forced gaze traj ectories . For example , 500 ms of additional time-series data measuring 5 speci fic fixation points in a user' s field of view can improve model performance by an average 10% or up to 200% or more . It is also possible that the second machine learning model is re-trained in an unsupervised manner without active user input . For example , the second machine learning model is retrained by analyzing a data distribution from the sel f-mixing interferometry sensor or a further sensor, such as a camera .

[0027] Further, the computational demand for deploying the first and second machine learning models on a low-cost edge device can be reduced compared to a single machine learning model that combines both functions of feature extraction and feature analysis . In particular, in the latter case feature extraction needs to be performed for every new input , i . e . every time-series data, over the full past time window that is used for temporal analysis , leading to a duplicated data pass-through, increased number of model parameters , and higher computational demand . By separating feature extraction and feature analysis using the first and second machine learning models as described herein, the first machine learning model can focus on a frame-by- frame based feature extraction, wherein one frame corresponds to one instance of the time-series data, whereas the second machine learning model can focus on temporal feature exploitation over multiple frames .

[0028] According to at least one further aspect , the eye tracking method comprises a further step of pre-processing the timeseries data before using it as an input of the first machine learning model . For example , pre-processing the time-series data includes trans forming the time-series data to frequency representation using a Fourier trans formation, a discrete Fourier trans formation, or a discrete Cosine trans formation . Furthermore , pre-processing can include signal normali zation, signal scaling, temporal averaging, averaging of frequency time windows , i . e . the power spectral density, or other techniques for improving a signal-to-noise ratio of the predicted gaze data and / or other techniques for preparing the data for processing with the first machine learning model .

[0029] According to at least one further aspect , the eye tracking method comprises a further step of temporal smoothening of the output of the second machine learning model using a digital filter . In particular, the digital filter improves the signal-to-noise ratio of the predicted gaze data . For example , the raw gaze data predictions obtained from the second machine learning model can be digitally filtered using a moving average , a Gaussian filter, and / or a Savit zky-Golay filter . For example , the digital filter has a temporal window with a length between at least 10 milliseconds and at most 1000 milliseconds , or between at least 50 milliseconds and at most 200 milliseconds .

[0030] According to at least one further aspect of the eye tracking method, the first machine learning model is a feedforward neural network . For example , the feedforward neural network comprises an input layer, one or more hidden layers and an output layer . For example , each layer comprises or consists of a plurality of nodes or neurons for processing the input data by computing a weighted sum over all inputs of each node , respectively . In particular, information or data flows from the input layer to the output layer in one direction, i . e . uni-directionally . In other words , the feedforward neural network does not comprise feedback loops .

[0031] For example , the feedforward neural network is a convolutional neural network designed for feature extraction from the time-series data . For example , the convolutional neural network comprises or consists of residual connection blocks , squeeze-and-excitation blocks , global average pooling layers , dropout layers , batch normali zation layers , and / or fully connected layers .

[0032] It is also possible that the first machine learning model has a trans former architecture and / or comprises an attention layer or an attention mechanism for processing and generating sequences of data in a paralleli zable manner . In particular, the trans former architecture comprises a structure with a series of encoders and decoders .

[0033] According to at least one further aspect of the eye tracking method, the second machine learning model is a recurrent neural network . In particular, the recurrent neural network comprises one or more feedback loops , where the information or data can propagate through the neural network in both directions , i . e . from the input to the output and / or vice versa . In particular, the recurrent neural network can be used for time-series analysis , such as recogni zing patterns in sequential data that has temporal relationships . For example , the recurrent neural network comprises or consists of an input layer, an output layer, at least one long short-term memory ( short : LSTM) layer and / or at least one gated recurrent unit ( short : GRU) layer . Moreover, the recurrent neural network can further comprise fully connected layers . Alternatively, it is also possible that the second machine learning model comprises or consists of a convolutional neural network where the input data comprises or consists of the temporally stacked output of the first machine learning model , i . e . the latent feature space .

[0034] According to at least one further aspect of the eye tracking method, the second machine learning model has a trans former architecture . For example , the second machine learning model comprises an attention layer or an attention mechanism for processing and generating sequences of data in a paralleli zable manner .

[0035] According to at least one further aspect , the eye tracking method comprises a further step of using a further temporally stacked output of the first machine learning model as an input of a third machine learning model , wherein an output of the third machine learning model comprises user speci fic parameters and / or system speci fic parameters . For example , the further temporally stacked output of the first machine learning model comprises or consists of the extracted features from the first machine learning model , i . e . the latent feature space , concatenated over a past time window, as for the temporally stacked output of the first machine learning model described above . In particular, the temporally stacked output described above and the further temporally stacked output described here only di f fer in the length or si ze of the past time window . For example , the past time window of the further temporally stacked output has a fixed length in time and spans a time interval of at least 1 second and at most 500 seconds up to the current time .

[0036] For example , the user speci fic parameters comprise or consist of an eye-ball shape , an eye-ball si ze , an eye-ball curvature , and / or an eye-ball position relative to a position of the sel f-mixing interferometry sensor . For example , the system speci fic parameters comprise or consist of parameters of the sel f-mixing interferometry sensor, such as a wavelength, a power, a bandwidth, and / or a directional characteristic of the electromagnetic radiation emitted by the sel f-mixing interferometry sensor, and / or geometric properties of the arrangement of the sel f-mixing interferometry sensor relative to the tracked eye . Further, the system speci fic parameters can comprise parameters of optical components , such as refractive or di f fractive elements , along an optical path between the sel f-mixing interferometry sensor and the tracked eye .

[0037] According to at least one further aspect of the eye tracking method, the third machine learning model is a recurrent neural network . For example , the third machine learning model has a similar or comparable architecture as the second machine learning model , whereas their number of layers , hidden neurons and / or other layer speci fic characteristics can di f fer . In particular, the second and the third machine learning model di f fer in the structure of their output layers .

[0038] According to at least one further aspect of the eye tracking method, the output of the third machine learning model is used as a further input of the second machine learning model . For example , the user speci fic parameters and / or system speci fic parameters , such as a spatial relationship and / or a distance between the eye-ball and the sel f-mixing interferometry sensor, are used to increase the prediction performance of the second machine learning model .

[0039] According to at least one further aspect of the eye tracking method, the second machine learning model uses calibration data from a further sensor as a further input . For example , the calibration data comprises or consists of the measured eye-ball shape , the eye-ball si ze , the eye-ball curvature , and / or the eye-ball position relative to the position of the sel f-mixing interferometry sensor .

[0040] According to at least one further aspect of the eye tracking method, the further sensor is a camera or an inertial measurement unit . For example , the latter comprises or consists of an accelerometer and / or a gyroscope . For example , the camera is used to determine the eye-ball shape , the eyeball si ze , the eye-ball curvature , eye-ball morphology and / or the eye-ball position . Further, the camera can be used to determine gaze data, for example . In particular, the second machine learning model can be configured to predict gaze data based on a combination of camera based eye-tracking and sel fmixing interferometry sensor based eye-tracking .

[0041] According to at least one further aspect of the eye tracking method, the second machine learning model is re-trained during eye tracking using the output of the third machine learning model and / or using the calibration data . Due to the separation of feature extraction and feature analysis by use of the first and second machine learning model , respectively, re-training of the second machine learning model can be much faster and can be performed on-the- fly compared to an architecture where only a single machine learning model is used for gaze data prediction . Accordingly, the performance of the gaze data prediction by the second machine learning model can be increased in situations , where the position of the eye-ball changes with respect to the sel f-mixing interferometry sensor during operation, for example .

[0042] According to at least one further aspect of the eye tracking method, the second machine learning model is used to predict gaze data at a series of future time steps . In other words , the second machine learning model is used to forecast a gaze traj ectory . For example , due to its recurrent neural network architecture , the second machine learning model can be used to forecast the gaze data, in particular the gaze direction and / or the gaze speed, at least 10 milliseconds or at least 30 milliseconds or at least 50 milliseconds into the future . This forecasting can be used to decrease a response-time or a lag of a device , such as a virtual reality or augmented reality device , that uses the gaze data as a user input , for example .

[0043] According to at least one further aspect of the eye tracking method, the first machine learning model is trained using a supervised contrastive regression loss method such that an embedding of features in the latent feature space has an increased robustness . For example , an output layer of the first machine learning model has a secondary output branch in the form of a proj ection head besides a primary output branch in the form of a regression head . For example , the proj ection head and the regression head are connected to the latent feature space of the first machine learning model . For example , the proj ection head comprises one or more dense layers ending with an output layer with a proj ection dimension between 2 and 2000 , inclusive , or between 10 and 100 , inclusive . For example , the regression head comprises one or more fully connected layers and has an output layer comprising output channels for the gaze data, in particular the gaze direction and / or the gaze speed .

[0044] For example , the first machine learning model is trained on the output of the proj ection head alone for a first number of epochs and is further trained on the output of the regression head alone for a second number of epochs , or vice versa . It is also possible to train on the outputs of the proj ection head and the regression head simultaneously . In particular, the output of the proj ection head is passed through a supervised contrastive regression loss during training of the first machine learning model . The supervised contrastive regression loss enforces that the first machine learning model learns a feature representation that captures an intrinsic ordered relationship between the input samples . Speci fically, input samples that are close together in the output space of the regression head, i . e . where the gaze data is close together, will also be close in the latent feature space . Advantageously, the supervised contrastive regression loss ensures that gaze traj ectories of di f ferent users and at varying eye-ball volume positions , for example , can be mapped to the same or similar locations in the latent feature space . Accordingly, the first machine learning model is as agnostic as possible to di f ferent users and varying eye-ball volume positions , for example .

[0045] For example , the supervised contrastive regression loss depends on a metric in the latent feature space and on a metric defined on the gaze data, e . g . a distance between gaze positions . For example , the supervised contrastive regression loss is defined such that for each pair of inputs the loss is large i f their distance in the latent feature space is large while their distance between predicted gaze positions is small . This ensures that the distance of features in the latent feature space correlates with the distance between predicted gaze positions , for example . For example , the regression head and the proj ection head are removed from the first machine learning model after the training of the first machine learning model is finished .

[0046] Further, an eye tracking sensor is speci fied herein . In particular, the eye tracking sensor uses the eye tracking method speci fied above . All features of the eye tracking method are also disclosed for the eye tracking sensor and vice versa .

[0047] According to an embodiment , the eye tracking sensor comprises :

[0048] - a sel f-mixing interferometry sensor, and

[0049] - a processing unit configured for predicting gaze data, wherein

[0050] - the processing unit uses time-series data from the sel fmixing interferometry sensor as an input of a first machine learning model , wherein the first machine learning model is configured for feature extraction and wherein an output of the first machine learning model comprises a latent feature space , and

[0051] - the processing unit uses a temporally stacked output of the first machine learning model as an input of a second machine learning model , wherein the second machine learning model is configured for feature analysis and wherein an output of the second machine learning model comprises the predicted gaze data .

[0052] For example, the processing unit comprises or consists of an application specific integrated circuit (short: ASIC) . For example, the computational power of the processing unit can be insufficient for training the first machine learning model. For example, the computational power of the processing unit can be sufficient for re-training the second machine learning model during operation of the eye tracking sensor.

[0053] According to a further aspect of the eye tracking sensor, the self-mixing interferometry sensor comprises at least one vertical cavity surface emitting laser diode (short: VCSEL) . In particular, the active laser medium of the VCSEL comprises a semiconductor layer stack with a pn-junction for converting an electric operating current into electromagnetic radiation. For example, an optical axis of the optical resonator of the VCSEL is arranged perpendicular to a main extension plane of semiconductor layers of the semiconductor layer stack. For example, the optical resonator comprises or consist of two mirrors, such as dielectric Bragg reflectors, arranged on opposite main faces of the semiconductor layer stack.

[0054] For example, the self-mixing interferometry sensor comprises or consists of one, two, three or more VCSELs . The two, three or more VCSELs emit electromagnetic laser radiation at the same or at different wavelengths. For example, the two, three or more VCSELs emit electromagnetic laser radiation towards different parts or locations of the tracked eye. In particular, the VCSELs emit electromagnetic laser radiation in the infrared spectral range. According to a further aspect of the eye tracking sensor, the time-series data from the sel f-mixing interferometry sensor comprises or consists of an output from a monitoring photodiode of the VCSEL . For example , the monitoring photodiode is arranged on a backside-mirror of the optical resonator . In particular, the backside-mirror has a higher reflectivity than a f rontside-mirror of the optical resonator . For example , during operation of the VCSEL, electromagnetic laser radiation is coupled out of the VCSEL mainly via the f rontside-mirror . It is also possible that the monitoring photodiode is arranged laterally adj acent to the VCSEL, where " lateral" refers to a direction perpendicular to a direction of the optical axis of the optical resonator of the VCSEL . For example , the time-series data comprises or consists of a voltage and / or a photo-current generated by the monitoring photodiode .

[0055] According to a further aspect of the eye tracking sensor, the time-series data from the sel f-mixing interferometry sensor comprises a measured forward voltage and / or an operating current of the VCSEL . For example , the forward voltage and / or the operating current is modulated in time due to the sel fmixing interference .

[0056] Further advantageous embodiments and further embodiments of the method for obtaining spectral data and the sensor become apparent from the following exemplary embodiments described in connection with the figures .

[0057] Figure 1 shows a schematic flow diagram of the eye tracking method according to an exemplary embodiment . Figure 2 shows a schematic illustration of the first machine learning model of the eye tracking method according to an exemplary embodiment .

[0058] Figure 3 shows a schematic illustration of the second machine learning model of the eye tracking method according to an exemplary embodiment .

[0059] Figure 4 shows a schematic illustration of the third machine learning model of the eye tracking method according to an exemplary embodiment .

[0060] Figure 5 shows simulated displacements of an eye-ball position for the eye tracking method according to an exemplary embodiment .

[0061] Figures 6A, 6B, 7A and 7B show a performance of the eye tracking method according to di f ferent exemplary embodiments .

[0062] Figures 8A, 8B, 8C and 8D show predicted displacements of an eye-ball position by a third machine learning model 3 of the eye tracking method according to an exemplary embodiment

[0063] Figures 9A and 9B show schematic cross sections of an eye tracking sensor according to di f ferent exemplary embodiments .

[0064] Elements that are identical , similar, or have the same ef fect , are denoted by the same reference signs in the figures . The figures and the proportions of the elements shown in the figures are not to be regarded as true to scale . Rather, individual elements , may be shown exaggeratedly large for better representability and / or better understanding . In a first step of the eye tracking method according to the exemplary embodiment of figure 1 , time-series data 41 from a sel f-mixing interferometry sensor 4 is read and used as an input 11 of a first machine learning model 1 . In particular, the time-series data 41 is fed to the first machine learning model 1 on a frame-by- frame basis , wherein one frame of the of the time-series data 41 comprises a set of 50 to 50000 sampled data points from the sel f-mixing interferometry sensor 4 , for example , and a time interval between these data points is between 0 . 1 ps and 100 ps , for example .

[0065] Optionally, the time-series data 41 is pre-processed before feeding it to the first machine learning model 1 . Preprocessing steps can include a trans formation of the timeseries data 41 to frequency space using a discrete Fourier trans formation, signal normali zation, signal scaling, temporal averaging, an averaging of frequency time windows , i . e . an averaging over a power spectral density, or other common routines that help to improve a signal-to-noise ratio and prepare the time-series data 41 for the first machine learning model 1 .

[0066] The first machine learning model 1 is a feedforward neural network, in particular a convolutional neural network, configured for feature extraction from the time-series data 41 for eye tracking . The extracted features from the time series data 41 form the latent feature space 13 . For example , the latent feature space 13 has a dimension between 2 and 5000 , inclusive .

[0067] In a second step of the method, a temporally stacked output 12 of the first machine learning model 1 is used as an input 21 of a second machine learning model 2 . In particular, the temporally stacked output 12 consists of 10 to 1000 samples of the latent feature space 13 of the first machine learning model 1 that are taken during a sliding past time window with a length between 5 ms to 5000 ms , for example .

[0068] Optionally, calibration data 61 of a further sensor 6 , such as a camera or an inertial measurement unit , and / or the output 32 of a third machine learning model 3 to be described below are used as further input 21 of the second machine learning model 2 in order to increase a prediction performance of the gaze data 23 . In particular, the output 32 of the third machine learning model 3 consists of user speci fic parameters 33 , such as an eye-ball shape , an eyeball si ze , an eye-ball curvature , and / or an eye-ball position relative to a position of the sel f-mixing interferometry sensor, and / or system speci fic parameters 33 , such as a wavelength, a power, a bandwidth, and / or a directional characteristic of the electromagnetic radiation emitted by the sel f-mixing interferometry sensor, or a geometric arrangement of the sel f-mixing interferometry sensor 4 relative to the tracked eye . The calibration data 61 comprises or consists of a measured eye-ball shape , eye-ball si ze , eye-ball curvature , and / or eye-ball position relative to the position of the sel f-mixing interferometry sensor, for example .

[0069] The second machine learning model 2 is a recurrent neural network that has a smaller number of parameters compared to the first machine learning model 1 and that is configured for feature analysis . In particular, the output 22 of the second machine learning model 2 comprises or consists of gaze data 23 , in particular a vertical gaze position, a hori zontal gaze position and a gaze speed . Due to its bi-directional nature , the recurrent neural network of the second machine learning model 2 is particularly suited for extracting temporal relationships between the features in the latent feature space 13 as a function of time . Accordingly, an accuracy of the raw predicted gaze data 23 can be increased . Moreover, the second machine learning model 2 allows to forecast the gaze traj ectory at future times .

[0070] In an optional third step, the raw predicted gaze data 23 that is provided at the output 22 of the second machine learning model 2 is digitally filtered using a digital filter 5 . In particular, the digital filter 5 improves a signal-to- noise ratio of the predicted gaze data 23 . Digital filters 5 can comprise a moving average computation, a Gaussian filter, and / or a Savit zky-Golay filter, for example . For example , a length of a temporal window of the digital filter is between 10 ms and 1000 ms , inclusive .

[0071] The third machine learning model 3 is a recurrent neural network with a similar architecture as the second machine learning model 2 . The input 31 of the third machine learning model 3 is a further temporally stacked output 12 of the first machine learning model 1 . However, the past time window of the third machine learning model 3 is longer than the past time window of the second machine learning model 2 . In particular, the past time window of the third machine learning model 3 has a length between and including 1 second and 500 seconds or more . The output 32 of the third machine learning model 3 comprises or consists of user and / or system speci fic parameters 33 . The first machine learning model 1 of the eye tracking method according to the exemplary embodiment in Figure 2 is a convolutional neural network with between 25000 and 2500000 parameters , inclusive . The first machine learning model 1 comprises residual connect blocks , squeeze-and-excitation blocks and a global average pooling layer for reducing the number of parameters before the architecture moves to a convolutional block with one to ten fully connected layers 16 , for example . After the convolutional block, a global averaging or flattening layer can be applied to proj ect the extracted features into a latent feature space 13 with a dimension between 2 and 5000 , inclusive .

[0072] A regression head 14 and a proj ection head 15 comprising multiple dense layers and a proj ection dimension between 10 and 1000 are connected to the latent feature space 13 for training the first machine learning model . During training, the output of the proj ection head 15 is passed through a supervised contrastive regression loss function in order to achieve a robust embedding of the extracted features in the latent feature space 13 .

[0073] The regression head 14 comprises fully connected layers that end in the same output channels as the second machine learning model 2 . Speci fically, the output channels of the regression head 14 correspond to the gaze data 23 , in particular the hori zontal and vertical gaze position as well as the gaze speed . For example , the first machine learning 1 model is first trained on the output for the proj ection head 15 alone for a first number of epochs , and is subsequently trained on the output of the regression head 14 alone for a second number of epochs , or vice versa, or the first machine learning model 1 is trained simultaneously on the outputs of the regression head 14 and the proj ection head 15 . After training, the proj ection head 15 and the regression head 14 are removed from the first machine learning model 1 .

[0074] The time-series data 41 is fed frame-by- frame to the input 11 of the first machine learning model 1 , wherein one frame comprises between 100 and 10000 samples of data points of the sel f-mixing interferometry sensor 4 taken at subsequent time steps . In particular, each data point comprises a readout from a monitoring photodiode 43 as well as a forward voltage of each VCSEL 42 of the sel f-mixing interferometry sensor 4 ( see e . g . Figure 9A) . For example , the sel f-mixing interferometry sensor 4 comprises 1 to 5 VCSELs .

[0075] The second machine learning model 2 of the eye tracking method according to the exemplary embodiment in Figure 3 is a recurrent neural network with between 2000 and 200000 parameters , inclusive . In particular, the number of parameters of the second machine learning model 2 is smaller than the number of parameters of the first machine learning model 1 .

[0076] The second machine learning model 2 comprises a recurrent block with one or multiple long short-term memory and / or gated recurrent unit layers , followed by one or more fully connected layers before ending in an output layer with output channels corresponding to the gaze data 23 , in particular the hori zontal and vertical gaze position and the gaze speed . It is also possible that the recurrent block is replaced by convolutional layers or dense layers . Further, a trans former architecture could replace the recurrent block or could be added to the recurrent block, and / or the recurrent block can have additional attention layers . The input 21 of the second machine learning model 2 are the extracted features 13 of the first machine learning model 1 concatenated over a past time window consisting of between 10 and 1000 past time points , inclusive . Moreover, additional calibration data 61 from a further sensor 6 and / or user / system speci fic parameters 33 from the third machine learning model 3 can be used as further input 21 of the second machine learning model 2 .

[0077] The third machine learning model 3 of the eye tracking method according to the exemplary embodiment in Figure 4 is a recurrent neural network with between 5000 and 500000 parameters , inclusive , that has a comparable architecture as the second machine learning model 2 . The third machine learning model 3 is configured for extracting long-term temporal dependencies . By contrast , the second machine learning model 2 is configured for extracting short-term temporal dependencies . In particular, the third machine learning model 3 can be used to extract the eye-ball volume position directly from the spatial and temporal information stored in the extracted features 13 of the first machine learning model 1 , such that an external sensor 6 for measuring the eye-ball position is not necessary .

[0078] Figure 5 shows an example of simulated deviations of an eyeball position that are used for estimating the performance of the second machine learning model 2 and of the third machine learning model 3 in the following figures 6A, 6B and 8A to 8D . Here , Ax denotes a hori zontal displacement , whereas Az denotes a depth displacement of the eye-ball position relative to the sel f-mixing interferometry sensor 4 . Here and in the following, "depth" corresponds to a direction perpendicular to the hori zontal direction and perpendicular to the vertical direction .

[0079] Figures 6A and 6B both show a mean absolute error score MAE of the gaze data 23 predicted by the second machine learning model 2 that receives the extracted features 13 from the first machine learning model 1 as an input 21 . The timeseries data 41 for the input 11 of the first machine learning model 1 was recorded at 7 x 7 = 49 di f ferent eye-ball volume positions from x = 11 . 5 mm to x = 12 . 5 mm and z = 9 . 5 mm to z = 10 . 5 mm, wherein x and z correspond to a hori zontal position and to a depth position of the eye-ball , respectively, with respect to the sel f-mixing interferometry sensor 4 . In both figures , the first machine learning model 1 was only trained using data from the central eye-ball position at x = 12 mm and z = 10 mm .

[0080] The mean absolute error MAE score shown in Fig . 6B corresponds to a second machine learning model 2 that was trained with calibration data 61 as further input 21 . Here , the calibration data 61 consists of information about the eye-ball position during recording of the time-series data 41 . By contrast , the mean absolute error MAE score shown in Fig . 6A corresponds to a second machine learning model 2 that was trained without calibration data 61 as further input 21 . Instead, the second machine learning model 2 corresponding to Fig . 6A was trained under the assumption that all time-series data 41 was recorded at the central eye-ball position x = 12 mm and z = 10 mm . As can be seen from Fig . 6B, providing the second machine learning model 2 with the calibration data 61 as further input 21 drastically reduces the error of the predicted gaze data 23 . Figures 7A and 7B show an exemplary mean absolute error MAE and an exemplary increase of an error E in percent , respectively, for a gaze data 32 forecasted by the second machine learning model 2 described in connection with Figure 3 up to a time t of 35 milliseconds into the future .

[0081] Figures 8A to 8D show predictions of eye-ball displacements in hori zontal Ax and depth Az directions obtained from the third machine learning model 3 described in connection with Figure 4 . In particular, the input 31 of the third machine learning model 3 are the extracted features 13 of the first machine learning model 1 for a past time window . Figures 8A and 8C shows the predicted eye-ball displacements for a training set of inputs 31 that is used to train the third machine learning model 3 , whereas Figure 8B and 8D shows the predicted eye-ball displacements for a test set of inputs 31 .

[0082] In Figures 8A and 8B the length of the past time window is 5 seconds . In Figures 8C and 8D the past time window comprises 10 time points where each time point is an average of the extracted features 13 of the first machine learning model 1 over the preceding 5 seconds and each of these ten time points is temporally spaced by 2 . 5 seconds . Accordingly, the third machine learning model 3 was able to consider a total of 25 seconds of information from the first machine learning model 1 for the results shown in Figures 8C and 8D . Consequently, the predicted eye-ball displacements corresponding to Figures 8C and 8D are more precise . Figures 8A to 8D thus show that the extracted features of the first machine learning model 1 over a large past time window can be used to identi fy system or user speci fic parameters 33 , such as the eye-ball displacement . The eye tracking sensor 10 according to the exemplary embodiment in Figure 9A comprises a sel f-mixing interferometry sensor 4 and a processing unit 7 . The sel fmixing interferometry sensor 7 further comprises a VCSEL 42 that emits electromagnetic laser radiation 8 in the infrared spectral range towards an eye 9 whose gaze position and gaze speed is to be tracked during operation . At least a part of the electromagnetic laser radiation 8 is reflected by the eye 9 and coupled back into the VCSEL 42 . A monitoring photodiode 43 that is arranged on a backside-mirror of the optical resonator of the VCSEL 42 is used to generate a part of the time-series data 41 of the sel f-mixing interferometry sensor 4 . Furthermore , the forward voltage of the VCSEL is used as further part of the time-series data 41 that is provided by the sel f-mixing interferometry sensor 4 to the processing unit 7 .

[0083] The processing unit 7 comprises a pre-compiled first machine learning model 1 and a second machine learning model 2 for predicting gaze data 23 from the time-series data 41 as described in connection with Figures 1 , 2 and 3 . In particular, the smaller second machine learning model 2 can be re-trained by the processing unit 7 on-the- fly, i f needed .

[0084] In Figure 9B, an optical element 91 is arranged along the optical path of the electromagnetic laser radiation 8 between eye tracking sensor 10 and the tracked eye 9 . For example , the eye tracking sensor 10 is mounted on a frame of eyeglasses (not shown) and the optical element 91 is a spectacle lens of the eyeglasses through which a user looks during operation of the eye tracking sensor 10 . In particular, the electromagnetic laser radiation 8 emitted by the eye tracking sensor 10 during operation is at least partially deflected towards the tracked eye 9 by the spectacle lens 91 .

[0085] This patent application claims the priority of German patent application DE 102023124967 . 3 , the disclosure content of which is hereby incorporated by reference .

[0086] The invention is not restricted to the exemplary embodiments by the description on the basis of said exemplary embodiments . Rather, the invention encompasses any new feature and also any combination of features , which in particular comprises any combination of features in the patent claims and any combination of features in the exemplary embodiments , even i f this feature or this combination itsel f is not explicitly speci fied in the patent claims or exemplary embodiments .

[0087] References

[0088] 1 first machine learning model

[0089] 11 input of the first machine learning model

[0090] 12 output of the first machine learning model

[0091] 13 latent feature space

[0092] 14 regression head

[0093] 15 proj ection head

[0094] 16 layers

[0095] 2 second machine learning model

[0096] 21 input of the second machine learning model

[0097] 22 output of the second machine learning model

[0098] 23 gaze data

[0099] 3 third machine learning model

[0100] 31 input of the third machine learning model

[0101] 32 output of the third machine learning model

[0102] 33 user / system speci fic parameters

[0103] 4 sel f-mixing interferometry sensor

[0104] 41 time-series data

[0105] 42 VCSEL

[0106] 43 monitoring photodiode

[0107] 5 digital filter

[0108] 6 further sensor

[0109] 61 calibration data

[0110] 7 processing unit

[0111] 8 electromagnetic laser radiation

[0112] 9 eye

[0113] 91 optical element

[0114] 10 eye tracking sensor

[0115] MAE mean absolute error

[0116] E error t time x hori zontal eye-ball position z depth of eye-position

[0117] Ax hori zontal deviation of eye-ball position

[0118] Az depth deviation of eye-ball position

Claims

Claims1. An eye tracking method, comprising the steps of:- reading time-series data (41) from a self-mixing interferometry sensor (4) ;- using the time-series data (31) as an input (11) of a first machine learning model (1) , wherein the first machine learning model (1) is configured for feature extraction and wherein an output (12) of the first machine learning model(1) comprises a latent feature space (13) ;- using a temporally stacked output (12) of the first machine learning model (1) as an input (21) of a second machine learning model (2) , wherein the second machine learning model(2) is configured for feature analysis and wherein an output (22) of the second machine learning model (2) comprises predicted gaze data (23) .

2. The eye tracking method according to the previous claim, comprising the further steps of:- pre-processing the time-series data (41) before using it as an input (11) of the first machine learning model (1) , and / or- temporal smoothening of the output (22) of the second machine learning model (2) using a digital filter (5) .

3. The eye tracking method according to any of the previous claims, wherein the first machine learning model (1) is a feedforward neural network.

4. The eye tracking method according to any of the previous claims, wherein the second machine learning model (2) is a recurrent neural network.

5. The eye tracking method according to any of the previous claims, comprising a further step of using a further temporally stacked output (12) of the first machine learning model (1) as an input (31) of a third machine learning model (3) , wherein an output (32) of the third machine learning model (3) comprises user specific parameters (33) and / or system specific parameters (33) .

6. The eye tracking method according to the previous claim, wherein the third machine learning model (3) is a recurrent neural network.

7. The eye tracking method according to the previous claim, wherein the output (32) of the third machine learning model(3) is used as a further input (21) of the second machine learning model (2) .

8. The eye tracking method according to any of the previous claims, wherein the second machine learning model (2) uses calibration data (61) from a further sensor (6) as a further input (21) .

9. The eye tracking method according to the previous claim, wherein the further sensor (6) is a camera or an inertial measurement unit.

10. The eye tracking method according to any of claims 7 to 9, wherein the second machine learning model (2) is retrained during eye tracking using the output (32) of the third machine learning model (3) and / or using the calibration data ( 1 ) .

11. The eye tracking method according to any of the previous claims, wherein the second machine learning model (2) is used to predict gaze data (23) at a series of future time steps.

12. The eye tracking method according to any of the previous claims, wherein the first machine learning model (1) is trained using a supervised contrastive regression loss method such that an embedding of features in the latent feature space (13) has an increased robustness.

13. An eye tracking sensor (10) , comprising:- a self-mixing interferometry sensor (4) , and- a processing unit (7) configured for predicting gaze data (23) , wherein- the processing unit (7) uses time-series data (41) from the self-mixing interferometry sensor (4) as an input (11) of a first machine learning model (1) , wherein the first machine learning model (1) is configured for feature extraction and wherein an output (12) of the first machine learning model(1) comprises a latent feature space (13) , and- the processing unit (7) uses a temporally stacked output (12) of the first machine learning model (1) as an input (21) of a second machine learning model (2) , wherein the second machine learning model (2) is configured for feature analysis and wherein an output (22) of the second machine learning model (2) comprises the predicted gaze data (23) .

14. The eye tracking sensor (10) according to the previous claim, wherein the self-mixing interferometry sensor (4) comprises at least one vertical cavity surface emitting laser diode (42) , VCSEL.

15. The eye tracking sensor (10) according to the previous claim, wherein the time-series data (41) from the self-mixing interferometry sensor (4) comprises:- an output from a monitoring photodiode (43) of the VCSEL ( 42 ) , and / or- a measured forward voltage and / or an operating current of the VCSEL (42) .

Images