Adaptive eye tracking method and device based on deep learning

Abstract

The disclosed embodiment discloses an adaptive eye tracking method and device based on deep learning. The method includes: obtaining eye images under various lighting conditions; preprocessing the eye images to obtain preprocessed eye images; constructing an initial eye tracking model based on a pulse neural network, and training the initial eye tracking model to obtain a trained eye tracking model; inputting the preprocessed eye images into the trained eye tracking model, and obtaining the movement trajectory of the eye in the eye image output by the trained eye tracking model; based on the movement trajectory, analyzing the eye data and reading behavior of the target user to obtain an analysis result. This method can not only maintain high-precision eye tracking under various lighting conditions and adapt to different lighting environments, but also analyze the reader's reading behavior in real time, providing strong support for personalized reading experience and content recommendation.

Description

Technical Field

[0001] The present disclosure relates to the field of eye tracking technology, and in particular to an adaptive eye tracking method and device based on deep learning. Background Art

[0002] Eye tracking and analysis is a technology based on image processing and computer vision that is used to track and analyze the movement of the human eye in images or videos in real time. Eye tracking technology has broad application prospects in reading analysis, human-computer interaction, medical diagnosis and other fields.

[0003] In the related art, a 3D dynamic model is displayed on a display screen to observe and induce pupil movements. However, the eye tracking technology of the related art often has the following technical defects when facing different lighting conditions: errors are prone to occur when the light changes, reducing the accuracy and precision of tracking; calibration is required under a fixed lighting environment, and it is difficult to adapt to dynamically changing lighting conditions, and the adaptability is poor; some high-precision eye tracking systems often require a long calculation time when processing complex lighting conditions and cannot meet real-time requirements; it is difficult to quickly adapt to the eye characteristics of different users, such as individual differences in eye shape, pupil size, etc., and the adaptability to body differences is low; eye tracking equipment may cause discomfort to users, cause eye fatigue after long-term use, and affect the long-term reading experience. Summary of the Invention

[0004] In view of this, the embodiments of the present disclosure provide an adaptive eye tracking method and device based on deep learning. By combining the ViT (Vision Transformer, a deep learning model based on the attention mechanism) adaptive token on the pulse neural network and the optimal decision tree method for approximate logical synthesis, the accurate capture and analysis of the reader's eye movements are achieved. The system can not only maintain high-precision eye tracking under various lighting conditions and adapt to different lighting environments, but also analyze the reader's reading behavior in real time, providing strong support for personalized reading experience and content recommendation.

[0005] In a first aspect, the embodiments of the present disclosure provide an adaptive eye tracking method based on deep learning, which adopts the following technical solutions:

[0006] Acquire eye images under various lighting conditions;

[0007] Preprocessing the eyeball image to obtain a preprocessed eyeball image;

[0008] Building an initial eye tracking model based on a spiking neural network, and training the initial eye tracking model to obtain a trained eye tracking model;

[0009] Inputting the preprocessed eye image into the trained eye tracking model, and obtaining a motion trajectory of the eye in the eye image output by the trained eye tracking model;

[0010] Based on the movement trajectory, the target user's eye data and reading behavior are analyzed to obtain analysis results.

[0011] In some embodiments, images of the eyeball are acquired under various lighting conditions, including:

[0012] Set the image acquisition environment with different light source intensities or angles;

[0013] The eye movements of a plurality of sampled users under various lighting conditions in the image acquisition environment are captured by a photographing device; wherein the plurality of sampled users have different ages and / or eye features.

[0014] In some embodiments, preprocessing the eye image to obtain a preprocessed eye image includes:

[0015] Performing denoising on the eyeball image to obtain a denoised eyeball image;

[0016] Performing enhancement processing on the denoised eyeball image to obtain an enhanced eyeball image;

[0017] Detecting the position and boundary of the eyeball region in the enhanced eyeball image, and performing segmentation processing on the eyeball region according to the detected position and boundary;

[0018] The segmented eyeball region is subjected to standardization and normalization to obtain a pre-processed eyeball image.

[0019] In some embodiments, constructing an initial eye tracking model based on a spiking neural network includes:

[0020] Define the spiking neuron model;

[0021] Based on the spiking neuron model, build the Vision Transformer architecture;

[0022] Initialize several learnable tokens, where each token is a D-dimensional vector;

[0023] During each forward propagation, adaptively adjusting the plurality of learnable tokens according to the light conditions of the input eye image;

[0024] Concatenating the adjusted number of learnable tokens with the image patch embedding of the Vision Transformer architecture;

[0025] The concatenated image block embedding and the plurality of learnable tokens are input into the Transformer encoder of the VisionTransformer architecture to obtain the initial eye tracking model.

[0026] In some embodiments, training the initial eye tracking model to obtain a trained eye tracking model includes:

[0027] Acquire a data set of eyeball image samples of several sampled users, wherein the data set of eyeball image samples includes eyeball image samples under various lighting conditions;

[0028] Inputting the eyeball image samples under the multiple light conditions into the initial eyeball tracking model, and obtaining motion trajectory samples of the eyeball samples output by the initial eyeball tracking model in the eyeball image samples;

[0029] Calculating the loss between the motion trajectory sample and the target motion trajectory using a defined loss function to obtain a loss value;

[0030] Based on the loss value, optimizing and adjusting relevant model parameters of the initial eye tracking model through an optimizer until the loss value is less than or equal to a preset threshold, and outputting a trained eye tracking model;

[0031] The loss function is expressed as follows:

[0032] L total = α * L position + β * L gaze + γ * L adaptive + λ * L reg ;

[0033] Where, L position Represents the MSE loss of eye feature point positioning, which includes eye corners and pupil centers; L gaze Represents the angular error of the line of sight direction prediction; L adaptive represents the consistency loss of the adaptive token; L reg Represents the regularization term to prevent overfitting; α is L position The weight of L gaze The weight of L adaptive The weight of L reg The weight of .

[0034] In some embodiments, the method further comprises:

[0035] Extracting a plurality of key eye features from the eye image sample, wherein the key eye features include pupil center coordinates, pupil size, eye corner position, eyelid position, iris edge point, eye rotation angle, eye movement speed or pupil light reflection position;

[0036] Calculate the importance of each of the key eye features using the trained random forest model;

[0037] selecting a target eye feature from the plurality of key eye features based on the importance of the key eye features;

[0038] Constructing a decision tree and selecting the target eye feature that can minimize impurity or error as the optimal splitting feature;

[0039] Determining an optimal splitting threshold based on the optimal splitting feature;

[0040] Constructing a subtree by a recursive method, and iteratively executing the steps of selecting the target eye feature that can minimize impurity or error as the best splitting feature through the subtree, and determining the best splitting threshold based on the best splitting feature, until an iteration stop condition is reached;

[0041] Optimizing the decision tree by post-pruning method to obtain an optimal decision tree;

[0042] Performing ensemble learning on the optimal decision tree using a random forest model and a gradient boosting tree model;

[0043] The initial eye tracking model is verified and evaluated by using k-fold cross validation, and the trained eye tracking model is determined according to the verification and evaluation results.

[0044] In some embodiments, based on the motion trajectory, the target user's eye data and reading behavior are analyzed to obtain analysis results, including:

[0045] Calculating the target user's eye gaze point and eye gaze duration based on the motion trajectory;

[0046] Identifying the target user's eye movements and gaze-returning behaviors;

[0047] generating a heat map and a reading path based on the saccadic behavior and the reversion behavior;

[0048] Extracting the target user's reading behavior characteristics based on the heat map and reading path;

[0049] Identifying the target user's reading pattern through cluster analysis based on the reading behavior characteristics;

[0050] Based on the reading pattern, a personalized reading analysis system is established.

[0051] In a second aspect, the present disclosure also provides an adaptive eye tracking device based on deep learning, which adopts the following technical solutions:

[0052] an acquisition unit, configured to acquire eyeball images under various lighting conditions;

[0053] a preprocessing unit, configured to preprocess the eyeball image to obtain a preprocessed eyeball image;

[0054] a model training unit configured to construct an initial eye tracking model based on a spiking neural network, and train the initial eye tracking model to obtain a trained eye tracking model;

[0055] an input / output unit configured to input the preprocessed eye image into the trained eye tracking model and obtain a motion trajectory of the eye in the eye image output by the trained eye tracking model;

[0056] The analyzing unit is configured to analyze the eyeball data and reading behavior of the target user based on the motion trajectory to obtain an analysis result.

[0057] In a third aspect, the embodiments of the present disclosure further provide a computer device that adopts the following technical solution:

[0058] The computer device comprises:

[0059] at least one processor; and,

[0060] a memory communicatively connected to the at least one processor; wherein,

[0061] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned deep learning-based adaptive eye tracking methods.

[0062] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned deep learning-based adaptive eye tracking methods.

[0063] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0064] The embodiments of the present disclosure provide an adaptive eye tracking method based on deep learning. By combining ViT (Vision Transformer, a deep learning model based on the attention mechanism) on a spiking neural network for approximate logic synthesis, the method achieves accurate capture and analysis of the reader's eye movements. The system can not only maintain high-precision eye tracking under various lighting conditions and adapt to different lighting environments, but also analyze the reader's reading behavior in real time, providing strong support for personalized reading experience and content recommendation.

[0065] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] Figure 1 A flowchart of an adaptive eye tracking method based on deep learning provided by an embodiment of the present disclosure;

[0068] Figure 2 A schematic diagram of a process for optimizing the model training process of an initial eye tracking model provided in an embodiment of the present disclosure;

[0069] Figure 3 A flowchart illustrating the optimal decision tree implementation process provided by the embodiment of the present disclosure;

[0070] Figure 4 A schematic diagram of a flow chart for optimizing the performance of an eye tracking system under different lighting conditions using an optimal decision tree method according to an embodiment of the present disclosure;

[0071] Figure 5 A schematic diagram of the structure of an adaptive eye tracking device based on deep learning provided by an embodiment of the present disclosure;

[0072] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0074] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0075] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0076] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0077] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0078] like Figure 1 As shown, Figure 1 This is a flow chart of an adaptive eye tracking method based on deep learning provided by an embodiment of the present disclosure. The adaptive eye tracking method based on deep learning provided by an embodiment of the present disclosure includes the following steps:

[0079] S101. Acquire eyeball images under various lighting conditions.

[0080] S102: Preprocess the eyeball image to obtain a preprocessed eyeball image.

[0081] S103: constructing an initial eye tracking model based on a spiking neural network, and training the initial eye tracking model to obtain a trained eye tracking model.

[0082] The embodiments of the present disclosure achieve accurate capture and analysis of the reader's (i.e., the target user in the embodiments of the present disclosure) eye movements by combining the adaptive tokens of the Vision Transformer deep learning model on the spiking neural network and the optimal decision tree method for approximate logic synthesis, and can maintain high-precision eye tracking under various lighting conditions.

[0083] S104: Input the pre-processed eye image into the trained eye tracking model, and obtain the movement trajectory of the eye in the eye image output by the trained eye tracking model.

[0084] S105 : Analyze the target user's eye data and reading behavior based on the movement trajectory to obtain analysis results.

[0085] The embodiments of the present disclosure provide an adaptive eye tracking method based on deep learning. By combining ViT (Vision Transformer, a deep learning model based on the attention mechanism) on a spiking neural network for approximate logic synthesis, the method achieves accurate capture and analysis of the reader's eye movements. The system can not only maintain high-precision eye tracking under various lighting conditions and adapt to different lighting environments, but also analyze the reader's reading behavior in real time, providing strong support for personalized reading experience and content recommendation.

[0086] In some embodiments, images of the eyeball are acquired under various lighting conditions, including:

[0087] Set the image acquisition environment with different light source intensities or angles;

[0088] The eye movements of a plurality of sampled users under various lighting conditions in the image acquisition environment are captured by a photographing device; wherein the plurality of sampled users have different ages and / or eye features.

[0089] Optionally, the photographing device may be a high-resolution camera or other device. The user may select a suitable photographing device according to actual needs, and the embodiments of the present disclosure do not limit this.

[0090] In some embodiments, preprocessing the eye image to obtain a preprocessed eye image includes:

[0091] Performing denoising on the eyeball image to obtain a denoised eyeball image;

[0092] Performing enhancement processing on the denoised eyeball image to obtain an enhanced eyeball image;

[0093] Detecting the position and boundary of the eyeball region in the enhanced eyeball image, and performing segmentation processing on the eyeball region according to the detected position and boundary;

[0094] The segmented eyeball region is subjected to standardization and normalization to obtain a pre-processed eyeball image.

[0095] In some embodiments, constructing an initial eye tracking model based on a spiking neural network includes:

[0096] Define the spiking neuron model;

[0097] Based on the spiking neuron model, build the Vision Transformer architecture;

[0098] Initialize several learnable tokens, where each token is a D-dimensional vector;

[0099] During each forward propagation, adaptively adjusting the plurality of learnable tokens according to the light conditions of the input eye image;

[0100] Concatenating the adjusted number of learnable tokens with the image patch embedding of the Vision Transformer architecture;

[0101] The concatenated image block embedding and the plurality of learnable tokens are input into the Transformer encoder of the VisionTransformer architecture to obtain the initial eye tracking model.

[0102] Optionally, the spiking neuron model is the basis of the biomimetic neural network, which simulates the behavior of biological neurons. In the embodiment of the present disclosure, the Leaky Integrate-and-Fire (LIF) model is adopted, and its mathematical expression is as follows: τ(dV / dt) = -(V - V rest ) + RI(t); where V represents the membrane potential, V rest represents the resting potential, τ represents the time constant, R represents the membrane resistance, and I(t) represents the input current. When V reaches the threshold V th When V≥V th , reset the pulse to V = V reset .

[0103] Optionally, the core of the Vision Transformer architecture is to segment the eye image into fixed-size blocks, and then input these segmented blocks as a sequence into the Transformer encoder. The main components of the Vision Transformer architecture include: 1. Block embedding: used to segment the eye image into N blocks, each block is converted into a D-dimensional vector through linear projection; 2. Position encoding: adding position information to each block; 3. Transformer encoder: contains a multi-layer self-attention mechanism and a feedforward neural network.

[0104] The disclosed embodiment uses an adaptive token mechanism to enable the initial eye tracking model to dynamically adjust its focus point to adapt to different lighting conditions.

[0105] In some embodiments, training the initial eye tracking model to obtain a trained eye tracking model includes:

[0106] Acquire a data set of eyeball image samples of several sampled users, wherein the data set of eyeball image samples includes eyeball image samples under various lighting conditions;

[0107] Inputting the eyeball image samples under the multiple light conditions into the initial eyeball tracking model, and obtaining motion trajectory samples of the eyeball samples output by the initial eyeball tracking model in the eyeball image samples;

[0108] Calculating the loss between the motion trajectory sample and the target motion trajectory using a defined loss function to obtain a loss value;

[0109] Based on the loss value, optimizing and adjusting relevant model parameters of the initial eye tracking model through an optimizer until the loss value is less than or equal to a preset threshold, and outputting a trained eye tracking model;

[0110] The loss function is expressed as follows:

[0111] L total = α * L position + β * L gaze + γ * L adaptive + λ * L reg ;

[0112] Where, L position Represents the MSE loss of eye feature point positioning, which includes eye corners and pupil centers; L gaze Represents the angular error of the line of sight direction prediction; L adaptive represents the consistency loss of the adaptive token; L reg Represents the regularization term to prevent overfitting; α is Lposition The weight of L gaze The weight of L adaptive The weight of L reg The weight of .

[0113] Optionally, the embodiment of the present disclosure divides the collected eye image sample data set into a training set, a validation set, and a test set in a ratio of 6:2:2 to ensure that each subset contains eye image sample data under various lighting conditions.

[0114] It should be noted that the user can divide the eye image sample data set according to the proportion of actual needs, and the embodiments of the present disclosure do not limit this.

[0115] Optionally, the optimizer uses the Adam optimizer, the initial learning rate is set to 1e-4, and the cosine annealing scheduling strategy is used.

[0116] Optionally, embodiments of the present disclosure monitor the model training process and implement an early stopping strategy. Specifically, tools such as TensorBoard (a set of visualization tools provided by TensorFlow) are used to monitor the model training process in real time, recording metrics such as loss and accuracy. If the performance of the validation set does not improve after 10 consecutive epochs (a common charting library for application developers and visualization designers), model training is stopped, implementing the early stopping strategy. Furthermore, model parameters with the lowest loss are saved on the validation set.

[0117] like Figure 2 As shown, Figure 2 A schematic diagram of a process for optimizing the model training process of an initial eye tracking model provided in an embodiment of the present disclosure is provided. The present disclosure embodiment optimizes the model training process of the initial eye tracking model, including the following steps:

[0118] Step S1: Use the Bayesian optimization method to automatically search for the optimal hyperparameter combination and tune the hyperparameters.

[0119] Among them, the main hyperparameters to be tuned may include: the number of layers and heads of the Transformer, the number of adaptive tokens, and the weight coefficients of each component in the loss function.

[0120] Step S2: compress and quantize the spiking neuron model.

[0121] Specifically, the spiking neuron model is pruned to remove unimportant connections and neurons. For example, L1 regularization is used to force the weights of some components in the loss function to be set to zero. A smaller student model is trained through knowledge distillation to mimic the large model, compressing the spiking neuron model. The floating-point weights in the spiking neuron model are converted to 8-bit or 16-bit integers, reducing the size of the spiking neuron model and inference time.

[0122] Step S3: Pre-train the initial eye tracking model on a large-scale eye image dataset, and fine-tune the pre-trained initial eye tracking model using a small amount of data from the target scene to make the pre-trained initial eye tracking model adaptable to the specific application environment.

[0123] This disclosure describes how to combine spiking neural networks and the Vision Transformer architecture to achieve high-precision eye tracking.

[0124] Step A1: Prepare the input eye image.

[0125] Step A1.1: Preprocess the eyeball image.

[0126] Step A1.1.1: Resize the input eyeball image to a uniform size of 224x224 pixels.

[0127] Step A1.1.2: Apply contrast-limited adaptive histogram equalization (CLAHE) to enhance the details of the eye image.

[0128] Step A1.1.3: Convert the eyeball image in RGB format to a grayscale image to reduce the amount of calculation.

[0129] Step A1.2: Pulse-encode the eye image.

[0130] Step A1.2.1: Convert the grayscale values ​​of the grayscale image into a pulse train using rate coding.

[0131] For example, for an 8-bit grayscale image, the probability of a spike at pixel value x is set to: p(spike) = x / 255.

[0132] Step A1.2.2: Generate a pulse train of T time steps in the time dimension.

[0133] Step A2: Process the eye image through a spiking neural network.

[0134] Step A2.1: Construct a spike convolution layer.

[0135] Step A2.1.1: Use the IF neuron model to perform a pulse convolution operation on the eye image.

[0136] Step A2.1.2: Design a three-layer impulse convolution structure, using 32, 64, and 128 3x3 convolution kernels respectively.

[0137] Step A2.2: Perform temporal dimension aggregation processing on the eyeball image.

[0138] Step A2.2.1: Use the maximum pooling operation in the time dimension to aggregate the feature maps of the eye image of T time steps into a single feature map.

[0139] Step A2.2.2: Use spatial max pooling to reduce the size of the feature map of the eye image to 14x14.

[0140] Step A3: Use the Vision Transformer architecture to process the feature map of the eye image.

[0141] Step A3.1: Divide and embed the feature map of the eye image.

[0142] Step A3.1.1: Divide the feature map of the eye image of size 14x14 into 2x2 blocks, resulting in a total of 49 blocks.

[0143] Step A3.1.2: Embed each block into a 768-dimensional vector space using linear projection.

[0144] Step A3.2: Fuse the tokens with the Vision Transformer architecture.

[0145] Step A3.2.1: Initialize a number of learnable tokens (10 in the embodiment of the present disclosure), each token is a 768-dimensional vector.

[0146] Step A3.2.2: Use the attention mechanism to adjust these tokens based on the input eye image.

[0147] Step A3.2.3: Concatenate the adjusted token and the image patch embedding to form a sequence of length 59.

[0148] Step A3.3: Build the Transformer encoder.

[0149] Step A3.3.1: Build a 12-layer Transformer encoder, with each layer containing an 8-head self-attention mechanism.

[0150] Step A3.3.2: Perform normalization and residual connection processing in the application layer after each layer of Transformer encoder.

[0151] Step A4: Use the initial eye tracking model to perform data output processing and prediction.

[0152] Step A4.1: Extract features from the eyeball image.

[0153] Step A4.1.1: Use the output of the adaptive token as the global feature of the eye image.

[0154] Step A4.1.2: Map the global features of the eye image to the desired output dimension through a fully connected layer.

[0155] Step A4.2: Output the global features of the eye image through multi-task.

[0156] Step A4.2.1: Use the MSE loss function to predict the coordinates of eye feature points (such as eye corners, pupil center, etc.).

[0157] Step A4.2.2: Predict the gaze direction (horizontal and vertical angles) using the L1 loss function.

[0158] Step A4.2.3: Estimate pupil size using relative error loss function.

[0159] Step A5: Adapting eye tracking to different lighting conditions

[0160] Step A5.1: Encode the lighting conditions.

[0161] Step A5.1.1: Encode the lighting conditions using global statistics of the eye image (e.g., average brightness, contrast, etc.).

[0162] Step A5.1.2: Inject light condition information into the adaptive token.

[0163] Step A5.2: Data enhancement processing of the eyeball image.

[0164] Step A5.2.1: Randomly adjust the brightness, contrast, and noise level of the eye image during the initial eye tracking model training process.

[0165] Step A5.2.2: Simulate different light source directions and intensities to enhance the generalization ability of the initial eye tracking model.

[0166] Step A5.3: Perform domain adaptation training on the initial eye tracking model.

[0167] Step A5.3.1: Introduce a domain discriminator to distinguish eye features under different lighting conditions.

[0168] Step A5.3.2: Through adversarial training, the eye features are made invariant under different lighting conditions.

[0169] Based on the above steps, the disclosed embodiment constructs an eye-tracking model that combines a spiking neural network with the Vision Transformer architecture. This model effectively processes time series information and captures the dynamic characteristics of eye movements. It also leverages the powerful representation capabilities of the Vision Transformer architecture's Transformer encoder to adapt to varying lighting conditions. Furthermore, an adaptive token mechanism further enhances the eye-tracking model's adaptability to environmental changes, maintaining high-precision eye-tracking performance under various lighting conditions.

[0170] The embodiments of the present disclosure demonstrate how to integrate multiple advanced deep learning technologies into an eye tracking system, laying the foundation for robust and high-precision eye movement analysis. In practical applications, the embodiments of the present disclosure can be further optimized and adjusted based on specific hardware limitations and accuracy requirements, which are not limited by the embodiments of the present disclosure.

[0171] In some embodiments, the method further comprises:

[0172] Extracting a plurality of key eye features from the eye image sample, wherein the key eye features include pupil center coordinates, pupil size, eye corner position, eyelid position, iris edge point, eye rotation angle, eye movement speed or pupil light reflection position;

[0173] Calculate the importance of each of the key eye features using the trained random forest model;

[0174] selecting a target eye feature from the plurality of key eye features based on the importance of the key eye features;

[0175] Constructing a decision tree and selecting the target eye feature that can minimize impurity or error as the optimal splitting feature;

[0176] Determining an optimal splitting threshold based on the optimal splitting feature;

[0177] Constructing a subtree by a recursive method, and iteratively executing the steps of selecting the target eye feature that can minimize impurity or error as the best splitting feature through the subtree, and determining the best splitting threshold based on the best splitting feature, until an iteration stop condition is reached;

[0178] Optimizing the decision tree by post-pruning method to obtain an optimal decision tree;

[0179] Performing ensemble learning on the optimal decision tree using a random forest model and a gradient boosting tree model;

[0180] The initial eye tracking model is verified and evaluated by using k-fold cross validation, and the trained eye tracking model is determined according to the verification and evaluation results.

[0181] like Figure 3 As shown, Figure 3 This is a flow chart of the optimal decision tree implementation process provided by the embodiment of the present disclosure. The optimal decision tree implementation process provided by the embodiment of the present disclosure includes the following steps:

[0182] Step a1: Extract key eye features.

[0183] The disclosed embodiment extracts and calculates several key eye features from the output of the initial eye tracking model: pupil center coordinates (x, y), pupil size (diameter or area), eye corner position (coordinates of the inner and outer corners of the eye), eyelid position (contour points of the upper and lower eyelids), iris edge points (at least 8 evenly distributed points), eye rotation angle (horizontal and vertical directions), eye movement speed (position changes based on consecutive frames), and pupil light reflex position.

[0184] Step a2: Calculate the importance of key eye features.

[0185] The feature importance evaluation method of the random forest model is used to determine the importance of each key eye feature. The specific process is as follows:

[0186] Step a2.1: Train a random forest model to predict the accurate gaze direction using the key eye features extracted above.

[0187] Step a2.2: Calculate the importance score of each key eye feature using Gini impurity or feature ranking method.

[0188] Step a2.3: Sort the key eye features by importance score.

[0189] Step a3: Perform feature selection and dimensionality reduction based on the feature importance of key eye features.

[0190] Step a3.1: Select the top 80% of the key eyeball features according to the importance scores.

[0191] Step a3.2: Apply principal component analysis (PCA) to process the selected key eye features and retain the principal components that explain 95% of the variance.

[0192] Step a3.2.3: If there is a high correlation between key eye features, use the Lasso regression model for further feature selection.

[0193] Step a4: Select the best split feature and build a decision tree using the CART (Classification and Regression Trees) algorithm.

[0194] Optionally, for each node, the Gini impurity or mean square error of all possible feature splits is calculated, and the feature that can minimize the impurity or error is selected as the best split feature.

[0195] Step a5: Determine the optimal splitting threshold for the selected optimal splitting feature.

[0196] Optionally, the eigenvalues ​​of the best splitting features are sorted, all possible splitting points are tried, and the threshold that maximizes the information gain is selected as the best splitting threshold. A dynamic programming algorithm is used to optimize the search process and improve computational efficiency.

[0197] Step a6: Repeat step a4 above for the left and right subtrees (i.e., each child node) using a recursive method to select the best splitting feature and construct a decision tree using the CART (Classification and Regression Trees) algorithm; and step a5 above: determine the optimal splitting threshold for the selected best splitting feature until the iteration stopping condition is reached.

[0198] Optionally, the iteration stopping condition is set to: when the iteration process reaches the maximum depth, the minimum number of samples, the minimum impurity reduction, etc., the iteration is stopped.

[0199] Step a7: Optimize the decision tree by post-pruning to obtain the optimal decision tree.

[0200] Generate a complete decision tree and evaluate each non-leaf node from the bottom up. If deleting a subtree can improve the performance of the validation set, prune the subtree and use the cost complexity parameter α to balance the size and performance of the decision tree.

[0201] Step a8: Use the random forest model and the gradient boosting tree model to perform ensemble learning on the optimal decision tree to further improve the performance of the optimal decision tree.

[0202] Optionally, using a random forest model to perform ensemble learning on the optimal decision tree includes the following steps:

[0203] Use Bootstrap sampling to generate multiple training sets;

[0204] Build a decision tree for each training set, considering only a subset of features at each split;

[0205] The final prediction of the output result is made using the average value of all subtrees or the majority vote of the subtrees.

[0206] Optionally, using a gradient boosting tree model (e.g., the optimized distributed gradient boosting library XGBoost) to perform ensemble learning on the optimal decision tree includes the following steps:

[0207] Initialize a simple model; iteratively train new decision trees to fit the residual; use the step size (learning rate) to control the contribution of each decision tree.

[0208] Step a9: Validate and evaluate the initial eye tracking model using k-fold cross validation, and select the best initial eye tracking model.

[0209] Optionally, the training set is divided into k parts (for example, k can be set to 5 or 10); the parameters of the initial eye tracking model are configured (for example, parameters such as tree depth and minimum number of samples of leaf nodes are configured), and the initial eye tracking model is trained and validated k times; the average performance indicators (for example, MAE, RMSE, etc.) and standard deviation are calculated; and the model configuration with the best performance and the smallest variance is selected to obtain the best initial eye tracking model.

[0210] The embodiments of the present disclosure describe in detail how to use the optimal decision tree method to optimize the performance of an eye tracking system under different lighting conditions to improve eye tracking accuracy.

[0211] like Figure 4 As shown, Figure 4 This is a flow chart of using an optimal decision tree method to optimize the performance of an eye tracking system under different lighting conditions according to an embodiment of the present disclosure. The optimal decision tree method to optimize the performance of an eye tracking system under different lighting conditions according to an embodiment of the present disclosure includes the following steps:

[0212] Step A1: Extract and preprocess key eye features.

[0213] Step A1.1: Extract key eye features from the output of the initial eye tracking model.

[0214] Step A1.1.1. Extract pupil center coordinates (x pupil ,y pupil );

[0215] Step A1.1.2. Calculate pupil diameter d pupil ;

[0216] Step A1.1.3, extract the inner corner coordinates (x inner , y inner ), and extract the outer corner coordinates (x outer ,y outer );

[0217] Step A1.1.4. Calculate the eye width w eye = x outer -x inner ;

[0218] Step A1.1.5: Calculate the relative pupil position r x = (x pupil -x inner ) / w eye , r y = (y pupil -y inner ) / w eye .

[0219] Step A1.2: Calculate light-related features.

[0220] Step A1.2.1. Calculate the average brightness b of the image avg ;

[0221] Step A1.2.2, calculate the image contrast;

[0222] Step A1.2.3: Detect and quantify glare in the eye area intensity .

[0223] Step A1.3: Standardize the light-related features.

[0224] Step A1.3.1. Perform Z-score normalization on all light-related features.

[0225] Step A1.3.2: Based on the outliers in the Z-score normalization results, outliers exceeding 3 standard deviations are truncated.

[0226] Step A2: Select key eye features.

[0227] Step A2.1: Use the random forest model to evaluate the importance of each key eye feature;

[0228] Step A2.1.1: Train the random forest model with the target variable being the gaze direction. x , gaze y );

[0229] Step A2.1.2, calculate the importance score of each key eye feature;

[0230] Step A2.1.3: Select the top 80% of the key eye features with the highest importance scores.

[0231] Step A2.2: Apply principal component analysis (PCA) to perform dimensionality reduction on the selected key eye features and select the principal component with an explained variance of 95%.

[0232] Step A3: Build a decision tree.

[0233] Step A3.1, initialize the decision tree;

[0234] Step A3.1.1. Set the maximum depth of the decision tree to 10;

[0235] Step A3.1.2, set the minimum number of leaf node samples to 20;

[0236] Step A3.1.3, use mean square error as the splitting criterion;

[0237] Step A3.2, recursively construct subtrees;

[0238] Step A3.2.1. For each node, calculate the optimal splitting point of all key eye features;

[0239] Step A3.2.2: Select the key eye features and splitting points that can minimize the mean square error to obtain the optimal classification features;

[0240] Step A3.2.3: Create left and right child nodes and recursively build the subtree;

[0241] Step A4: Optimize the decision tree.

[0242] Step A4.1: Optimize the decision tree by post-pruning to obtain the optimal decision tree.

[0243] Step A4.1.1. Evaluate each non-leaf node in the decision tree from the bottom up.

[0244] Step A4.1.2: If pruning the subtree reduces the validation set error, prune the subtree.

[0245] Step A4.1.3. Use the cost complexity parameter α = 0.01 to control the degree of pruning to balance the size and performance of the decision tree.

[0246] Step A4.2: Build a random forest model.

[0247] Step A4.2.1. Create 100 decision trees.

[0248] Step A4.2.2: Each decision tree uses a bootstrap sampled dataset;

[0249] Step A4.2.3: Each split randomly selects a subset of features, where the subset size is the square root of the total number of features;

[0250] Step A4.3: Build a gradient boosting tree (for example, the gradient boosting tree is XGBoost);

[0251] The maximum number of trees is set to 1000, the number of early stopping steps is set to 50, the learning rate is set to 0.01, and the maximum depth of each tree is set to 6.

[0252] Step A5: Verify, evaluate and select the initial eye tracking model.

[0253] Step A5.1: Perform 5-fold cross validation on the initial eye tracking model.

[0254] Among them, cross-validation was performed on a single decision tree, random forest model, and XGBoost respectively; the average MAE, RMSE, and standard deviation of each initial eye tracking model were calculated.

[0255] Step A5.2: Test the initial eye tracking model under different lighting conditions.

[0256] Step A5.2.1. Prepare test sets under different lighting conditions (e.g., normal light, low light, bright light, side light, etc.);

[0257] Step A5.2.2. Evaluate the performance of each initial eye tracking model under various lighting conditions.

[0258] Taking into account the average performance, stability and model performance under different lighting conditions, the model with the best performance, smallest variance and most stability is selected as the final model to obtain the best initial eye tracking model.

[0259] Optionally, the disclosed embodiment optimizes the selected best eye tracking model, for example, by analyzing the importance of eye features in the finally selected eye tracking model and identifying eye features that play a key role under different lighting conditions.

[0260] Optionally, the embodiment of the present disclosure analyzes the decision path of typical training samples in a decision tree or random forest model to identify key decision points and thresholds.

[0261] Optionally, the embodiments of the present disclosure fine-tune the selected optimal eye tracking model based on the analysis results of the importance of eye features or the analysis results of the decision paths of typical training samples, for example, adjusting the process of extracting key eye features, optimizing model parameters (for example, the depth of the decision tree, the minimum number of leaf node samples, etc.), and considering adding specialized models for specific lighting conditions, etc.

[0262] In summary, the embodiments of the present disclosure utilize an optimal decision tree method to optimize the performance of an eye tracking system under different lighting conditions. This method combines the powerful feature extraction capabilities of a deep learning model with the interpretability and adaptability of a decision tree model. Through key eye feature extraction, model integration, and targeted optimization, the embodiments of the present disclosure can achieve high-precision eye tracking under various lighting conditions. It can not only capture complex nonlinear relationships and provide interpretability for the model decision process, but also quickly adapt to different lighting conditions, has high computational efficiency, and is suitable for real-time applications.

[0263] In some embodiments, based on the motion trajectory, the target user's eye data and reading behavior are analyzed to obtain analysis results, including:

[0264] Calculating the target user's eye gaze point and eye gaze duration based on the motion trajectory;

[0265] Identifying the target user's eye movements and gaze-returning behaviors;

[0266] generating a heat map and a reading path based on the saccadic behavior and the reversion behavior;

[0267] Extracting the target user's reading behavior characteristics based on the heat map and reading path;

[0268] Identifying the target user's reading pattern through cluster analysis based on the reading behavior characteristics;

[0269] Based on the reading pattern, a personalized reading analysis system is established.

[0270] Optionally, the target user's reading behavior characteristics include reading speed, concentration, etc.

[0271] Optionally, a personalized reading analysis system can be established to make intelligent content recommendations for target users, and reading assistance tools (e.g., automatic adjustment of font size, line spacing, etc.) can be used to improve the target users' reading experience.

[0272] The disclosed embodiment can not only adapt to different lighting environments and maintain high-precision eye tracking, but also analyze the reader's reading behavior in real time, providing strong support for personalized reading experience and content recommendation.

[0273] Optionally, the hardware integration method of the embodiment of the present disclosure includes: selecting a suitable eye tracker and processor, designing a low-latency data transmission solution, and implementing hardware acceleration (such as GPU or FPGA).

[0274] Optionally, the software development of the embodiment of the present disclosure includes: implementing a real-time image acquisition and preprocessing module, integrating a deep learning model inference engine, and developing a decision tree real-time prediction module.

[0275] Optionally, the model optimization of the embodiment of the present disclosure is achieved through parallel computing and multi-threaded optimization, and also includes a cache mechanism and optimization through adaptive sampling rate adjustment and other methods.

[0276] The disclosed embodiments describe in detail how to use the Vision Transformer adaptive tokens on a spiking neural network to extract eye features to adapt to different lighting conditions.

[0277] Step A1: Construct a spiking neural network.

[0278] Step A1.1: Define the spiking neuron model.

[0279] Step A1.1.1. Implement the Leaky Integrate-and-Fire (LIF) neuron model.

[0280] Step A1.1.2: Set the threshold and reset mechanism of the LIF neuron model;

[0281] Step A1.1.3: Define a pulse encoding function to convert the input eye image into a pulse sequence.

[0282] Step A1.2: Build a spiking neural network layer, implement a spiking convolutional layer, implement a spiking pooling layer, and implement a spiking fully connected layer.

[0283] Step A2: Design the Vision Transformer architecture.

[0284] Step A2.1, divide the eye image into blocks and embed. Specifically, divide the input eye image into blocks of fixed size, perform linear projection on each image block to convert it into a D-dimensional vector, and embed the image block; add position coding information to the image block.

[0285] Step A2.2: Design a Transformer encoder, implement a multi-head self-attention mechanism, implement a feedforward neural network, and perform normalization and residual connection processing in the application layer after each Transformer encoder layer.

[0286] Step A3: Implement an adaptive token mechanism.

[0287] Step A3.1, initialize the adaptive token;

[0288] Step A3.1.1. Randomly initialize a set of learnable tokens;

[0289] Step A3.1.2: Design a token renewal strategy.

[0290] Step A3.1.3: Implement the fusion mechanism of token and input embedding;

[0291] Step A3.2: Adapt to light conditions.

[0292] Step A3.2.1. Design a light condition encoder.

[0293] Step A3.2.2, injecting light condition information into the adaptive token;

[0294] Step A3.2.3: Implement dynamic adjustment of the adaptive token to different lighting conditions.

[0295] Step A4: Eye feature extraction. Specifically, design a feature extraction head, map the output of the Transformer encoder to the eye feature space, define key eye features (such as pupil center, eye corner position, etc.), and implement a feature point positioning algorithm.

[0296] Step 5: Design a loss function. Specifically, define the eye feature point positioning loss and add a regularization term to prevent overfitting. Design a light adaptability loss to encourage the eye tracking model to remain stable under different lighting conditions.

[0297] Step 6: Optimize the training strategy. Specifically, implement progressive learning to gradually increase the complexity of light changes; adopt a contrastive learning strategy to improve the ability to discriminate key eye features; and use knowledge distillation technology to extract key knowledge from the large model.

[0298] The disclosed embodiments construct an eye feature extraction model that can adapt to different lighting conditions. This model utilizes the temporal dynamic characteristics of spiking neural networks and the powerful representation capabilities of the Vision Transformer architecture, combined with an adaptive token mechanism, to achieve accurate capture and extraction of eye features. This lays the foundation for how to apply advanced deep learning technology to the field of eye tracking and for subsequent high-precision eye movement analysis.

[0299] It should be noted that, in practical applications, the embodiments of the present disclosure may further optimize and adjust the model according to specific needs, and the embodiments of the present disclosure do not limit this.

[0300] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an adaptive eye tracking device based on deep learning provided by an embodiment of the present disclosure. The present disclosure also provides an adaptive eye tracking device based on deep learning, including:

[0301] an acquisition unit 21, configured to acquire eyeball images under various lighting conditions;

[0302] a preprocessing unit 22 configured to preprocess the eyeball image to obtain a preprocessed eyeball image;

[0303] a model training unit 23 configured to construct an initial eye tracking model based on a spiking neural network, and train the initial eye tracking model to obtain a trained eye tracking model;

[0304] an input / output unit 24 configured to input the preprocessed eye image into the trained eye tracking model and obtain a motion trajectory of the eye in the eye image output by the trained eye tracking model;

[0305] The analyzing unit 25 is configured to analyze the eye data and reading behavior of the target user based on the motion trajectory to obtain an analysis result.

[0306] A computer device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc.

[0307] The processor can be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and can control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the deep learning-based adaptive eye tracking method described in various embodiments of the present disclosure.

[0308] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0309] like Figure 6 The present invention provides a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 6 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0310] like Figure 6As shown, a computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) or programs loaded from a storage device into a random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0311] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes and hard disks; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 6 A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0312] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of a deep learning-based adaptive eye tracking method of an embodiment of the present disclosure are performed.

[0313] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0314] According to an embodiment of the present disclosure, a computer-readable storage medium stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the deep learning-based adaptive eye tracking method described in each embodiment of the present disclosure are performed.

[0315] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0316] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0317] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0318] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0319] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0320] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0321] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0322] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0323] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An adaptive eye tracking method based on deep learning, characterized in that: include: Acquire eye images under various lighting conditions; Preprocessing the eyeball image to obtain a preprocessed eyeball image; Building an initial eye tracking model based on a spiking neural network, and training the initial eye tracking model to obtain a trained eye tracking model; Inputting the preprocessed eye image into the trained eye tracking model, and obtaining a motion trajectory of the eye in the eye image output by the trained eye tracking model; Based on the movement trajectory, analyzing the target user's eye data and reading behavior to obtain analysis results; Build an initial eye tracking model based on a spiking neural network, including: Define the spiking neuron model; Based on the spiking neuron model, build the Vision Transformer architecture; Initialize several learnable tokens, where each token is a D-dimensional vector; During each forward propagation, adaptively adjusting the plurality of learnable tokens according to the light conditions of the input eye image; Concatenating the adjusted number of learnable tokens with the image patch embedding of the Vision Transformer architecture; The concatenated image block embedding and the plurality of learnable tokens are input into the Transformer encoder of the Vision Transformer architecture to obtain the initial eye tracking model.

2. The deep learning-based adaptive eye tracking method according to claim 1, characterized in that Acquire eye images under various lighting conditions, including: Set the image acquisition environment with different light source intensities or angles; The eye movements of a plurality of sampled users under various lighting conditions in the image acquisition environment are captured by a photographing device; wherein the plurality of sampled users have different ages and / or eye features.

3. The adaptive eye tracking method based on deep learning according to claim 1, characterized in that Preprocessing the eyeball image to obtain a preprocessed eyeball image includes: Performing denoising on the eyeball image to obtain a denoised eyeball image; Performing enhancement processing on the denoised eyeball image to obtain an enhanced eyeball image; Detecting the position and boundary of the eyeball region in the enhanced eyeball image, and performing segmentation processing on the eyeball region according to the detected position and boundary; The segmented eyeball region is subjected to standardization and normalization to obtain a pre-processed eyeball image.

4. The deep learning-based adaptive eye tracking method according to claim 1, wherein: Training the initial eye tracking model to obtain a trained eye tracking model includes: Acquire a data set of eyeball image samples of several sampled users, wherein the data set of eyeball image samples includes eyeball image samples under various lighting conditions; Inputting the eyeball image samples under the multiple light conditions into the initial eyeball tracking model, and obtaining motion trajectory samples of the eyeball samples output by the initial eyeball tracking model in the eyeball image samples; Calculating the loss between the motion trajectory sample and the target motion trajectory using a defined loss function to obtain a loss value; Based on the loss value, optimizing and adjusting relevant model parameters of the initial eye tracking model through an optimizer until the loss value is less than or equal to a preset threshold, and outputting a trained eye tracking model; The loss function is expressed as follows: L total = a * L position + β * L gaze + c * L adaptive + λ * L reg ; Where, L position Represents the MSE loss of eye feature point positioning, which includes eye corners and pupil centers; L gaze Represents the angular error of the line of sight direction prediction; L adaptive represents the consistency loss of the adaptive token; L reg Represents the regularization term to prevent overfitting; α is L position The weight of L gaze The weight of L adaptive The weight of L reg The weight of .

5. The deep learning-based adaptive eye tracking method according to claim 1, wherein: Based on the movement trajectory, the target user's eye data and reading behavior are analyzed to obtain analysis results, including: Calculating the target user's eye gaze point and eye gaze duration based on the motion trajectory; Identifying the target user's eye movements and gaze-returning behaviors; generating a heat map and a reading path based on the saccadic behavior and the reversion behavior; Extracting the target user's reading behavior characteristics based on the heat map and reading path; Identifying the target user's reading pattern through cluster analysis based on the reading behavior characteristics; Based on the reading pattern, a personalized reading analysis system is established.

6. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based adaptive eye tracking method described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the deep learning-based adaptive eye tracking method described in any one of claims 1 to 5.

8. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images