Eye movement behavior classification method based on Gaussian mixture model and hidden Markov model

Abstract

The present invention relates to an eye movement behavior classification method based on a Gaussian mixture model and a hidden Markov model, and belongs to the field of human-computer interaction technology. The hardware system includes a desktop eye tracker, a host computer, and a display screen; the host computer has a Gaussian mixture model program and a hidden Markov model program; the eye movement behavior classification operation steps are as follows: (1) collecting the original eye movement data of the measured person; (2) preprocessing the original eye movement data, using a clustering method to group the original eye movement data into three or more groups of eye movement data; (3) establishing a Gaussian mixture model and a hidden Markov model; (4) based on the Gaussian mixture model and the hidden Markov model, classifying the eye movement behavior of the three or more preprocessed groups of eye movement data, and inferring whether the current eye movement behavior is fixation, saccade, or tracking eye movement behavior. The classification method of the present invention improves classification accuracy, reduces complexity, and has significant advantages in adapting to practical applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and in particular relates to a high-accuracy eye movement behavior classification method. Background Art

[0002] Eye tracking technology monitors and records individual eye movements using specialized equipment, such as an eye tracker. It aims to precisely capture spatial and temporal changes in eye movements to analyze and understand an individual's visual behavior during specific tasks or situations. As a cutting-edge human factors analysis technology, it not only reveals the complexity of the visual system but also has broad applications across multiple fields. In user experience research, it helps designers gain a deep understanding of users' attentional focus and visual preferences during interactions, providing substantial data support for optimizing product interfaces. In medical research, eye tracking has become a valuable tool for studying cognitive function and diagnosing diseases. In ophthalmology, eye movement analysis can effectively assess an individual's visual state. In traffic safety, eye tracking is widely used in driving behavior research. By monitoring a driver's gaze, it assesses their attention to road conditions and supports improvements in intelligent driving systems.

[0003] Despite the broad application prospects of eye tracking technology in various fields, effectively classifying eye movement data has always been a major research challenge. The ternary eye movement classification task aims to classify eye movement behaviors into three main types: fixation, saccade, and smooth pursuit. These three eye movement types reflect the different movement states of the human eye when observing visual scenes, each with its own unique characteristics and functions. Existing eye movement classification methods have some significant shortcomings, including but not limited to the following aspects: 1) Lack of robustness in uncontrolled test subject environments. In test subject environments where the test subject cannot be controlled in advance, eye movement data is more diverse and difficult to predict. Existing algorithms lack robustness and adaptability in dealing with these uncertainties, which affects classification performance. 2) Inconsistent classification of specific eye movement behaviors. Existing methods have confusion and errors in classifying the three typical eye movement behaviors, resulting in inconsistent performance of the classifier in distinguishing these different behaviors, reducing the accuracy of interpreting eye movement data.

[0004] Methods for eye movement classification are primarily categorized into threshold and probabilistic methods. Common threshold methods include velocity threshold detection (I-VT) and dispersion threshold detection (I-DT). The velocity threshold method distinguishes between fixations and rapid saccades by setting a threshold based on the velocity of a sample point. Samples below the threshold are labeled as fixations, while those above the threshold are labeled as saccades. The dispersion threshold method, on the other hand, calculates the dispersion of the coordinate information of sample points within a certain interval and uses the dispersion threshold to separate fixations from saccades. Building on these binary classification algorithms, the velocity-velocity threshold detection (I-VVT) method has emerged, employing two velocity thresholds for triadic eye movement classification. Furthermore, the I-VDT algorithm combines these two thresholds, using velocity thresholds to detect saccades and adjusting the dispersion to distinguish between fixations and smooth pursuits, classifying each movement type within subsegments based on its unique characteristics. However, these methods are limited by their reliance on statistical data. The threshold determination is constrained by the size and quality of the data, resulting in limited adaptability in practical applications. In addition to the threshold method, there is also a class of probabilistic algorithms, among which the hidden Markov model (HMM) is a common representative. HMM recognition uses probabilistic analysis to infer the most likely eye movement pattern and characterizes the probability distribution of gaze shifts through bimodal HMM. The I-BDT algorithm based on Bayesian decision theory calculates the posterior classification probability through prior and likelihood information, and has the advantage of online recognition. However, the performance of I-BDT depends on the accuracy of prior and likelihood information. Inaccurate prior information or data distribution deviation from the likelihood assumption may lead to errors. The I-VDT-HMM fusion algorithm is built on the basis of the I-VDT concept and uses speed and discreteness as feature parameters to hierarchically train the HMM model. From the perspective of model training, this method does not fully realize the feature fusion of data classification. Summary of the Invention

[0005] In order to achieve high-accuracy and high-robustness ternary eye movement classification, the present invention relates to a method and system for eye movement behavior recognition and classification, in particular a method and system suitable for commercial-grade desktop eye trackers.

[0006] The method of the present invention combines a Gaussian mixture model with a hidden Markov model and adopts a hierarchical training method to achieve efficient and accurate classification of eye movement data.

[0007] An eye movement behavior classification method based on a Gaussian mixture model and a hidden Markov model, wherein the hardware system for implementing the eye movement behavior classification method includes a desktop eye tracker, a host computer, and a display screen; the host computer is a host computer, and the host computer has a Gaussian mixture model program and a hidden Markov model program;

[0008] The steps for eye movement behavior classification are as follows:

[0009] (1) Collecting the original eye movement data of the test subjects (1.1) When the hardware system is working, a red dot with a diameter of 80 pixels is displayed on the screen. The red dot is located at any position on the screen and the duration of each appearance is 500 to 1000 milliseconds;

[0010] (1.2) The subject sits in front of a display screen and clicks a start button on the screen with a mouse. A red dot visual stimulus appears on the screen three or more times in a regular pattern. The visual stimulus includes three types of eye movement behaviors to be identified: fixation, tracking, and saccades.

[0011] (1.3) During the operation of the hardware system, the desktop eye tracker continuously collects the position coordinates and timestamps of the subject's gaze point on the display screen, and transmits them to the host computer. The host computer processes them to obtain raw eye movement data. Each piece of data in the raw eye movement data consists of the pixel coordinates of the gaze point and the velocity of the gaze point;

[0012] (2) Raw eye movement data preprocessing

[0013] Performing spatial feature preprocessing on the raw eye movement data, using a clustering method to group the pixel coordinates of the gaze point and the velocity of the gaze point as a whole into at least three groups, each group of eye movement data having no less than fifty pieces, to obtain at least three groups of preprocessed eye movement data;

[0014] (3) Establishing Gaussian mixture model and hidden Markov model

[0015] (3.1) Establishing Gaussian mixture model

[0016] (3.1.1) performing statistical analysis on the three or more pre-processed eye movement data sets, and based on the similarity in velocity among the three or more pre-processed eye movement data sets, i.e., when the velocity difference between any two data points is less than a preset threshold of 0.002, the two data points are considered similar and are classified into the same cluster;

[0017] The three or more groups of eye movement data for statistical analysis were divided into three clusters according to the speed difference. Each cluster corresponds to a predefined eye movement behavior, namely the gaze Gaussian distribution model, the pursuit Gaussian distribution model and the saccade Gaussian distribution model.

[0018] Initialize three predefined eye movement behaviors. For the gaze Gaussian distribution model, take the center position of the cluster with the lowest average speed as the center position of the gaze Gaussian distribution model, set the initial weight of the gaze Gaussian distribution to 1 / 3, and set the initial covariance matrix of the gaze Gaussian distribution to the identity matrix.

[0019] For the saccade Gaussian distribution model, the center position of the cluster with the largest average speed is used as the center position of the saccade Gaussian distribution model. Similarly, the initial weight of the saccade Gaussian distribution is set to 1 / 3, and the initial covariance matrix of the saccade Gaussian distribution is set to the identity matrix.

[0020] For the tracking Gaussian distribution model, the center position of the remaining third cluster is used as the center position of the tracking Gaussian distribution model, the initial weight of the tracking Gaussian distribution is set to 1 / 3, and the initial covariance matrix of the tracking Gaussian distribution is set to the identity matrix;

[0021] The initial Gaussian mixture model is composed of the gaze Gaussian distribution model, the pursuit Gaussian distribution model and the saccade Gaussian distribution model;

[0022] (3.1.2) Adjusting the parameters of the initialized Gaussian mixture model through iterative optimization to better fit the actual probability distribution of each eye movement behavior, with the number of iterations set to at least 50; and determining the degree of optimization of the Gaussian mixture model by introducing a Q function to obtain an optimized Gaussian mixture model;

[0023] (3.2) Establishing a Hidden Markov Model (3.2.1) Initializing the three parameters of the Hidden Markov Model:

[0024] Initial probability matrix π=[π1,π2,…,π N ], represents the initial probability of each eye movement behavior at the initial moment, and the initial probability of each eye movement behavior is set to 1 / 3;

[0025] The transition probability matrix A = [a ij ], represents the transition probability from one eye movement behavior to another, ensuring that the sum of each row element of the transition probability matrix A is 1;

[0026] Emission probability matrix B = [b j (o t )], which represents the probability that each data in the eye movement data belongs to one of the three eye movement behaviors. Each row element of the emission probability matrix B is determined by the parameters of the Gaussian mixture model in (3.1.2);

[0027] By initializing the above three parameters of the hidden Markov model, an initialized hidden Markov model is obtained;

[0028] (3.2.2) Optimizing the initialized hidden Markov model using the three or more pre-processed eye movement data using an optimization algorithm to obtain an optimized hidden Markov model;

[0029] (4) Classification of eye movement behavior

[0030] Based on the Gaussian mixture model optimized in step (3.1.2) and the hidden Markov model optimized in step (3.2.2), the eye movement behavior of the three or more preprocessed eye movement data is classified; based on the eye movement data at each time point, it is inferred whether the current eye movement behavior is fixation, saccade, or tracking eye movement behavior.

[0031] Further technical solutions are as follows:

[0032] The desktop eye tracker is connected to the host computer via a universal serial bus; the display screen is larger than 13.3 inches and is located at least 75 centimeters away from the person being measured; the display screen is connected to the host computer via a universal serial bus.

[0033] The host computer connected to the desktop eye tracker and the display screen is equipped with a central processing unit with a basic frequency of more than 2.0 GHz and an operating system of Windows 10 64-bit operating system.

[0034] In step (1.2), the number of fixations on the visual stimulus of the red dot is 3 times, 5 times, or more than 10 times. The subject completes the fixation task within the set number of fixations, and the interval between two consecutive fixation stimuli is 0.5 seconds or 1 second. Within the range of the display screen, the host computer randomly generates the pixel coordinates of the fixation position.

[0035] In step (1.2), the gaze refers to an eye movement behavior in which the gaze stays in any area for at least 500 milliseconds, indicating that the subject is interested in the content of the area or is processing information;

[0036] Tracking refers to the eye movement behavior of smoothly following a moving target, including tracking moving objects or text on a display screen.

[0037] Saccades refer to eye movements in which the line of sight quickly jumps from one fixation point to another. They are fast, short eye movements that usually occur between fixations and are used to shift visual attention from one area to another.

[0038] In step (2), the original eye movement data are grouped using a clustering method. The specific operations are as follows:

[0039] (2.1) Clustering algorithm:

[0040] a) Initialize cluster centers:

[0041] Randomly select K data from the original eye movement data as the initial clustering centers, and divide the original eye movement data into K eye movement data groups, where the value of K is at least 4;

[0042] b) Assign data to the nearest cluster center:

[0043] Calculate the distance between each original eye movement data in the K eye movement data groups and the K cluster centers, assign the original eye movement data closest to a cluster center to the nearest eye movement data group, and obtain K new eye movement data groups;

[0044] c) Update cluster centers:

[0045] Calculating the data average of each of the K new eye movement data groups, and using the data average as a new cluster center;

[0046] d) Iterative processing:

[0047] Repeat steps b) and c) until a predetermined number of iterations of 50 is reached; obtaining K final cluster centers and K final eye movement data sets;

[0048] (2.2) Elbow rule determines the optimal number of groups:

[0049] Apply the elbow rule to determine the optimal number of groups. The specific steps are as follows:

[0050] Calculate the sum of squared errors (Sum of Squared Errors) of the clustering results of the K final cluster centers. The sum of squared errors formula (1) is as follows:

[0051]

[0052] In formula (1), k is the number of cluster centers, n i is the number of data points in the i-th group, is the jth data point in the i-th group, c i is the cluster center of the i-th group;

[0053] When the number of groups k gradually increases, the error sum squared SSE(k) usually shows a decreasing trend, because more groups can better fit the distribution of the data and reduce the distance between the data point and the cluster center of the group to which it belongs. After a certain number of groups k, the decrease in the error sum squared SSE(k) will slow down significantly and form an inflection point. The inflection point represents the optimal balance between clustering performance and the number of groups. Based on the trend that the error sum squared of the clustering results decreases as the number of groups k increases, the number of cluster centers k corresponding to the point where the error sum squared SSE(k) decreases is selected to determine the optimal number of groups. For the k final cluster centers, the preprocessed k-1 groups of eye movement data are obtained.

[0054] In step (3.1.2), the operation of adjusting the Gaussian mixture model parameters by the iterative optimization method is as follows:

[0055] a). Initial parameter definition of Gaussian mixture model

[0056] The formula (2) of the Gaussian mixture model is as follows:

[0057]

[0058] In formula (2), P(x) represents the probability of observing eye movement data x, k represents the kth eye movement behavior, which is gaze, tracking, and saccade in sequence, c k represents the weight of the k-th eye movement behavior, N(x|μ k ,∑ k ) represents the probability density function of the Gaussian distribution of the k-th eye movement behavior, μ k and ∑ k Represent the mean and covariance matrix of the k-th Gaussian distribution respectively;

[0059] Taking the three eye movement behaviors of fixation, tracking, and saccade as examples, the probability density function (3) of the Gaussian mixture model is as follows:

[0060] P(x)=c1·N(x|μ1,∑1)+c2·N(x|μ2,∑2)+c3·N(x|μ3,∑3)(3)

[0061] In formula (3), the weights c1, c2, c3, the mean values ​​μ1, μ2, μ3 and the covariance matrices ∑1, ∑2, ∑3 are all parameters to be optimized in the Gaussian mixture model;

[0062] b). Calculation of responsibility

[0063] To adjust the Gaussian mixture model parameters, it is necessary to calculate each eye movement data x n The probability of the three eye movement behaviors of fixation, tracking and saccade is called responsibility γ(z nk ), responsibility γ(z nk ) is calculated as follows:

[0064]

[0065] In formula (4), x n represents the nth observed eye movement data, z nk is a symbolic variable, indicating that the nth eye movement data belongs to the kth Gaussian distribution;

[0066] c). Update Gaussian mixture model parameters

[0067] Responsibility for use γ(z nk ) to update the parameters of the Gaussian mixture model, which include the weight c k , average value μ k and covariance matrix ∑ k ;

[0068] Weight c k The update formula (5) is as follows:

[0069]

[0070] Average value μ k The update formula (6) is as follows:

[0071]

[0072] Covariance matrix ∑ k The update formula (7) is as follows:

[0073]

[0074] Repeat the calculation of formula (5)-(7) at least 50 times to update the parameters of the Gaussian mixture model;

[0075] In order to determine the degree of optimization of the Gaussian mixture model, the Q function is introduced. The Q function formula (8) is as follows:

[0076]

[0077] In formula (8), θ represents the Gaussian mixture model parameters in the current iteration, including the mixing coefficients c of each Gaussian distribution k , mean μ k and covariance matrix ∑ k ,θ (t) represents the Gaussian mixture model parameters in the previous iteration, P(x i ,z ik |θ) Under the current model parameters θ, the eye movement data x i The probability of belonging to the kth Gaussian distribution, P(z ik |x i ,θ (t) ) represents the model parameter θ in the previous iteration (t) Next, eye movement data x i The probability of belonging to the kth Gaussian distribution;

[0078] The Q function measures the degree of model optimization by comparing the parameters of the current Gaussian mixture model with the parameters of the previous iteration. The larger the value of the Q function, the greater the improvement of the current model parameters compared with the previous iteration. When the value of the Q function is less than the set threshold of 0.001, or the number of calculations reaches the preset number of repetitions, the parameter optimization of the Gaussian mixture model is completed, and the parameters of the Gaussian mixture model at this time, including the weight c, are recorded. k , average value μ k and covariance matrix ∑ k , that is, the optimized Gaussian mixture model is obtained.

[0079] In step (3.2.2), the method for optimizing the parameters of the hidden Markov model is as follows:

[0080] a). Define and calculate forward probability

[0081] The observation state sequence of eye movement behavior is defined as O = (o1, o2, ..., o T ), the above sequence represents the eye movement behavior observed from the initial moment to the last moment. In the process of optimizing the hidden Markov model, the forward probability α is first calculated. t (i), the forward probability is calculated as follows (9):

[0082]

[0083] Formula (9) indicates that the state at time t is i, and the observation sequence of eye movement behavior is (o1, o2,…, o t ), in formula (9), a ji represents the transition probability from state j at time t-1 to state i at time t, b i represents the emission probability of state i, a ji and b i are all parameters of the hidden Markov model;

[0084] b) Define and calculate backward probability

[0085] Calculate the backward probability β t (i), the backward probability is calculated as follows (10):

[0086]

[0087] Formula (10) indicates that the state at time t is i, and the observation sequence of eye movement behavior is (o t+1 ,o t+1 ,…,o t ), in formula (10), a ij represents the transition probability from state j at time t+1 to state i at time t, b j represents the emission probability of state j, a ij and b j are all parameters of the hidden Markov model;

[0088] c). Update the parameters of the hidden Markov model

[0089] Using the forward probability and the backward probability, calculate the prior probability γ t (i) and the joint probability ξ t (i, j) is used to update the parameters of the hidden Markov model, where the prior probability γ t(i) represents the probability of state i at time t, and the joint probability ξ t (i, j) represents the probability that the state is i at time t and the state is j at time t+1, and the prior probability γ t The calculation formula (11) for (i) is as follows:

[0090]

[0091] Joint probability ξ t The calculation formula (12) of (i, j) is as follows:

[0092]

[0093] In formulas (11) and (12), the transition probability a ij represents the probability of transitioning from state i to state j, and the emission probability b j (o t+1 ) indicates that the observation value in state j is o t+1 The probability of ; by using the prior probability γ t (i) and the joint probability ξ t (i, j) is used to update the three parameters of the hidden Markov model, the initial probability matrix π, the transition probability matrix A, and the emission probability matrix B. The specific update operation is as follows:

[0094] The update formula (13) of the initial state probability vector π is as follows:

[0095] π i =γ1(i) (13)

[0096] In formula (13), π i Represents the i-th element of the optimized initial state probability vector;

[0097] The update formula (14) of the state transition probability matrix A is as follows:

[0098]

[0099] In formula (14), a ij Represents the element in the i-th row and j-th column of the optimized state transition probability matrix;

[0100] The update formula (15) of the emission probability matrix B is as follows:

[0101]

[0102] In formula (15), b j (k) represents the jth row and kth column element of the optimized emission probability matrix, v k represents the kth possible observation value;

[0103] By repeating the calculation of formulas (13)-(15) at least 100 times, the three parameters of the above hidden Markov model are optimized so that the hidden Markov model can continuously fit the eye movement data. When the preset number of repetitions is reached, the hidden Markov model is considered to have been optimized, and the optimized hidden Markov model is finally obtained.

[0104] In step (4), the eye movement behavior classification is performed on the pre-processed three or more groups of eye movement data to obtain the eye movement behavior at each moment. The specific operation is as follows:

[0105] (4.1) Determine the optimal path probability at each moment

[0106] Assume that the hidden state sequence of eye movement behavior is S=(s1,s2,...,s T ), the hidden state of the eye movement behavior represents the actual eye movement behavior, which is one of the eye movement behaviors of fixation, pursuit and saccade, and the sequence S represents all the actual eye movement behaviors from the initial moment to the final moment;

[0107] Assume that the observation state sequence of eye movement behavior is O=(o1,o2,…,o T ), the observed state of the eye movement behavior represents the eye movement behavior estimated based on the observed eye movement data, which is also an eye movement behavior among fixation, pursuit and saccade, and the sequence O represents all the eye movement behaviors estimated based on the observed eye movement data from the initial moment to the final moment;

[0108] In order to obtain the hidden state sequence with the highest probability, that is, to obtain the most likely eye movement behavior, the optimal sequence function is defined as δ t (i), optimal sequence function δ t (i) indicates that when the hidden state is i at time t, the observed state is (o1, o2, ..., o t ), the maximum probability of the optimal sequence function δ t The specific value of (i) is obtained by recursive calculation, and the optimal sequence function δ t The recursive formula (16) of (i) is as follows:

[0109]

[0110] In formula (16), a ij represents the transition probability from state i to state j, Indicates that o is observed in state j t The emission probability, δ t-1 (i) represents the probability of state i at time t-1.

[0111] By recursively formula (16), the maximum probability δ corresponding to each hidden state i is calculated at each time t t(i); (4.2) Determine the optimal precursor state at each moment

[0112] Get the maximum probability δ of the hidden state at time t t (i) Based on this, we need to record the most likely previous state of the hidden state i at time t, that is, the optimal predecessor state at time t; define the optimal predecessor state ψ t (i), record the most likely previous eye movement behavior of state i at time t, the optimal precursor state ψ t Formula (17) of (i) is as follows:

[0113]

[0114] Formula (17) records the optimal precursor state at each moment t, and recursively finds the hidden state from the final moment to the initial moment, thereby classifying the eye movement behavior at each time point;

[0115] (4.3) Classification of eye movement behavior

[0116] Using the optimal path probability δ t (i) and the optimal precursor state ψ t (i) Calculate the maximum probability P of the hidden state sequence * and the terminal state of the hidden state sequence Classify eye movement behavior;

[0117] The maximum probability P of the hidden state sequence * Indicates that at the final moment T, the optimal path probability δ corresponding to the three hidden states of gaze, pursuit and saccade is calculated respectively t (i) And select the optimal path probability value with the largest value as the maximum probability P of the hidden state sequence * , the maximum probability P * The calculation formula (18) is as follows:

[0118]

[0119] In formula (18), N represents the number of hidden states, and the upper limit here is 3;

[0120] The maximum probability P of the hidden state sequence at the final moment T is obtained * On this basis, it is also necessary to determine the terminal state of the hidden state sequence Terminal State The calculation formula (19) is as follows:

[0121]

[0122] When the termination status is obtained When , we start to recurse from the final moment T to the initial moment to obtain the hidden state at each moment; the recursive formula (20) is as follows:

[0123]

[0124] In formula (20), represents the optimal hidden state at time t, represents the optimal hidden state at time t+1 The most likely previous state corresponding to the hidden state is obtained; the hidden state of each moment of the entire hidden state sequence is determined by formula (20), that is, the eye movement behavior from the initial moment to the end moment is classified; it is inferred that the current eye movement behavior belongs to which type of gaze, saccade or tracking eye movement behavior, and the eye movement behavior classification is completed.

[0125] The three types of eye movement behaviors are specifically classified as follows:

[0126] Attention: When the probability δ corresponding to the hidden state t (i) When it exceeds 0.9 and the state lasts for more than 200 milliseconds, it indicates that the current eye movement behavior is fixation;

[0127] Eye saccade: When the hidden state changes within 50 milliseconds, and the corresponding transition probability a ij If it is greater than 0.8, it indicates that the current eye movement behavior is saccade;

[0128] Tracking: When the hidden state changes within 500 milliseconds, and the corresponding transition probability a ij If it exceeds 0.9, it indicates that the current eye movement behavior is tracking.

[0129] The beneficial technical effects of the present invention are embodied in the following aspects:

[0130] 1. The present invention adopts an algorithm that combines a Gaussian Mixture Model (GMM) with a Hidden Markov Model (HMM). GMM is a mathematical model composed of multiple Gaussian distributions, each of which represents a subgroup or mode in the data. This multimodal modeling feature enables GMM to better adapt to the different distribution states of multiple eye movement behaviors in the data. Mathematically, GMM models the probability distribution of the data and determines the parameters of each Gaussian distribution through maximum likelihood estimation, thereby achieving reasonable modeling of the data. HMM is a statistical model for modeling sequence data, which performs probabilistic analysis on state transitions in the sequence. In eye movement classification, HMM can be used to capture state changes between eye movement patterns, thereby more accurately inferring the relationship between different eye movement behaviors. Mathematically, HMM consists of a state set, an observation set, a state transition probability matrix, and an observation probability matrix. By adjusting and optimizing these elements, HMM can better reflect the dynamic changes of eye movement behavior. By combining these two approaches, the algorithm can more flexibly adapt to multimodal eye movement behavior and more deeply model the relationships between different eye movement behaviors through probabilistic analysis. This combination of mathematical models provides a more accurate and comprehensive tool for eye movement classification, thus overcoming the limitations of existing methods.

[0131] 2. A clustering algorithm is used to finely segment the eye movement path data, dividing the eye movement path into different subsegments based on the position and velocity characteristics of the original eye movement path. By calculating the sum of squared errors (SSE) of clustering for different numbers of segments, the optimal number of segments is selected, corresponding to the point where the SSE decreases most slowly. This method strikes a balance between clustering performance and computational complexity, avoiding overfitting or underfitting. This method effectively captures local features in the eye movement data, providing more granular information for subsequent eye movement behavior classification.

[0132] 3. Compatible with commercial-grade desktop eye trackers: This invention was designed with the characteristics of commercial-grade desktop eye trackers in mind, resulting in strong real-time performance. Traditional eye movement behavior classification methods may require extensive eye movement data for model pre-training or rely on extensive statistical data to determine appropriate thresholds to ensure the model can better understand the characteristics of the eye movement data. Compared to traditional methods, this invention does not require complex pre-training or model building, thereby reducing the burden of initial data processing.

[0133] 4. High Accuracy and Stability: The introduction of a hidden Markov model to model eye movement behavior in a temporal manner and the adoption of a hierarchical modeling approach significantly improves the accuracy of eye movement classification. This invention employs a path segmentation strategy, combined with the elbow rule to segment eye movement paths, extract key eye movement data features, and then performs a two-layer classification using a Gaussian hidden Markov model on the segmented paths. The first layer features coordinate information, while the second layer primarily features velocity information, enabling accurate classification of eye movement behaviors such as gaze, saccade, and smooth pursuit. Experiments have shown that this invention outperforms traditional algorithms in terms of accuracy, precision, and recall, and is particularly stable when processing different eye movement behaviors. Compared to traditional algorithms such as I-BDT and I-VDT, the proposed method performs more consistently, with less variability in both precision and recall. The proposed GMM-HMM algorithm achieved an accuracy of 94.39% ± 2.08%, a precision of 95.31% ± 3.71%, and a recall of 94.98% ± 3.93% in the classification of three types of eye movement behaviors. For gaze behavior, the algorithm achieved a precision of 0.9743, a recall of 0.9665, and an F1 score of 0.9699, demonstrating a balanced approach in accurately identifying true location events while minimizing false positives and false negatives. For tracking behavior classification, the algorithm achieved a precision of 0.8784, a recall of 0.9076, and an F1 score of 0.8893, demonstrating its ability to effectively handle complex tracking behaviors. For saccade classification, the proposed Gaussian Hidden Markov Model (GMM) algorithm achieved a high accuracy of 0.9301, outperforming the I-BDT algorithm.

[0134] The specific data and performance of these beneficial technical effects indicate that the eye movement behavior classification algorithm of the present invention has achieved significant advantages in improving classification accuracy, reducing complexity, and adapting to practical applications, providing strong support for progress in the field of eye movement data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0135] Figure 1 This is a flowchart of the visual stimulation program executed by the host computer in an embodiment of the present invention;

[0136] Figure 2 This is a segmentation result diagram of selecting three cluster centers for the acquired eye movement sequence in Example 1 of the present invention;

[0137] Figure 3 This is a segmentation result diagram of selecting four cluster centers for the acquired eye movement sequence in Example 1 of the present invention;

[0138] Figure 4 This is a segmentation result diagram of selecting 5 cluster centers for the acquired eye movement sequence in Example 1 of the present invention;

[0139] Figure 5 is the SSE value of the eye movement sequence under different clustering numbers in Example 1 of the present invention, which is used to determine the optimal segmentation method diagram;

[0140] Figure 6 This is a visualization diagram of the original eye movement trajectory in Example 1 of the present invention;

[0141] Figure 7 This is a visualization result diagram of the ternary eye movement classification performed on the original eye movement trajectory using the present invention in Example 1 of the present invention;

[0142] Figure 8 is a classification result diagram of the x-axis of the eye movement trajectory in Example 1 of the present invention;

[0143] Figure 9 is a classification result diagram of the y-axis of the eye movement trajectory in Example 1 of the present invention;

[0144] Figure 10 This is a segmentation result diagram of selecting 9 cluster centers for the acquired eye movement sequence in Example 2 of the present invention;

[0145] Figure 11 is the SSE value of the eye movement sequence under different clustering numbers in Example 2 of the present invention, which is used to determine the optimal segmentation method diagram;

[0146] Figure 12 This is a visualization diagram of the original eye movement trajectory in Example 2 of the present invention;

[0147] Figure 13 This is a visualization result diagram of the ternary eye movement classification performed on the original eye movement trajectory using the present invention in Example 2 of the present invention;

[0148] Figure 14 is a classification result diagram of the x-axis of the eye movement trajectory in Example 2 of the present invention;

[0149] Figure 15 This is a classification result diagram of the y-axis of the eye movement trajectory in Example 2 of the present invention.

[0150] Attached specific implementation

[0151] The present invention will be further described below with reference to the accompanying drawings through embodiments. Example

[0152] An eye movement behavior classification method based on a Gaussian mixture model and a hidden Markov model, and a hardware system for implementing the eye movement behavior classification method include a desktop eye tracker, a host computer and a display screen.

[0153] The host computer is a computer. The desktop eye tracker is connected to the computer via a universal serial bus. The computer contains a Gaussian mixture model program and a hidden Markov model program. The computer is equipped with a central processing unit with a base frequency of 2.0 GHz or higher and operates on a Windows 10 64-bit operating system. The display screen is larger than 13.3 inches and is located at least 75 cm away from the person being measured. The display screen is connected to the computer via a universal serial bus.

[0154] The steps for eye movement behavior classification are as follows:

[0155] (1) Collecting the original eye movement data of the subjects

[0156] See also Figure 1 , showing the workflow for collecting the original eye movement data of the subjects, including four stages: start, user gaze behavior, rapid positioning (including eye saccade and tracking behavior), and end.

[0157] (1.1) When the system is working, a red dot with a diameter of 80 pixels is displayed on the screen. The red dot is located at any position on the screen and the duration of each appearance is 500 to 1000 milliseconds.

[0158] (1.2) The subject sits in front of a display screen and clicks a start button on the display screen with a mouse. A red dot visual stimulus appears on the display screen three or more times in a regular pattern. The visual stimulus includes three types of eye movement behaviors to be identified, namely, fixation, tracking, and saccades.

[0159] In the above step (1.2), the number of fixations on the visual stimulus of the red dot is 4 times. The subject completes the fixation task within the set number of fixations, and the interval between two consecutive fixation stimuli is 0.5 seconds. Within the range of the display screen, the host computer randomly generates the pixel coordinates of the fixation position.

[0160] The term "gaze" refers to an eye movement behavior in which the gaze stays in any area for at least 500 milliseconds, indicating that the subject is interested in the content of the area or is processing information;

[0161] Tracking refers to the eye movement behavior of smoothly following a moving target, including tracking moving objects or text on a display screen.

[0162] Saccades refer to eye movements in which the line of sight quickly jumps from one fixation point to another. They are fast, short eye movements that usually occur between fixations and are used to shift visual attention from one area to another.

[0163] (1.3) During system operation, the desktop eye tracker continuously collects the position coordinates and timestamps of the subject's gaze point on the display screen and transmits them to the host computer. After processing by the host computer, raw eye movement data is obtained. Each piece of data in the raw eye movement data consists of the pixel coordinates of the gaze point and the velocity of the gaze point.

[0164] In this embodiment 1, the following program is set up in the host computer: at the initial moment, a target circular pattern appears in the center of the display screen to prompt the person being tested to start the program and start sampling. After waiting for about 1000 milliseconds, the stimulus begins to be presented, and a circular target appears above the display screen. 500 milliseconds after it appears, the circular target will quickly move to the left side of the display screen, with a movement time of 1000 milliseconds. It will remain on the left side of the display screen for 500 milliseconds, and then quickly flash to the bottom of the display screen and the right side of the display screen in turn, both remaining for 1000 milliseconds. Finally, the circular target returns to the center of the display screen, stays for 500 milliseconds, and then disappears, indicating that the program ends and sampling stops.

[0165] (2) Raw eye movement data preprocessing

[0166] The raw eye movement data is subjected to spatial feature preprocessing, and the pixel coordinates of the gaze point and the speed of the gaze point are grouped as a whole using a clustering method. In this embodiment, the data are divided into three groups, each with fifty pieces of eye movement data, to obtain three groups of preprocessed eye movement data.

[0167] The specific operations of using clustering method to group the original eye movement data are as follows:

[0168] (2.1) Clustering algorithm:

[0169] The specific operation process is as follows:

[0170] a) Initialize cluster centers:

[0171] Randomly select K data from the original eye movement data as initial cluster centers, and divide the original eye movement data into K eye movement data groups. In this embodiment, the maximum value of K is 4;

[0172] b) Assign data to the nearest cluster center:

[0173] The distance between each original eye movement data in the K eye movement data groups and the K cluster centers is calculated, and the original eye movement data closest to a cluster center is assigned to the nearest eye movement data group to obtain K new eye movement data groups.

[0174] c) Update cluster centers:

[0175] Calculate the data average of each new eye movement data group in the K new eye movement data groups, and use the data average as a new cluster center.

[0176] d) Iterative processing:

[0177] Repeat steps b) and c) until a predetermined number of iterations of 50 is reached; and obtain K final cluster centers and K final eye movement data groups.

[0178] (2.2) Elbow rule to determine the optimal number of groups

[0179] See also Figure 2 、 3 ,4,The scattered points in the above three figures all represent the gaze point trajectory, and the gaze point trajectory coordinates are all expressed in normalized coordinates. Figure 2 It represents the result of clustering the original eye movement data into 3 groups. Figure 3 It represents the result of clustering the original eye movement data into 4 groups. Figure 4 represents the result of clustering the raw eye movement data into five groups. The stimulation program written by the host computer in step (1.3) includes four fixed stimulus targets to represent gaze behavior. When the number of clusters set is close to the subject's gaze behavior, the eye movement data can better describe the subject's gaze behavior.

[0180] Apply the elbow rule to determine the optimal number of groups and calculate the sum of squared errors of the clustering results of the K final cluster centers. The sum of squared errors formula (1) is as follows:

[0181]

[0182] In formula (1), k is the number of cluster centers, n i is the number of data points in the i-th group, is the jth data point in the i-th group, c i is the cluster center of the i-th group.

[0183] When the number of groups k gradually increases, the error sum squared SSE(k) usually shows a decreasing trend, because more groups can better fit the distribution of the data and reduce the distance between the data point and the cluster center of the group to which it belongs. After a certain number of groups k, the decrease in the error sum squared SSE(k) will slow down significantly and form an inflection point. The inflection point represents the optimal balance between clustering performance and the number of groups. Based on the trend that the error sum squared of the clustering results decreases as the number of groups k increases, the number of cluster centers k corresponding to the point where the error sum squared SSE(k) decreases is selected to determine the optimal number of groups. For the k final cluster centers, the preprocessed k-1 groups of eye movement data are obtained.

[0184] See also Figure 5 , for this embodiment, the optimal number of cluster centers k can be determined to be 4, and 3 groups of eye movement data are obtained, corresponding to Figure 5 The inflection point is when k=4.

[0185] See also Figure 6 ,The black scattered points in the trajectory diagram of the original eye movement data represent the trajectory of the gaze point, Figure 6 The x-axis and y-axis represent the coordinates of the gaze point, using normalized coordinates, and the z-axis represents the velocity of the gaze point. Figure 7 ,Using the determined number of groups as 3, the original eye movement data is clustered into 3 groups, and each group is marked with yellow, green and blue scatter points.

[0186] (3) Establishment of Gaussian mixture model and hidden Markov model

[0187] (3.1) Establishing Gaussian mixture model

[0188] (3.1.1) The three sets of preprocessed eye movement data were statistically analyzed. Based on the similarity in speed of the three sets of preprocessed eye movement data, that is, when the speed difference between any two data points is less than the preset threshold of 0.002, the two data points are considered similar and are classified into the same cluster.

[0189] The three groups of eye movement data for statistical analysis were divided into three clusters according to the speed differences. Each cluster corresponds to a predefined eye movement behavior, namely the gaze Gaussian distribution model, the pursuit Gaussian distribution model and the saccade Gaussian distribution model.

[0190] Initialize three predefined eye movement behaviors. For the gaze Gaussian distribution model, the center position of the cluster with the lowest average speed is used as the center position of the gaze Gaussian distribution model, and the initial weight of the gaze Gaussian distribution is set to 1 / 3, and the initial covariance matrix of the gaze Gaussian distribution is set to the identity matrix.

[0191] For the saccade Gaussian distribution model, the center position of the cluster with the largest average speed is used as the center position of the saccade Gaussian distribution model. Similarly, the initial weight of the saccade Gaussian distribution is set to 1 / 3, and the initial covariance matrix of the saccade Gaussian distribution is set to the identity matrix.

[0192] For the tracking Gaussian distribution model, the center position of the remaining third cluster is used as the center position of the tracking Gaussian distribution model, the initial weight of the tracking Gaussian distribution is set to 1 / 3, and the initial covariance matrix of the tracking Gaussian distribution is set to the unit matrix.

[0193] The initial Gaussian mixture model is composed of the gaze Gaussian distribution model, the pursuit Gaussian distribution model and the saccade Gaussian distribution model.

[0194] (3.1.2) The parameters of the initialized Gaussian mixture model are adjusted through iterative optimization to better fit the actual probability distribution of each eye movement behavior. The number of iterations is set to at least 50. The degree of optimization of the Gaussian mixture model is determined by introducing the Q function to obtain the optimized Gaussian mixture model.

[0195] The specific optimization operations are as follows:

[0196] a) Initial parameter definition of Gaussian mixture model

[0197] The formula (2) of the Gaussian mixture model is as follows:

[0198]

[0199] In formula (2), P(x) represents the probability of observing eye movement data x, k represents the kth eye movement behavior, which is gaze, tracking, and saccade in sequence, c k represents the weight of the k-th eye movement behavior, N(x|μ k ,∑ k ) represents the probability density function of the Gaussian distribution of the k-th eye movement behavior, μ k and ∑ k denote the mean and covariance matrix of the k-th Gaussian distribution respectively.

[0200] Taking the three eye movement behaviors of fixation, tracking, and saccade as examples, the probability density function (3) of the Gaussian mixture model is as follows:

[0201] P(x)=c1·N(x|μ1,∑1)+c2·N(x|μ2,∑2)+c3·N(x|μ3,∑3)(3)

[0202] In formula (3), the weights c1, c2, c3, the mean values ​​μ1, μ2, μ3 and the covariance matrices ∑1, ∑2, ∑3 are all parameters to be optimized in the Gaussian mixture model.

[0203] b) Calculation of liability

[0204] To adjust the Gaussian mixture model parameters, it is necessary to calculate each eye movement data x n The probability of the three eye movement behaviors of fixation, tracking and saccade is called responsibility γ(z nk ), responsibility γ(z nk ) is calculated as follows:

[0205]

[0206] In formula (4), x n represents the nth observed eye movement data, z nkis a symbolic variable, indicating that the nth eye movement data belongs to the kth Gaussian distribution.

[0207] c) Update Gaussian mixture model parameters

[0208] Responsibility for use γ(z nk ) to update the parameters of the Gaussian mixture model, which include the weight c k , average value μ k and covariance matrix ∑ k ;

[0209] Weight c k The update formula (5) is as follows:

[0210]

[0211] Average value μ k The update formula (6) is as follows:

[0212]

[0213] Covariance matrix ∑ k The update formula (7) is as follows:

[0214]

[0215] Repeat the calculation of formulas (5)-(7) at least 50 times to update the parameters of the Gaussian mixture model.

[0216] In order to determine the degree of optimization of the Gaussian mixture model, the Q function is introduced. The Q function formula (8) is as follows:

[0217]

[0218] In formula (8), θ represents the Gaussian mixture model parameters in the current iteration, including the mixing coefficients c of each Gaussian distribution k , mean μ k and covariance matrix ∑ k ,θ (t) represents the Gaussian mixture model parameters in the previous iteration, P(x i ,z ik |θ) Under the current model parameters θ, the eye movement data x i The probability of belonging to the kth Gaussian distribution, P(z ik |x i ,θ (t) ) represents the model parameter θ in the previous iteration (t) Next, eye movement data x i The probability of belonging to the kth Gaussian distribution.

[0219] The Q function measures the degree of model optimization by comparing the parameters of the current Gaussian mixture model with the parameters of the previous iteration. The larger the value of the Q function, the greater the improvement of the current model parameters compared with the previous iteration. When the value of the Q function is less than the set threshold of 0.001, or the number of calculations reaches the preset number of repetitions, the parameter optimization of the Gaussian mixture model is completed, and the parameters of the Gaussian mixture model at this time, including the weight c, are recorded. k , average value μ k and covariance matrix ∑ k , that is, the optimized Gaussian mixture model is obtained.

[0220] (3.2) Establishing Hidden Markov Model

[0221] (3.2.1) Initialize the three parameters of the hidden Markov model:

[0222] Initial probability matrix π=[π1,π2,…,π N ], represents the initial probability of each eye movement behavior at the initial moment, and the initial probability of each eye movement behavior is set to 1 / 3;

[0223] The transition probability matrix A = [a ij ], represents the transition probability from one eye movement behavior to another, ensuring that the sum of each row element of the transition probability matrix A is 1;

[0224] Emission probability matrix B = [b j (o t )], which represents the probability that each data in the eye movement data belongs to one of the three eye movement behaviors. Each row element of the emission probability matrix B is determined by the parameters of the Gaussian mixture model in (3.1.2).

[0225] By initializing the above three parameters of the hidden Markov model, an initialized hidden Markov model is obtained.

[0226] (3.2.2) Optimizing the initialized hidden Markov model using the three sets of preprocessed eye movement data through an optimization algorithm to obtain an optimized hidden Markov model.

[0227] The specific optimization steps are as follows:

[0228] a) Define and calculate forward probability

[0229] The observation state sequence of eye movement behavior is defined as O = (o1, o2, ..., o T ), the above sequence represents the eye movement behavior observed from the initial moment to the last moment. In the process of optimizing the hidden Markov model, the forward probability α is first calculated. t (i), the forward probability is calculated as follows (9):

[0230]

[0231] Formula (9) indicates that the state at time t is i, and the observation sequence of eye movement behavior is (o1, o2,…, o t ), in formula (9), a ji represents the transition probability from state j at time t-1 to state i at time t, b i represents the emission probability of state i, a ji and b i are the parameters of the hidden Markov model.

[0232] b) Define and calculate backward probability

[0233] Calculate the backward probability β t (i), the backward probability is calculated as follows (10):

[0234]

[0235] Formula (10) indicates that the state at time t is i, and the observation sequence of eye movement behavior is (o t+1 ,o t+1 ,…,o t ), in formula (10), a ij represents the transition probability from state j at time t+1 to state i at time t, b j represents the emission probability of state j, a ij and b j are the parameters of the hidden Markov model.

[0236] c) Update the parameters of the hidden Markov model

[0237] Using the forward probability and the backward probability, calculate the prior probability γ t (i) and the joint probability ξ t (i, j) is used to update the parameters of the hidden Markov model, where the prior probability γ t (i) represents the probability of state i at time t, and the joint probability ξ t (i, j) represents the probability that the state is i at time t and the state is j at time t+1, and the prior probability γ t The calculation formula (11) for (i) is as follows:

[0238]

[0239] Joint probability ξ t The calculation formula (12) of (i, j) is as follows:

[0240]

[0241] In formulas (11) and (12), the transition probability a ij represents the probability of transitioning from state i to state j, and the emission probability b j (o t+1 ) indicates that the observation value in state j is o t+1 The probability of ; by using the prior probability γ t (i) and the joint probability ξ t (i, j) is used to update the three parameters of the hidden Markov model, the initial probability matrix π, the transition probability matrix A, and the emission probability matrix B. The specific update operation is as follows:

[0242] The update formula (13) of the initial state probability vector π is as follows:

[0243] π i =γ1(i) (13)

[0244] In formula (13), π i Represents the i-th element of the optimized initial state probability vector;

[0245] The update formula (14) of the state transition probability matrix A is as follows:

[0246]

[0247] In formula (14), a ij Represents the element in the i-th row and j-th column of the optimized state transition probability matrix;

[0248] The update formula (15) of the emission probability matrix B is as follows:

[0249]

[0250] In formula (15), b j (k) represents the jth row and kth column element of the optimized emission probability matrix, v k represents the kth possible observation value.

[0251] By repeating the calculation of formulas (13)-(15) at least 100 times, the three parameters of the above hidden Markov model are optimized so that the hidden Markov model can continuously fit the eye movement data. When the preset number of repetitions is reached, the hidden Markov model is considered to have been optimized, and the optimized hidden Markov model is finally obtained.

[0252] (4) Classification of eye movement behavior

[0253] Based on the Gaussian mixture model optimized in step (3.1.2) and the hidden Markov model optimized in step (3.2.2), the three sets of preprocessed eye movement data are classified into eye movement behaviors; based on the eye movement data at each time point, it is inferred which of the three eye movement behaviors the current eye movement behavior is, namely, fixation, saccade, or tracking.

[0254] (4.1) Determine the optimal path probability at each moment

[0255] Assume that the hidden state sequence of eye movement behavior is S=(s1,s2,...,s T ), the hidden state of the eye movement behavior represents the actual eye movement behavior, which is a type of eye movement behavior among gaze, pursuit and saccade, and the sequence S represents all actual eye movement behaviors from the initial moment to the final moment.

[0256] Assume that the observation state sequence of eye movement behavior is O=(o1,o2,…,o T ), the observed state of the eye movement behavior represents the eye movement behavior estimated based on the observed eye movement data, which is also a type of eye movement behavior among gaze, pursuit and saccade. The sequence O represents all eye movement behaviors estimated based on the observed eye movement data from the initial moment to the final moment.

[0257] In order to obtain the hidden state sequence with the highest probability, that is, to obtain the most likely eye movement behavior, the optimal sequence function is defined as δ t (i), optimal sequence function δ t (i) indicates that when the hidden state is i at time t, the observed state is (o1, o2, ..., o t ), the maximum probability of the optimal sequence function δ t The specific value of (i) is obtained by recursive calculation, and the optimal sequence function δ t The recursive formula (16) of (i) is as follows:

[0258]

[0259] In formula (16), a ij represents the transition probability from state i to state j, Indicates that o is observed in state j t The emission probability, δ t-1 (i) represents the probability of state i at time t-1.

[0260] By recursively formula (16), the maximum probability δ corresponding to each hidden state i is calculated at each time t t (i). (4.2) Determine the optimal precursor state at each moment

[0261] Get the maximum probability δ of the hidden state at time tt (i) Based on this, we need to record the most likely previous state of the hidden state i at time t, that is, the optimal predecessor state at time t; define the optimal predecessor state ψ t (i), record the most likely previous eye movement behavior of state i at time t, the optimal precursor state ψ t Formula (17) of (i) is as follows:

[0262]

[0263] Formula (17) records the optimal precursor state at each moment t, and recursively finds the hidden state from the final moment to the initial moment, thereby classifying the eye movement behavior at each time point.

[0264] (4.3) Classification of eye movement behavior

[0265] Using the optimal path probability δ t (i) and the optimal precursor state ψ t (i) Calculate the maximum probability P of the hidden state sequence * and the terminal state of the hidden state sequence Classify eye movement behavior.

[0266] The maximum probability P of the hidden state sequence * Indicates that at the final moment T, the optimal path probability δ corresponding to the three hidden states of gaze, pursuit and saccade is calculated respectively t (i) And select the optimal path probability value with the largest value as the maximum probability P of the hidden state sequence * , the maximum probability P * The calculation formula (18) is as follows:

[0267]

[0268] In formula (18), N represents the number of hidden states, and the upper limit here is 3;

[0269] The maximum probability P of the hidden state sequence at the final moment T is obtained * On this basis, it is also necessary to determine the terminal state of the hidden state sequence Terminal State The calculation formula (19) is as follows:

[0270]

[0271] When the termination status is obtained When , we start to recurse from the final moment T to the initial moment to obtain the hidden state at each moment; the recursive formula (20) is as follows:

[0272]

[0273] In formula (20), represents the optimal hidden state at time t, represents the optimal hidden state at time t+1 The most likely previous state is obtained by using formula (20) to determine the hidden state of each moment in the entire hidden state sequence, that is, to classify the eye movement behavior from the initial moment to the final moment. It is inferred which of the three eye movement behaviors the current eye movement behavior belongs to, namely, fixation, saccade, or tracking, and the eye movement behavior classification is completed.

[0274] See also Figure 8 、 Figure 9 , and output the results after state recognition of 3 sets of eye movement data. Figure 8 The a in the figure represents the trajectory of the gaze point x-axis changing with time. The horizontal axis represents time in seconds, the left vertical axis represents the gaze point position in normalized coordinates, the right vertical axis represents the gaze point speed change curve, the blue curve represents the gaze point x-axis position indicator, and the red curve represents the gaze point speed. Figure 8 The b in the figure is the classification result. The horizontal axis represents time in seconds, and the vertical axis represents the classification result. The blue dots represent gaze behavior, which is concentrated in the time period with low speed and little change in position index. The yellow dots represent tracking behavior, which appears in the time period with relatively stable change in position index but certain speed. The red dots represent saccade behavior, which is concentrated in the time period with rapid change in speed. Figure 9 The a in the figure represents the trajectory of the gaze point's y-axis changing with time. The horizontal axis represents time in seconds, the left vertical axis represents the gaze point's position in normalized coordinates, the right vertical axis represents the gaze point's speed change curve, the blue curve represents the gaze point's y-axis position indicator, and the red curve represents the gaze point's speed. Figure 9 The b in the figure represents the classification result, the horizontal axis represents time in seconds, and the vertical axis represents the classification result. Similarly, the blue dot represents the gaze behavior, the yellow dot represents the tracking behavior, and the red dot represents the saccade behavior.

[0275] The three types of eye movement behaviors are specifically classified as follows:

[0276] Attention: When the probability δ corresponding to the hidden state t (i) When it exceeds 0.9 and the state lasts for more than 200 milliseconds, it indicates that the current eye movement behavior is fixation;

[0277] Eye saccade: When the hidden state changes within 50 milliseconds, and the corresponding transition probability a ij If it is greater than 0.8, it indicates that the current eye movement behavior is saccade;

[0278] Tracking: When the hidden state changes within 500 milliseconds, and the corresponding transition probability a ijIf it exceeds 0.9, it indicates that the current eye movement behavior is tracking.

[0279] Example 2

[0280] This embodiment 2 is an implementation process for classifying eye movement behaviors in complex eye movement trajectories. The hardware system used in this embodiment 2 is the same as that in embodiment 1, and the operating steps of this embodiment 2 are basically the same as those in embodiment 1.

[0281] The steps for eye movement behavior classification are as follows:

[0282] (1) Collecting the original eye movement data of the subjects

[0283] In this step (1), the host computer sets the visual stimulus to appear at different positions on the display screen in a random manner, see Figure 10 , Figure 10 The blue scattered points in the figure represent the fixation points, and the red scattered points represent that 9 visual stimuli were set to guide the subjects to perform multiple fixation eye movement behaviors.

[0284] Other operations are the same as in Example 1.

[0285] (2) Raw eye movement data preprocessing

[0286] See also Figure 11 , 12, Figure 12 The scatter points in the figure represent the trajectory of the original eye movement data, and the black scatter points represent the trajectory of the fixation points. Figure 12 The x-axis and y-axis represent the coordinates of the gaze point, which are expressed in normalized coordinates, and the z-axis represents the speed of the gaze point. The raw eye movement data is preprocessed with spatial features, and the pixel coordinates of the gaze point and the speed of the gaze point are grouped as a whole using a clustering method. The optimal number of groups is determined by analyzing the elbow rule. For this embodiment, the optimal number of cluster centers k can be determined to be 10, resulting in 9 groups of eye movement data, corresponding to Figure 11 The inflection point of k=10 is obtained, and 9 sets of eye movement data after preprocessing are obtained.

[0287] Other operations are the same as in Example 1.

[0288] (3) Establishment of Gaussian mixture model and hidden Markov model

[0289] In this embodiment 2, the operation process of step 3 is the same as that of embodiment 1, and the 9 sets of pre-processed eye movement data are modeled to establish an optimized Gaussian mixture model and an optimized hidden Markov model.

[0290] (4) Classification of eye movement behavior

[0291] In this embodiment 2, based on the step Gaussian mixture model and the hidden Markov model, the 9 sets of preprocessed eye movement data are classified into eye movement behaviors, and the current eye movement behaviors, such as fixation, saccade and tracking, are inferred based on the eye movement data at each time point.

[0292] See also Figure 13 , different eye movement behaviors are marked with different shapes on the original eye movement data, Figure 13 Medium blue dots represent saccadic behavior, green dots represent pursuit behavior, and orange dots represent fixation behavior.

[0293] refer to Figure 14 and Figure 15 ,The result output after state recognition of 3 sets of eye movement data, the two figures represent the trajectory of x-axis and y-axis changes over time, Figure 14 and Figure 15 Different types of eye movement behaviors are marked with different colors.

[0294] Figure 14 The a in the figure represents the trajectory of the gaze point x-axis changing with time. The horizontal axis represents time in seconds, the left vertical axis represents the gaze point position in normalized coordinates, the right vertical axis represents the gaze point speed change curve, the blue curve represents the gaze point x-axis position indicator, and the red curve represents the gaze point speed. Figure 14 The b in the figure is the classification result. The horizontal axis represents time in seconds, and the vertical axis represents the classification result. The blue dots represent gaze behavior, which is concentrated in the time period with low speed and little change in position index. The yellow dots represent tracking behavior, which appears in the time period with relatively stable change in position index but certain speed. The red dots represent saccade behavior, which is concentrated in the time period with rapid change in speed. Figure 15 The a in the figure represents the trajectory of the gaze point's y-axis changing with time. The horizontal axis represents time in seconds, the left vertical axis represents the gaze point's position in normalized coordinates, the right vertical axis represents the gaze point's speed change curve, the blue curve represents the gaze point's y-axis position indicator, and the red curve represents the gaze point's speed. Figure 15 The b in the figure represents the classification result, the horizontal axis represents time in seconds, and the vertical axis represents the classification result. Similarly, the blue dot represents the gaze behavior, the yellow dot represents the tracking behavior, and the red dot represents the saccade behavior.

[0295] Other operations are the same as in Example 1.

[0296] It is easy for those skilled in the art to understand that the above embodiments 1-2 are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An eye movement behavior classification method based on a Gaussian mixture model and a hidden Markov model, wherein the hardware system for implementing the eye movement behavior classification method includes a desktop eye tracker, a host computer, and a display screen; the host computer is a host computer, and the host computer has a Gaussian mixture model program and a hidden Markov model program; It is characterized in that the eye movement behavior classification operation steps are as follows: (1) Collecting the original eye movement data of the subjects (1.1) When the hardware system is working, a red dot with a diameter of 80 pixels is displayed on the screen. The red dot is located at any position on the screen and the duration of each appearance is 500 to 1000 milliseconds. (1.2) The subject sits in front of a display screen and clicks a start button on the screen with a mouse. A red dot visual stimulus appears on the screen three or more times in a regular pattern. The visual stimulus includes three types of eye movement behaviors to be identified: fixation, tracking, and saccades. (1.3) During the operation of the hardware system, the desktop eye tracker continuously collects the position coordinates and timestamps of the subject's gaze point on the display screen, and transmits them to the host computer. The host computer processes them to obtain raw eye movement data. Each piece of data in the raw eye movement data consists of the pixel coordinates of the gaze point and the velocity of the gaze point; (2) Preprocessing of raw eye movement data Performing spatial feature preprocessing on the raw eye movement data, using a clustering method to group the pixel coordinates of the gaze point and the velocity of the gaze point as a whole into at least three groups, each group of eye movement data having no less than fifty pieces, to obtain at least three groups of preprocessed eye movement data; (3) Establishing Gaussian mixture model and hidden Markov model (3.1) Establishing Gaussian mixture model (3.1.1) performing statistical analysis on the three or more pre-processed eye movement data sets, and based on the similarity in velocity among the three or more pre-processed eye movement data sets, i.e., when the velocity difference between any two data points is less than a preset threshold of 0.002, the two data points are considered similar and are classified into the same cluster; The three or more groups of eye movement data for statistical analysis were divided into three clusters according to the speed difference. Each cluster corresponds to a predefined eye movement behavior, namely the gaze Gaussian distribution model, the pursuit Gaussian distribution model and the saccade Gaussian distribution model. Initialize three predefined eye movement behaviors. For the gaze Gaussian distribution model, take the center position of the cluster with the lowest average speed as the center position of the gaze Gaussian distribution model, set the initial weight of the gaze Gaussian distribution to 1 / 3, and set the initial covariance matrix of the gaze Gaussian distribution to the identity matrix. For the saccade Gaussian distribution model, the center position of the cluster with the largest average speed is used as the center position of the saccade Gaussian distribution model. Similarly, the initial weight of the saccade Gaussian distribution is set to 1 / 3, and the initial covariance matrix of the saccade Gaussian distribution is set to the identity matrix. For the tracking Gaussian distribution model, the center position of the remaining third cluster is used as the center position of the tracking Gaussian distribution model, the initial weight of the tracking Gaussian distribution is set to 1 / 3, and the initial covariance matrix of the tracking Gaussian distribution is set to the identity matrix; The initial Gaussian mixture model is composed of the gaze Gaussian distribution model, the pursuit Gaussian distribution model and the saccade Gaussian distribution model; (3.1.2) Adjusting the parameters of the initialized Gaussian mixture model through iterative optimization to better fit the actual probability distribution of each eye movement behavior, with the number of iterations set to at least 50; and determining the degree of optimization of the Gaussian mixture model by introducing a Q function to obtain an optimized Gaussian mixture model; (3.2) Establishing Hidden Markov Model (3.2.1) Initialize the three parameters of the hidden Markov model: Initial probability matrix π=[π1,π2,…,π N ], represents the initial probability of each eye movement behavior at the initial moment, and the initial probability of each eye movement behavior is set to 1 / 3; The transition probability matrix A = [a ij ], represents the transition probability from one eye movement behavior to another, ensuring that the sum of each row element of the transition probability matrix A is 1; Emission probability matrix B = [b j (o t )], which represents the probability that each data in the eye movement data belongs to one of the three eye movement behaviors. Each row element of the emission probability matrix B is determined by the parameters of the Gaussian mixture model in (3.1.2); By initializing the above three parameters of the hidden Markov model, an initialized hidden Markov model is obtained; (3.2.2) Optimizing the initialized hidden Markov model using the three or more pre-processed eye movement data using an optimization algorithm to obtain an optimized hidden Markov model; (4) Classification of eye movement behavior Based on the Gaussian mixture model optimized in step (3.1.2) and the hidden Markov model optimized in step (3.2.2), the eye movement behavior of the three or more preprocessed eye movement data is classified; based on the eye movement data at each time point, it is inferred whether the current eye movement behavior is fixation, saccade, or tracking eye movement behavior.

2. The eye movement behavior classification method according to claim 1, characterized in that: The desktop eye tracker is connected to the host computer via a universal serial bus; the display screen is larger than 13.3 inches and is located at least 75 centimeters away from the person being measured; The display screen is connected to the host computer via a universal serial bus.

3. The eye movement behavior classification method according to claim 1, characterized in that: The host computer connected to the desktop eye tracker and display is equipped with a central processing unit with a basic frequency of more than 2.0 GHz and a Windows 10 64-bit operating system.

4. The classification method according to claim 1, wherein: In step (1.2), the number of fixations on the visual stimulus of the red dot is 3 times, 5 times, or more than 10 times. The subjects complete the fixation task within the set number of fixations, and the interval between two consecutive fixation stimuli is 0.5 seconds or 1 second. Within the range of the display screen, the host computer randomly generates the pixel coordinates of the fixation position.

5. The classification method according to claim 1, wherein: In step (1.2), the gaze refers to an eye movement behavior in which the gaze stays in any area for at least 500 milliseconds, indicating that the subject is interested in the content of the area or is processing information; Tracking refers to the eye movement behavior of smoothly following a moving target, including tracking moving objects or text on a display screen. Saccades refer to eye movements in which the line of sight quickly jumps from one fixation point to another. They are fast, short eye movements that usually occur between fixations and are used to shift visual attention from one area to another.

6. The classification method according to claim 1, wherein: In step (2), a clustering method is used to group the original eye movement data. The specific operation is as follows: (2.1) Clustering algorithm: a) Initialize cluster centers: Randomly select K data from the original eye movement data as the initial clustering centers, and divide the original eye movement data into K eye movement data groups, where the value of K is at least 4; b) Assign data to the nearest cluster center: Calculate the distance between each original eye movement data in the K eye movement data groups and the K cluster centers, assign the original eye movement data closest to a cluster center to the nearest eye movement data group, and obtain K new eye movement data groups; c) Update cluster centers: Calculate the data average of each new eye movement data group in the K new eye movement data groups, and average the data The mean is used as the new cluster center; d) Iterative processing: Repeat steps b) and c) until a predetermined number of iterations of 50 is reached; obtaining K final cluster centers and K final eye movement data sets; (2.2) Elbow rule determines the optimal number of groups: Apply the elbow rule to determine the optimal number of groups. The specific steps are as follows: Calculate the sum of squared errors (Sum of Squared Errors) of the clustering results of the K final cluster centers. The sum of squared errors formula (1) is as follows: In formula (1), k is the number of cluster centers, n i is the number of data points in the i-th group, is the jth data point in the i-th group, c i is the cluster center of the i-th group; When the number of groups k gradually increases, the error sum squared SSE(k) usually shows a decreasing trend, because more groups can better fit the distribution of the data and reduce the distance between the data point and the cluster center of the group to which it belongs. After a certain number of groups k, the decrease in the error sum squared SSE(k) will slow down significantly and form an inflection point. The inflection point represents the optimal balance between clustering performance and the number of groups. Based on the trend that the error sum squared of the clustering results decreases as the number of groups k increases, the number of cluster centers k corresponding to the point where the error sum squared SSE(k) decreases is selected to determine the optimal number of groups. For the k final cluster centers, the preprocessed k-1 groups of eye movement data are obtained.

7. The classification method according to claim 1, wherein: In step (3.1.2), the operation of adjusting the Gaussian mixture model parameters by the iterative optimization method is as follows: a). Initial parameter definition of Gaussian mixture model The formula (2) of the Gaussian mixture model is as follows: In formula (2), P(x) represents the probability of observing eye movement data x, k represents the kth eye movement behavior, which is gaze, tracking, and saccade in sequence, c k represents the weight of the k-th eye movement behavior, N(x|μ k ,∑ k ) represents the probability density function of the Gaussian distribution of the k-th eye movement behavior, μ k and ∑ k Represent the mean and covariance matrix of the k-th Gaussian distribution respectively; The probability density function (3) of the Gaussian mixture model of the three eye movement behaviors of fixation, pursuit, and saccade is as follows: P(x)=c1·N(x|μ1,∑1)+c2·N(x|μ2,∑2)+c3·N(x|μ3,∑3)(3) In formula (3), the weights c1, c2, c3, the mean values ​​μ1, μ2, μ3 and the covariance matrices ∑1, ∑2, ∑3 are all parameters to be optimized in the Gaussian mixture model; b). Calculation of responsibility To adjust the Gaussian mixture model parameters, it is necessary to calculate each eye movement data x n The probability of the three eye movement behaviors of fixation, tracking and saccade is called responsibility γ(z nk ), responsibility γ(z nk ) is calculated as follows: In formula (4), x n represents the nth observed eye movement data, z nk is a symbolic variable, indicating that the nth eye movement data belongs to the kth Gaussian distribution; c). Update Gaussian mixture model parameters Responsibility for use γ(z nk ) to update the parameters of the Gaussian mixture model, which include the weight c k , average value μ k and covariance matrix ∑ k ; Weight c k The update formula (5) is as follows: Average value μ k The update formula (6) is as follows: Covariance matrix ∑ k The update formula (7) is as follows: Repeat the calculation of formula (5)-(7) at least 50 times to update the parameters of the Gaussian mixture model; In order to determine the degree of optimization of the Gaussian mixture model, the Q function is introduced. The Q function formula (8) is as follows: In formula (8), θ represents the Gaussian mixture model parameters in the current iteration, including the mixing coefficients c of each Gaussian distribution k , mean μ k and covariance matrix ∑ k ,θ (t) represents the Gaussian mixture model parameters in the previous iteration, P(x i ,z ik |θ) Under the current model parameters θ, the eye movement data x i The probability of belonging to the kth Gaussian distribution, P(z ik |x i ,θ (t) ) represents the model parameter θ in the previous iteration (t) Next, eye movement data x i The probability of belonging to the kth Gaussian distribution; The Q function measures the degree of model optimization by comparing the parameters of the current Gaussian mixture model with the parameters of the previous iteration. The larger the value of the Q function, the greater the improvement of the current model parameters compared with the previous iteration. When the value of the Q function is less than the set threshold of 0.001, or the number of calculations reaches the preset number of repetitions, the parameter optimization of the Gaussian mixture model is completed, and the parameters of the Gaussian mixture model at this time, including the weight c, are recorded. k , average value μ k and covariance matrix ∑ k , that is, the optimized Gaussian mixture model is obtained.

8. The classification method according to claim 1, wherein: In step (3.2.2), the method for optimizing the parameters of the hidden Markov model is as follows: a). Define and calculate forward probability The observation state sequence of eye movement behavior is defined as O = (o1, o2, ..., o T ), the above sequence represents the eye movement behavior observed from the initial moment to the last moment. In the process of optimizing the hidden Markov model, the forward probability α is first calculated. t (i), the forward probability is calculated as follows (9): Formula (9) indicates that the state at time t is i, and the observation sequence of eye movement behavior is (o1, o2,…, o t ), in formula (9), a ji represents the transition probability from state j at time t-1 to state i at time t, b i represents the emission probability of state i, a ji and b i are all parameters of the hidden Markov model; b) Define and calculate backward probability Calculate the backward probability β t (i), the backward probability is calculated as follows (10): Formula (10) indicates that the state at time t is i, and the observation sequence of eye movement behavior is (o t+1 ,o t+1 ,…,o t ), in formula (10), a ij represents the transition probability from state j at time t+1 to state i at time t, b j represents the emission probability of state j, a ij and b j are all parameters of the hidden Markov model; c). Update the parameters of the hidden Markov model Using the forward probability and the backward probability, calculate the prior probability γ t (i) and the joint probability ξ t (i, j) is used to update the parameters of the hidden Markov model, where the prior probability γ t (i) represents the probability of state i at time t, and the joint probability ξ t (i, j) represents the probability that the state is i at time t and the state is j at time t+1, and the prior probability γ t The calculation formula (11) for (i) is as follows: Joint probability ξ t The calculation formula (12) for (i, j) is as follows: In formulas (11) and (12), the transition probability a ij represents the probability of transitioning from state i to state j, and the emission probability b j (o t+1 ) indicates that the observation value in state j is o t+1 The probability of ; by using the prior probability γ t (i) and the joint probability ξ t (i, j) is used to update the three parameters of the hidden Markov model, the initial probability matrix π, the transition probability matrix A, and the emission probability matrix B. The specific update operation is as follows: The update formula (13) of the initial state probability vector π is as follows: p i =γ1(i) (13) In formula (13), π i Represents the i-th element of the optimized initial state probability vector; The update formula (14) of the state transition probability matrix A is as follows: In formula (14), a ij Represents the element in the i-th row and j-th column of the optimized state transition probability matrix; The update formula (15) of the emission probability matrix B is as follows: In formula (15), b j (k) represents the jth row and kth column element of the optimized emission probability matrix, v k Represents the kth possible observation value; by repeating formulas (13)-(15) for at least 100 times, the three parameters of the above hidden Markov model are optimized so that the hidden Markov model continuously fits the eye movement data. When the preset number of repetitions is reached, the hidden Markov model is considered to have been optimized, and finally the optimized hidden Markov model is obtained.

9. The classification method according to claim 1, characterized in that: In step (4), the eye movement behavior classification is performed on the pre-processed three or more groups of eye movement data to obtain the eye movement behavior at each moment. The specific operation is as follows: (4.1) Determine the optimal path probability at each moment Assume that the hidden state sequence of eye movement behavior is S=(s1,s2,...,s T ), the hidden state of the eye movement behavior represents the actual eye movement behavior, which is one of the eye movement behaviors of fixation, pursuit and saccade, and the sequence S represents all the actual eye movement behaviors from the initial moment to the final moment; Assume that the observation state sequence of eye movement behavior is O=(o1,o2,…,o T ), the observed state of the eye movement behavior represents the eye movement behavior estimated based on the observed eye movement data, which is also an eye movement behavior among fixation, pursuit and saccade, and the sequence O represents all the eye movement behaviors estimated based on the observed eye movement data from the initial moment to the final moment; In order to obtain the hidden state sequence with the highest probability, that is, to obtain the most likely eye movement behavior, the optimal sequence function is defined as δ t (i), optimal sequence function δ t (i) indicates that when the hidden state is i at time t, the observed state is (o1, o2, ..., o t ), the maximum probability of the optimal sequence function δ t The specific value of (i) is obtained by recursive calculation, and the optimal sequence function δ t The recursive formula (16) of (i) is as follows: In formula (16), a ij represents the transition probability from state i to state j, Indicates that o is observed in state j t The emission probability, δ t-1 (i) represents the probability of state i at time t-1; By recursively formula (16), the maximum probability δ corresponding to each hidden state i is calculated at each time t t (i); (4.2) Determine the optimal precursor state at each moment Get the maximum probability δ of the hidden state at time t t (i) Based on this, we need to record the most likely previous state of the hidden state i at time t, that is, the optimal predecessor state at time t; define the optimal predecessor state ψ t (i), record the most likely previous eye movement behavior of state i at time t, the optimal precursor state ψ t Formula (17) of (i) is as follows: Formula (17) records the optimal precursor state at each moment t, and recursively finds the hidden state from the final moment to the initial moment, thereby classifying the eye movement behavior at each time point; (4.3) Classification of eye movement behavior Using the optimal path probability δ t (i) and the optimal precursor state ψ t (i) Calculate the maximum probability P of the hidden state sequence * and the terminal state of the hidden state sequence Classify eye movement behavior; The maximum probability P of the hidden state sequence * Indicates that at the final moment T, the optimal path probability δ corresponding to the three hidden states of gaze, pursuit and saccade is calculated respectively t (i) And select the optimal path probability value with the largest value as the maximum probability P of the hidden state sequence * , the maximum probability P * The calculation formula (18) is as follows: In formula (18), N represents the number of hidden states, and the upper limit here is 3; The maximum probability P of the hidden state sequence at the final moment T is obtained * On this basis, it is also necessary to determine the terminal state of the hidden state sequence Terminal State The calculation formula (19) is as follows: When the termination status is obtained When , we start to recurse from the final moment T to the initial moment to obtain the hidden state at each moment; the recursive formula (20) is as follows: In formula (20), represents the optimal hidden state at time t, represents the optimal hidden state at time t+1 The most likely previous state corresponding to the hidden state; determine the hidden state of each moment of the entire hidden state sequence through formula (20), that is, classify the eye movement behavior from the initial moment to the final moment; infer whether the current eye movement behavior belongs to gaze, saccade, or tracking eye movement behavior, and complete the eye movement behavior classification; The three types of eye movement behaviors are specifically classified as follows: Attention: When the probability δ corresponding to the hidden state t (i) When it exceeds 0.9 and the state lasts for more than 200 milliseconds, it indicates that the current eye movement behavior is fixation; Eye saccade: When the hidden state changes within 50 milliseconds, and the corresponding transition probability a ij If it is greater than 0.8, it indicates that the current eye movement behavior is saccade; Tracking: When the hidden state changes within 500 milliseconds, and the corresponding transition probability a ij If it exceeds 0.9, it indicates that the current eye movement behavior is tracking.

Images