User behavior classification method and apparatus based on eye movement data, device and medium

Abstract

The present disclosure relates to the technical field of computers. Provided are a user behavior classification method and apparatus based on eye movement data, a device and a medium. The method comprises: acquiring eye movement data generated by a user executing a human-computer interaction task in an eye movement scenario; and inputting the eye movement data into a classification model, so as to obtain a user behavior classification result output by the classification model, the classification model being obtained by means of training based on an eye movement classification network and based on original eye movement coordinate data, eye movement task scenario pictures, and eye movement feature data used for classification. The present disclosure combines eye movement tracking technology with a machine learning algorithm to achieve user behavior classification based on eye movement data, thereby avoiding the problems of strong subjectivity and low accuracy of manual classification, and improving the accuracy of user behavior classification.

Description

User behavior classification methods, devices, equipment, and media based on eye-tracking data

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 2024109357997, filed on July 12, 2024, entitled "Method, Apparatus, Device and Medium for Classifying User Behavior Based on Eye-Tracking Data", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for classifying user behavior based on eye-tracking data. Background Technology

[0004] Traditional user behavior classification methods (such as those for autism) rely heavily on human intervention, which suffers from high subjectivity and low accuracy. Therefore, improving the accuracy of user behavior classification is a pressing issue that needs to be addressed.

[0005] Summary of the Invention

[0006] This disclosure provides a method, apparatus, device, and medium for user behavior classification based on eye-tracking data, which addresses the problem of low accuracy in user behavior classification in the prior art, and realizes user behavior classification based on eye-tracking data, thereby improving the accuracy of user behavior classification.

[0007] This disclosure provides a user behavior classification method based on eye-tracking data, including:

[0008] Acquire eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios;

[0009] The eye-tracking data is input into the classification model to obtain the classification result of the user behavior output by the classification model; the classification model is trained on the basis of the eye-tracking classification network, based on the original eye-tracking coordinate data, eye-tracking task scene images and eye-tracking feature data used for classification.

[0010] According to the user behavior classification method based on eye-tracking data provided in this disclosure, the classification model is trained in the following manner:

[0011] The raw eye-tracking coordinate data and the eye-tracking task scene images generated by the subject performing the human-computer interaction task in the eye-tracking scene were collected.

[0012] The eye movement feature data is determined based on the original eye movement coordinate data and the eye movement task scene image;

[0013] The original eye-tracking coordinate data, the eye-tracking task scene image, and the eye-tracking feature data are used to train the eye-tracking classification network to obtain the classification model.

[0014] The eye-tracking scene is a virtual scene that includes a virtual host, virtual objects, and a virtual background; the human-computer interaction task includes multiple levels, each level corresponding to a different interaction task.

[0015] According to the user behavior classification method based on eye movement data provided in this disclosure, the original eye movement coordinate data includes binocular coordinate data, gaze direction data, gaze point two-dimensional coordinate data, and eye closure degree data; the eye movement feature data includes total gaze duration, first gaze time, first gaze duration, average gaze time, gaze frequency, and gaze percentage.

[0016] The step of determining the eye movement feature data based on the original eye movement coordinate data and the eye movement task scene image includes:

[0017] Based on the eye-tracking task scene image, the eye-tracking scene is divided into multiple regions;

[0018] Based on the original eye movement coordinate data, the sum of the durations during which the two-dimensional coordinate data of the fixation point in each of the levels falls within each of the regions is taken as the total fixation duration for each region.

[0019] Based on the original eye movement coordinate data, the time difference between the first fixation time point of the fixation point two-dimensional coordinate data falling in each region and the start time point of each checkpoint is taken as the first fixation time of each region;

[0020] Based on the original eye movement coordinate data, the duration during which the two-dimensional coordinate data of the fixation point first falls into each region is taken as the first fixation duration for each region;

[0021] The time interval during which the two-dimensional coordinate data of the fixation point continuously falls within a single region is defined as one fixation count.

[0022] Based on the original eye-tracking coordinate data, the total number of fixations within each level duration is determined, and the ratio of the total fixation duration to the total number of fixations in each region is taken as the average fixation time for each region.

[0023] The ratio of the total number of fixations to the duration of the level in each region is taken as the fixation frequency of each region.

[0024] Based on the original eye movement coordinate data, the duration of the subject's fixation on each region during the experiment, as a percentage of the total experiment time, is used as the fixation percentage for each region.

[0025] According to the user behavior classification method based on eye-tracking data provided in this disclosure, the eye-tracking classification network includes an image feature encoding module, a temporal feature extraction module, and a feature fusion prediction module;

[0026] The image feature encoding module is used to generate eye-tracking heatmap data, and input the eye-tracking heatmap data into a convolutional neural network to obtain the eye-tracking image feature vector of the eye-tracking heatmap data;

[0027] The temporal feature extraction module is used to extract eye movement temporal feature vectors of all the original eye movement coordinate data within a fixed time period through a temporal neural network.

[0028] The feature fusion prediction module is used to concatenate the eye movement image feature vector, the eye movement temporal feature vector, and the eye movement feature data to obtain concatenated data; and to process the concatenated data through a multilayer perceptron network to obtain the classification result.

[0029] According to the user behavior classification method based on eye-tracking data provided in this disclosure, the eye-tracking heatmap data is generated in the following manner:

[0030] The original eye-tracking coordinate data is converted from the global coordinate system to the image coordinate system to obtain the coordinate information of the original eye-tracking coordinate data on the eye-tracking task scene image;

[0031] Based on the coordinate information, color is applied to the eye-tracking task scene image to obtain the eye-tracking heatmap data.

[0032] According to the user behavior classification method based on eye-tracking data provided in this disclosure, the convolutional neural network includes a first input layer, an encoding layer, an unfolding layer, and a decoding layer;

[0033] The first input layer is used to perform pooling processing on the eye-tracking heatmap data to obtain a pooled image;

[0034] The encoding layer is used to perform convolution processing on the pooled image to obtain convolutional image features;

[0035] The unfolding layer is used to unfold the convolutional image features into one-dimensional latent vector features; the one-dimensional latent vector features are the eye-tracking image feature vectors.

[0036] The decoding layer is used to recover the eye movement image feature vector into the eye movement heatmap data.

[0037] According to the user behavior classification method based on eye-tracking data provided in this disclosure, the temporal neural network includes a second input layer, a hidden layer, and an output layer;

[0038] The second input layer is used to process the local eye movement heatmap obtained by visualizing multiple frames of continuous raw eye movement coordinate data to obtain an intermediate feature vector;

[0039] The hidden layer is used to pass the intermediate feature vector through two temporal neural network units to obtain an output sequence;

[0040] The output layer is used to pass the output sequence through a temporal neural network unit to obtain the eye-tracking temporal feature vector.

[0041] This disclosure provides a user behavior classification device based on eye-tracking data, including:

[0042] The acquisition module is used to acquire eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios;

[0043] The classification module is used to input the eye-tracking data into the classification model and obtain the classification result of the user behavior output by the classification model. The classification model is trained based on the eye-tracking classification network, using the original eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data for classification.

[0044] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the user behavior classification method based on eye-tracking data as described above.

[0045] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the user behavior classification method based on eye-tracking data as described above.

[0046] This disclosure provides a user behavior classification method, apparatus, device, and medium based on eye-tracking data. It acquires eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios; inputs the eye-tracking data into a classification model to obtain the classification results of the user behavior output by the model; the classification model is trained on an eye-tracking classification network based on original eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data used for classification. This disclosure combines eye-tracking technology and machine learning algorithms to classify user behavior based on eye-tracking data, thereby avoiding the strong subjectivity and low accuracy problems of manual classification and improving the accuracy of user behavior classification. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 is a flowchart illustrating the user behavior classification method based on eye-tracking data provided in this disclosure.

[0049] Figure 2 is a schematic diagram of the structure of the user behavior classification device based on eye-tracking data provided in this disclosure.

[0050] Figure 3 is a schematic diagram of the structure of the electronic device provided in this disclosure. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0052] Eye-tracking technology, as an objective and quantitative research method, has been widely used in fields such as psychology and neuroscience. Eye-tracking data can reflect an individual's cognitive, emotional, and attentional mechanisms during visual information processing, providing new possibilities for the diagnosis of autism. Individuals with autism exhibit significant abnormalities in visual attention, face recognition, and eye contact; these abnormalities can be objectively measured and analyzed using eye-tracking data.

[0053] Currently, although some studies have attempted to use eye-tracking data for autism diagnosis, these diagnostic methods are limited to specific experimental tasks or scenarios and lack systematicity and universality. Therefore, this disclosure aims to combine eye-tracking technology and machine learning algorithms to propose a user behavior classification method based on eye-tracking data, which can achieve rapid and accurate identification of autism patients.

[0054] It should be noted that the user behavior classification method based on eye-tracking data proposed in this disclosure is implemented entirely by a computer or other device.

[0055] The following describes the user behavior classification method, apparatus, device, and medium based on eye-tracking data disclosed herein, with reference to Figures 1-3.

[0056] Figure 1 is a flowchart of the user behavior classification method based on eye-tracking data provided in this disclosure. As shown in Figure 1, the method includes the following steps 101 and 102.

[0057] Step 101: Obtain eye-tracking data generated by the user performing human-computer interaction tasks in an eye-tracking scenario.

[0058] Eye-tracking scenes are virtual scenes that include three elements: virtual host, virtual objects, and virtual backgrounds. Virtual hosts include two types: anthropomorphic animal models and human models; virtual objects include realistic animal models and everyday object models; and virtual backgrounds include everyday spaces such as kindergartens, grasslands, and rooms.

[0059] The human-computer interaction task consists of multiple levels, each corresponding to a different interactive task. For example, in a virtual environment formed by eye-tracking, the human-computer interaction task includes five levels in which the subject interacts with objects in the scene. For each level, a virtual facilitator introduces the task content, the participant then performs the interactive task, and finally the virtual facilitator verbally encourages the participant. The total training time is 10 minutes.

[0060] By designing eye-tracking scenarios and human-computer interaction tasks, the system can provide diverse visual stimuli and interactive experiences, thereby collecting richer and more accurate eye-tracking data.

[0061] By using Unreal Engine (UE) to build eye-tracking scenes, users can track their eye movements in real time through head-mounted devices or cameras, collecting eye-tracking data generated when performing human-computer interaction tasks within the scene. This eye-tracking data includes binocular coordinate data, gaze direction data, 2D coordinate data of the gaze point, eye closure degree data, blink frequency data, and eye-tracking images or video data.

[0062] Step 102: Input the eye-tracking data into the classification model to obtain the classification results of user behavior output by the classification model.

[0063] The collected eye-tracking data is input into a pre-trained classification model to obtain the classification results of user behavior. It can be understood that user behavior refers to the abnormal behaviors exhibited by individuals with autism. Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder characterized by abnormal behavioral patterns including abnormal language abilities, abnormal social skills, restricted interests, and obsessive-compulsive tendencies. The classification results output by the model can be used to determine the type of autism, such as mild, moderate, severe, or no risk.

[0064] The classification model is trained on top of an eye-tracking classification network, using raw eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data for classification. The raw eye-tracking coordinate data includes the subject's binocular coordinates, gaze direction data, gaze point 2D coordinates, and eye closure degree data. The eye-tracking task scene images refer to 2D images rendered from the eye-tracking scene during the human-computer interaction task, generated through mathematical calculations.

[0065] This disclosure provides a user behavior classification method based on eye-tracking data. It acquires eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios; inputs the eye-tracking data into a classification model to obtain the classification results of the user behavior output by the model; the classification model is trained on an eye-tracking classification network based on raw eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data used for classification. This disclosure combines eye-tracking technology and machine learning algorithms to classify user behavior based on eye-tracking data, thereby avoiding the strong subjectivity and low accuracy problems of manual classification and improving the accuracy of user behavior classification.

[0066] Based on the above embodiments, training the classification model includes the following steps 121 and 123.

[0067] Step 121: Collect the raw eye-tracking coordinate data and the eye-tracking task scene image generated by the subject performing the human-computer interaction task in the eye-tracking scene;

[0068] Step 122: Determine the eye movement feature data based on the original eye movement coordinate data and the eye movement task scene image;

[0069] Step 123: Using the original eye-tracking coordinate data, the eye-tracking task scene image, and the eye-tracking feature data, train the eye-tracking classification network to obtain the classification model.

[0070] Raw eye-tracking coordinate data and scene images were collected from subjects performing human-computer interaction tasks within an eye-tracking environment. These data were preprocessed to obtain eye-tracking feature data for classification. The eye-tracking feature data included total fixation duration, first fixation time, first fixation duration, average fixation time, fixation frequency, and fixation percentage. The determination method for each eye-tracking feature data is as follows:

[0071] Based on the eye-tracking task scene images, the eye-tracking scene is divided into multiple regions. For example, using an object recognition network, the bounding box positions of objects in the eye-tracking task scene image are identified, and the area enclosed by the bounding box is considered as one region. Based on the original eye-tracking coordinate data, the duration for which the fixation point's 2D coordinate data falls within each region in each level is calculated. The sum of the durations of each region is then taken as the total fixation duration for each region. Based on the original eye-tracking coordinate data, the time difference between the first fixation time when the fixation point's 2D coordinate data falls within each region and the start time of each level is calculated. This time difference is taken as the first fixation time for each region. Based on the original eye-tracking coordinate data, the duration for which the fixation point's 2D coordinate data first falls within each region is calculated. This duration is taken as the first fixation duration for each region. The time interval during which the two-dimensional coordinate data of the fixation point continuously falls within a single region is taken as one fixation count. Then, based on the original eye movement coordinate data, the total number of fixations within each level duration is calculated. The ratio of the total fixation duration to the total number of fixations in each region is taken as the average fixation time for each region. The ratio of the total number of fixations to the level duration in each region is taken as the fixation frequency for each region. Based on the original eye movement coordinate data, the proportion of time the subject fixates on each region during the experiment to the total time of the experiment is calculated, and this proportion is taken as the fixation percentage for each region.

[0072] An eye-tracking classification network was trained using raw eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data to obtain a classification model. The eye-tracking classification network includes an image feature encoding module, a temporal feature extraction module, and a feature fusion prediction module. The image feature encoding module generates eye-tracking heatmap data, which is then input into a convolutional neural network to obtain eye-tracking image feature vectors. The temporal feature extraction module extracts temporal feature vectors from all raw eye-tracking coordinate data within a fixed time period using a temporal neural network. The feature fusion prediction module concatenates the eye-tracking image feature vectors, temporal feature vectors, and eye-tracking feature data to obtain concatenated data. The concatenated data is then processed using a multilayer perceptron network to obtain the classification result.

[0073] Eye-tracking heatmap data is generated as follows: Raw eye-tracking coordinate data is transformed from a global coordinate system to an image coordinate system to obtain its coordinate information on the eye-tracking task scene image. Based on this coordinate information, the image is colored to obtain the eye-tracking heatmap data. For example, the raw eye-tracking coordinate data is transformed from a global coordinate system to an image coordinate system through coordinate axis transformation to obtain its xy coordinate points on the eye-tracking task scene image. These xy coordinate points are then colored; the more times a point falls within a given coordinate, the darker the color. After processing all the raw eye-tracking coordinate data, the final eye-tracking heatmap data is obtained.

[0074] The image feature encoding module takes eye-tracking heatmap data as input and outputs an eye-tracking image feature vector F. conv ={x0,x1,,...,x n} where n is the preset feature vector length parameter of the convolutional neural network. A convolutional neural network includes a first input layer, an encoding layer, an unwinding layer, and a decoding layer.

[0075] The input data for the first input layer is a two-dimensional RGB image of arbitrary size, i.e., an eye-tracking heatmap. The eye-tracking heatmap data can be pooled through the first input layer to output a pooled image of a fixed size.

[0076] The encoding layer consists of several 2D convolutional layers, using ReLU as the activation function, with a kernel size of 2×2 and a stride of 2. It is used to convolve the pooled image to obtain convolutional image features.

[0077] The unfolding layer is used to unfold the features of the convolutional image into one-dimensional latent vector features, i.e., the eye-tracking image feature vector.

[0078] The decoding layer is used to restore the eye-tracking image feature vectors into eye-tracking heatmap data. The decoding layer consists of several 2D convolutional layers, restoring the eye-tracking image feature vectors into a 2D grayscale image of the same size as the original input, i.e., the eye-tracking reconstruction image. Error backpropagation is performed using a pixel-level MSE loss function, which is:

[0079] Where I(i,j) is the grayscale value of a single pixel in the i-th row and j-th column of the eye-tracking heatmap. H is the grayscale value of a single pixel in the i-th row and j-th column of the eye-tracking reconstruction image, H is the height value of the eye-tracking heatmap and the eye-tracking reconstruction image, and W is the width value of the eye-tracking heatmap and the eye-tracking reconstruction image.

[0080] The temporal feature extraction module generates an eye-tracking temporal feature vector F by inputting raw eye-tracking coordinate data. time ={y0,y1,...yn1} where n is the preset feature vector length parameter for the temporal neural network, which is the same as the preset feature vector length parameter for the convolutional neural network. The temporal neural network includes a second input layer, a hidden layer, and an output layer.

[0081] The input data for the second input layer is a local eye-tracking heatmap (I) obtained by visualizing k consecutive frames of raw eye-tracking coordinate data. local The output is an intermediate feature vector;

[0082] The hidden layer is used to process multiple one-dimensional temporal vectors through two temporal neural network units to obtain the output sequence. The hidden layer employs a two-layer network structure, training the temporal neural network layer by layer, feeding the output of the previous hidden layer into the next for computation. After all computations are complete, the output sequence of the last hidden layer is used as the input to the output layer.

[0083] The output layer is used to pass the output sequence through a temporal neural network unit to obtain an eye-tracking temporal feature vector.

[0084] The feature fusion and prediction module concatenates eye-tracking image feature vectors, eye-tracking temporal feature vectors, and eye-tracking feature data used for classification. This concatenation is then processed through a multilayer perceptron network to output the probability that the feature data belongs to the autism category. Feature concatenation and prediction refer to forming a high-dimensional feature vector F = {f1, f2, ..., f...} by concatenating the eye-tracking image feature vectors, eye-tracking temporal feature vectors, and eye-tracking feature data used for classification. n After passing through a fully connected layer and a softmax function, it is mapped to a probability vector P = {p1, p2, ..., p...}. c} is used to assess the probability that a sample belongs to the autism category. Among them,

[0085] Where, p k Let be the k-th value of the probability vector output by the feature fusion prediction module, and c be the number of categories in the prediction task. The probability vector P contains c elements, each representing the probability of a category. In this embodiment, the categories include no risk, mild autism, moderate autism, and severe autism. The category with the highest value in the probability vector P is the predicted evaluation sample category. Cross-entropy loss is calculated between the actual evaluation sample category and the predicted evaluation sample category, and gradient optimization is used for training optimization. The formula for calculating cross-entropy loss is:

[0086] Where N is the number of samples participating in a single training iteration, s i To evaluate the classification of the i-th sample in a single training iteration, The classification of the actual evaluation sample for the i-th sample participating in a single training session.

[0087] This disclosure discloses an embodiment that classifies autism by training a classification model, thereby classifying user behavior based on eye-tracking data. This avoids the problems of strong subjectivity and low accuracy associated with manual classification, and improves the accuracy of user behavior classification.

[0088] To further explain the user behavior classification method based on eye-tracking data proposed in this disclosure, this disclosure specifically proposes an autism classification and diagnosis system based on eye-tracking data. The system includes a data acquisition module, a feature value acquisition module, and a classification model training module.

[0089] First, a virtual scene, namely an eye-tracking scene, is built using the UE engine. Then, raw eye-tracking coordinate data and scene images generated by the subject performing human-computer interaction tasks in the eye-tracking scene are collected. Specifically, the raw eye-tracking coordinate data in the data acquisition module is acquired using a Pico 4 Enterprise VR device with a sampling frequency of 60Hz.

[0090] In one embodiment, the subject wears a Pico 4 Enterprise VR device. After five-point eye-tracking calibration, an eye-tracking scenario is initiated, and the subject sequentially completes five levels of interaction with objects within the scenario. Each level is introduced by a virtual presenter, followed by the participant's interactive task. The virtual presenter then verbally encourages the participant. Each level lasts two minutes, and the total duration of the scenario is 10 minutes. During this scenario, the Pico 4 Enterprise VR device collects the subject's raw eye-tracking coordinate data and images of the eye-tracking task scenario, and saves them to a storage device.

[0091] The feature acquisition module preprocesses the collected raw eye-tracking coordinate data and eye-tracking task scene image data, using the processed data as eye-tracking feature data for classification. In one embodiment, the human-computer interaction task involves a virtual host waving and greeting the subject. The virtual host is positioned directly opposite the subject, 120 cm away, attracting the subject's attention through modal information such as voice, gestures, and facial expressions, and issuing a voice command requesting the subject to say their name. After the virtual host finishes issuing the voice command, there is a 10-second wait before proceeding to the next human-computer interaction task. This human-computer interaction task lasts for 1 minute. The feature acquisition module acquires the raw eye-tracking coordinate data within 1 minute. To quantify the participants' gaze data during this process and extract effective features that can be used for autism classification, the module processes the raw eye-tracking coordinate data and eye-tracking task scene image data to generate eye-tracking feature data for classification.

[0092] The data processing steps are as follows: The eye-tracking scene is divided into three regions: the virtual host's face region, the virtual host's body region, and the background region. Based on the original eye-tracking coordinate data, the total fixation duration is calculated by summing the durations at which the subject's gaze point 2D coordinate data falls in each region. The first fixation time is calculated by the time difference between the first fixation time and the start time of the level. The duration at which the subject's gaze point 2D coordinate data falls in each region is also calculated. The number of fixations is calculated by dividing the time period at which the subject's gaze point 2D coordinate data falls in a single region. The total number of fixations within the level duration is calculated. The average fixation time is calculated by dividing the total fixation duration in each region by the total number of fixations. The fixation frequency is calculated by dividing the total number of fixations in each region by the level duration. The percentage of time the subject spends fixating on a particular region in the entire experiment is calculated by dividing the total fixation duration by the level duration. The subject's gaze point 2D coordinate data is projected onto the eye-tracking task scene image data and colored to form an eye-tracking heatmap.

[0093] By utilizing an eye-tracking classification network structure, eye-tracking image feature vectors, eye-tracking temporal feature vectors, and eye-tracking feature data used for classification are concatenated and used for network parameter training. The trained network parameters are then used to build a classification model. Based on the established classification model, the system can automatically classify and identify autism in unknown children.

[0094] The eye-tracking classification network structure includes: an image feature encoding module, a temporal feature extraction module, and a feature fusion prediction module. The image feature encoding module generates an eye-tracking heatmap and processes the heatmap data into a convolutional neural network to obtain eye-tracking image feature vectors. The temporal feature extraction module extracts features from the raw eye-tracking coordinate data, using a temporal neural network to extract temporal feature vectors from all raw eye-tracking coordinate data within a fixed time period. The feature fusion prediction module concatenates the eye-tracking image feature vectors, temporal feature vectors, and eye-tracking feature data used for classification, then processes this concatenation through a multilayer perceptron network to output the probability that the feature data belongs to the autism category.

[0095] First, the eye-tracking heatmap data required by the image feature encoding module is generated. The calculation method is as follows: the raw eye-tracking coordinate data is transformed from the global coordinate system to the image coordinate system through coordinate axis transformation to obtain the x and y coordinate points of the eye-tracking data on the eye-tracking task scene image. Then, these x and y coordinate points on the eye-tracking task scene image are colored; the more times the data falls on a particular coordinate point, the darker the color. After processing all the raw eye-tracking coordinate data, the final eye-tracking heatmap data is obtained.

[0096] In one embodiment, the image feature encoding module includes a single Auto Encoder convolutional neural network that outputs eye-tracking image feature vectors from input eye-tracking heatmap data. The Auto Encoder convolutional neural network structure includes an input layer, an encoding layer, an unwinding layer, and a decoding layer. The input layer takes a two-dimensional RGB image of arbitrary size as input and outputs a pooled image of fixed size via a pooling layer. The encoding layer consists of several 2D convolutional layers using ReLU as the activation function, with a 2×2 kernel size and a stride of 2, processing the pooled image into convolutional image features. The unwinding layer expands the convolutional image features output from the encoding layer into one-dimensional latent vector features, i.e., the eye-tracking image feature vector, which is subsequently used as one of the inputs to the feature fusion prediction module. The decoding layer consists of several 2D convolutional layers that restore the eye-tracking image feature vectors to the original input two-dimensional RGB image and uses a pixel-level MSE loss function for error backpropagation.

[0097] In one embodiment, the temporal feature extraction module includes a single LSTM network structure. It generates an eye-movement temporal feature vector by inputting eye-movement feature data for classification and raw eye-movement coordinate data. The LSTM network structure includes an input layer, hidden layers, and an output layer. The input layer's input data is formed by concatenating the raw eye-movement coordinate data into a one-dimensional temporal vector. Then, data within the current time period is sampled to obtain n one-dimensional temporal vectors, meeting the input requirements of the LSTM network. For the hidden layer, a two-layer network structure is used for processing, training the LSTM layer by layer. The output of the previous LSTM hidden layer is fed into the corresponding LSTM neuron of the next LSTM hidden layer for computation. After all computations are completed, the value of the last hidden layer's output sequence is taken as the input of the output layer. The output layer contains a single temporal neural network unit, ultimately outputting the eye-movement temporal feature vector.

[0098] The feature fusion and prediction module concatenates eye-tracking image feature vectors, eye-tracking temporal feature vectors, and eye-tracking feature data used for classification. This concatenation is then processed through a multilayer perceptron network to output the probability that the feature data belongs to the autism category. Feature concatenation and prediction refer to forming a high-dimensional feature vector F = {f1, f2, ..., f...} by concatenating the eye-tracking image feature vectors, eye-tracking temporal feature vectors, and eye-tracking feature data used for classification. n After passing through a fully connected layer and a softmax function, it is mapped to a probability vector P = {p1, p2, ..., p...}. c} is used to assess the probability that a sample belongs to the autism category. Among them,

[0099] Where, p kLet be the k-th value of the probability vector output by the feature fusion prediction module, and c be the number of categories in the prediction task. The probability vector P contains c elements, each representing the probability of a category. In this embodiment, the categories include no risk, mild autism, moderate autism, and severe autism. The category with the highest value in the probability vector P is the predicted evaluation sample category. Cross-entropy loss is calculated between the actual evaluation sample category and the predicted evaluation sample category, and gradient optimization is used for training optimization. The formula for calculating cross-entropy loss is:

[0100] Where N is the number of samples participating in a single training iteration, s i To evaluate the classification of the i-th sample in a single training iteration, The classification of the actual evaluation sample for the i-th sample participating in a single training session.

[0101] The temporal feature extraction module and the feature fusion prediction module employ methods such as 5-fold cross-validation to evaluate the model's classification accuracy under different parameter combinations. Specifically, one-fifth of the dataset is used as the training set, and the remaining data is used as the test set. The network parameters are initialized and training begins. This process is repeated five times until every evaluation sample is used as a test object. Simultaneously, methods such as grid search are used to find the optimal hyperparameter combination, such as the learning rate and the number of iterations, within a preset parameter range.

[0102] This disclosure discloses embodiments that collect and analyze eye movement data from autistic patients and healthy controls, extract key indicators that reflect the characteristics of autism, and then construct a classification model to achieve automatic identification and classification of autistic patients, which helps to improve the accuracy and objectivity of autism diagnosis.

[0103] The user behavior classification device based on eye-tracking data provided in this disclosure is described below. The user behavior classification device based on eye-tracking data described below can be referred to in correspondence with the user behavior classification method based on eye-tracking data described above.

[0104] Referring to Figure 2, the user behavior classification device based on eye-tracking data includes an acquisition module 201 and a classification module 202.

[0105] The acquisition module 201 is used to acquire eye-tracking data generated by the user performing human-computer interaction tasks in an eye-tracking scenario;

[0106] The classification module 202 is used to input the eye-tracking data into the classification model and obtain the classification result of the user behavior output by the classification model; the classification model is trained on the basis of the eye-tracking classification network, based on the original eye-tracking coordinate data, eye-tracking task scene images and eye-tracking feature data for classification.

[0107] This disclosure provides a user behavior classification device based on eye-tracking data. It acquires eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios; inputs the eye-tracking data into a classification model to obtain the classification results of the user behavior output by the model; the classification model is trained on an eye-tracking classification network based on original eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data used for classification. This disclosure combines eye-tracking technology and machine learning algorithms to classify user behavior based on eye-tracking data, thereby avoiding the strong subjectivity and low accuracy problems of manual classification and improving the accuracy of user behavior classification.

[0108] In one embodiment, the user behavior classification device based on eye-tracking data further includes a training module for:

[0109] The raw eye-tracking coordinate data and the eye-tracking task scene images generated by the subject performing the human-computer interaction task in the eye-tracking scene were collected.

[0110] The eye movement feature data is determined based on the original eye movement coordinate data and the eye movement task scene image;

[0111] The original eye-tracking coordinate data, the eye-tracking task scene image, and the eye-tracking feature data are used to train the eye-tracking classification network to obtain the classification model.

[0112] The eye-tracking scene is a virtual scene that includes a virtual host, virtual objects, and a virtual background; the human-computer interaction task includes multiple levels, each level corresponding to a different interaction task.

[0113] In one embodiment, the training module is further configured to:

[0114] Based on the eye-tracking task scene image, the eye-tracking scene is divided into multiple regions;

[0115] Based on the original eye movement coordinate data, the sum of the durations during which the two-dimensional coordinate data of the fixation point in each of the levels falls within each of the regions is taken as the total fixation duration for each region.

[0116] Based on the original eye movement coordinate data, the time difference between the first fixation time point of the fixation point two-dimensional coordinate data falling in each region and the start time point of each checkpoint is taken as the first fixation time of each region;

[0117] Based on the original eye movement coordinate data, the duration during which the two-dimensional coordinate data of the fixation point first falls into each region is taken as the first fixation duration for each region;

[0118] The time interval during which the two-dimensional coordinate data of the fixation point continuously falls within a single region is defined as one fixation count.

[0119] Based on the original eye-tracking coordinate data, the total number of fixations within each level duration is determined, and the ratio of the total fixation duration to the total number of fixations in each region is taken as the average fixation time for each region.

[0120] The ratio of the total number of fixations to the duration of the level in each region is taken as the fixation frequency of each region.

[0121] Based on the original eye movement coordinate data, the duration of the subject's fixation on each region during the experiment, as a percentage of the total experiment time, is used as the fixation percentage for each region.

[0122] In one embodiment, the eye-tracking classification network includes an image feature encoding module, a temporal feature extraction module, and a feature fusion prediction module;

[0123] The image feature encoding module is used to generate eye-tracking heatmap data, and input the eye-tracking heatmap data into a convolutional neural network to obtain the eye-tracking image feature vector of the eye-tracking heatmap data;

[0124] The temporal feature extraction module is used to extract eye movement temporal feature vectors of all the original eye movement coordinate data within a fixed time period through a temporal neural network.

[0125] The feature fusion prediction module is used to concatenate the eye movement image feature vector, the eye movement temporal feature vector, and the eye movement feature data to obtain concatenated data; and to process the concatenated data through a multilayer perceptron network to obtain the classification result.

[0126] In one embodiment, the training module is further configured to:

[0127] The original eye-tracking coordinate data is converted from the global coordinate system to the image coordinate system to obtain the coordinate information of the original eye-tracking coordinate data on the eye-tracking task scene image;

[0128] Based on the coordinate information, color is applied to the eye-tracking task scene image to obtain the eye-tracking heatmap data.

[0129] In one embodiment, the convolutional neural network includes a first input layer, an encoding layer, an unfolding layer, and a decoding layer;

[0130] The first input layer is used to perform pooling processing on the eye-tracking heatmap data to obtain a pooled image;

[0131] The encoding layer is used to perform convolution processing on the pooled image to obtain convolutional image features;

[0132] The unfolding layer is used to unfold the convolutional image features into one-dimensional latent vector features; the one-dimensional latent vector features are the eye-tracking image feature vectors.

[0133] The decoding layer is used to recover the eye movement image feature vector into the eye movement heatmap data.

[0134] In one embodiment, the temporal neural network includes a second input layer, a hidden layer, and an output layer;

[0135] The second input layer is used to process the local eye movement heatmap obtained by visualizing multiple frames of continuous raw eye movement coordinate data to obtain an intermediate feature vector;

[0136] The hidden layer is used to pass the intermediate feature vector through two temporal neural network units to obtain an output sequence;

[0137] The output layer is used to pass the output sequence through a temporal neural network unit to obtain the eye-tracking temporal feature vector.

[0138] Figure 3 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 3, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a user behavior classification method based on eye-tracking data. This method includes: acquiring eye-tracking data generated by a user performing a human-computer interaction task in an eye-tracking scene; inputting the eye-tracking data into a classification model to obtain the classification result of the user behavior output by the classification model; the classification model is trained on an eye-tracking classification network based on original eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data used for classification.

[0139] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] On the other hand, this disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the user behavior classification method based on eye-tracking data provided by the above methods. The method includes: acquiring eye-tracking data generated by a user performing a human-computer interaction task in an eye-tracking scene; inputting the eye-tracking data into a classification model to obtain the classification result of the user behavior output by the classification model; the classification model is trained on an eye-tracking classification network based on original eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data used for classification.

[0141] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the user behavior classification method based on eye-tracking data provided by the above methods. The method includes: acquiring eye-tracking data generated by a user performing a human-computer interaction task in an eye-tracking scene; inputting the eye-tracking data into a classification model to obtain a classification result of the user behavior output by the classification model; wherein the classification model is trained on an eye-tracking classification network based on original eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data used for classification.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure. Industrial applicability

[0145] This disclosure provides a user behavior classification method, apparatus, device, and medium based on eye-tracking data. It acquires eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios; inputs the eye-tracking data into a classification model to obtain the classification results of the user behavior output by the model; the classification model is trained on an eye-tracking classification network based on original eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data used for classification. This disclosure combines eye-tracking technology and machine learning algorithms to classify user behavior based on eye-tracking data, thereby avoiding the strong subjectivity and low accuracy problems of manual classification and improving the accuracy of user behavior classification.

[0146] Furthermore, it is understood that the user behavior classification method, apparatus, device, and medium based on eye-tracking data provided in this disclosure are reproducible and can be used in various industrial applications. For example, the user behavior classification method, apparatus, device, and medium based on eye-tracking data provided in this disclosure can be used in the field of circuit control technology.

Claims

1. A user behavior classification method based on eye-tracking data, characterized in that, include: Acquire eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios; The eye-tracking data is input into the classification model to obtain the classification results of user behavior output by the classification model; The classification model is trained based on the eye-tracking classification network, using raw eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data for classification.

2. The user behavior classification method based on eye-tracking data according to claim 1, characterized in that, The classification model was trained using the following method: The raw eye-tracking coordinate data and the eye-tracking task scene images generated by the subject performing the human-computer interaction task in the eye-tracking scene were collected. The eye movement feature data is determined based on the original eye movement coordinate data and the eye movement task scene image; The original eye-tracking coordinate data, the eye-tracking task scene image, and the eye-tracking feature data are used to train the eye-tracking classification network to obtain the classification model. The eye-tracking scene is a virtual scene that includes a virtual host, virtual objects, and a virtual background; the human-computer interaction task includes multiple levels, each level corresponding to a different interaction task.

3. The user behavior classification method based on eye-tracking data according to claim 2, characterized in that, The raw eye movement coordinate data includes binocular coordinate data, gaze direction data, two-dimensional coordinate data of the gaze point, and eye closure degree data; the eye movement feature data includes total gaze duration, first gaze duration, first gaze duration, average gaze duration, gaze frequency, and gaze percentage. The step of determining the eye movement feature data based on the original eye movement coordinate data and the eye movement task scene image includes: Based on the eye-tracking task scene image, the eye-tracking scene is divided into multiple regions; Based on the original eye movement coordinate data, the sum of the durations during which the two-dimensional coordinate data of the fixation point in each of the levels falls within each of the regions is taken as the total fixation duration for each region. Based on the original eye movement coordinate data, the time difference between the first fixation time point of the fixation point two-dimensional coordinate data falling in each region and the start time point of each checkpoint is taken as the first fixation time of each region; Based on the original eye movement coordinate data, the duration during which the two-dimensional coordinate data of the fixation point first falls into each region is taken as the first fixation duration for each region; The time interval during which the two-dimensional coordinate data of the fixation point continuously falls within a single region is defined as one fixation count. Based on the original eye-tracking coordinate data, the total number of fixations within each level duration is determined, and the ratio of the total fixation duration to the total number of fixations in each region is taken as the average fixation time for each region. The ratio of the total number of fixations to the duration of the level in each region is taken as the fixation frequency of each region. Based on the original eye movement coordinate data, the duration of the subject's fixation on each region during the experiment, as a percentage of the total experiment time, is used as the fixation percentage for each region.

4. The user behavior classification method based on eye-tracking data according to claim 2, characterized in that, The eye-tracking classification network includes an image feature encoding module, a temporal feature extraction module, and a feature fusion prediction module; The image feature encoding module is used to generate eye-tracking heatmap data, and input the eye-tracking heatmap data into a convolutional neural network to obtain the eye-tracking image feature vector of the eye-tracking heatmap data; The temporal feature extraction module is used to extract eye movement temporal feature vectors of all the original eye movement coordinate data within a fixed time period through a temporal neural network. The feature fusion prediction module is used to concatenate the eye movement image feature vector, the eye movement temporal feature vector, and the eye movement feature data to obtain concatenated data; and to process the concatenated data through a multilayer perceptron network to obtain the classification result.

5. The user behavior classification method based on eye-tracking data according to claim 4, characterized in that, The eye-tracking thermal image data was generated based on the following method: The original eye-tracking coordinate data is converted from the global coordinate system to the image coordinate system to obtain the coordinate information of the original eye-tracking coordinate data on the eye-tracking task scene image; Based on the coordinate information, color is applied to the eye-tracking task scene image to obtain the eye-tracking heatmap data.

6. The user behavior classification method based on eye-tracking data according to claim 4, characterized in that, The convolutional neural network includes a first input layer, an encoding layer, an unfolding layer, and a decoding layer; The first input layer is used to perform pooling processing on the eye-tracking heatmap data to obtain a pooled image; The encoding layer is used to perform convolution processing on the pooled image to obtain convolutional image features; The unfolding layer is used to unfold the convolutional image features into one-dimensional latent vector features; the one-dimensional latent vector features are the eye-tracking image feature vectors. The decoding layer is used to recover the eye movement image feature vector into the eye movement heatmap data.

7. The user behavior classification method based on eye-tracking data according to claim 4, characterized in that, The temporal neural network includes a second input layer, a hidden layer, and an output layer; The second input layer is used to process the local eye movement heatmap obtained by visualizing multiple frames of continuous raw eye movement coordinate data to obtain an intermediate feature vector; The hidden layer is used to pass the intermediate feature vector through two temporal neural network units to obtain an output sequence; The output layer is used to pass the output sequence through a temporal neural network unit to obtain the eye-tracking temporal feature vector.

8. A user behavior classification device based on eye-tracking data, characterized in that, include: The acquisition module is used to acquire eye-tracking data generated by users performing human-computer interaction tasks in eye-tracking scenarios; The classification module is used to input the eye-tracking data into the classification model and obtain the classification results of user behavior output by the classification model; The classification model is trained based on the eye-tracking classification network, using raw eye-tracking coordinate data, eye-tracking task scene images, and eye-tracking feature data for classification.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the user behavior classification method based on eye-tracking data as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the user behavior classification method based on eye-tracking data as described in any one of claims 1 to 7.

Images