Eye movement event detection method and system based on attention drive and data segmentation

Abstract

The present invention discloses an eye movement event detection method and system based on attention drive and data segmentation, which relates to the technical field of eye movement event detection. The eye movement data of a target person to be detected within a set time period is obtained and preprocessed; the eye movement data is converted into time series data, input into a hybrid network model AttSeqNet, and an eye movement event detection result is output; wherein, after the time series data is input into the hybrid network model, it first passes through a convolutional neural network module, slides on the sequence data by moving a window, and extracts spatial feature information of the eye movement data; then, it passes through a bidirectional long short-term memory BLSTM module, extracts temporal feature information of the eye movement data, and then inputs the extracted spatial and temporal features into an attention mechanism module to generate a new spatial context information feature vector, and finally outputs the final probability distribution through softmax to realize the detection of eye movement events.

Description

Technical Field

[0001] The present disclosure relates to the technical field of eye movement event detection, and in particular to an eye movement event detection method and system based on attention drive and data segmentation. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Parkinson's disease (PD) is a prevalent neurodegenerative disorder. Eye movements have been identified as a potential biomarker for PD, with studies revealing that PD patients have different eye movement patterns compared to healthy controls. These studies have focused on extracting and analyzing eye movement features to help aid in the identification of PD. In particular, smooth pursuit (SP) eye movement abnormalities are common in PD, prompting more research on SP task abnormalities in PD patients.

[0004] The connection between the eyes and the nervous system is complex. Researchers can capture neural information through eye movements and use various algorithms to detect eye movement events. Eye movement features such as saccade frequency and amplitude are used to investigate visual search impairments in Parkinson's disease patients. In addition, some studies have enhanced the traditional speed threshold algorithm with advanced filters, focusing on capturing eye movement features that indicate changes in cognitive load; some studies have predicted the severity of dyslexia based on fixation and saccadic eye movements.

[0005] Choosing an effective eye movement event detection method is crucial. Existing algorithms mainly rely on manually set thresholds, such as velocity threshold (I-VT) and discrete threshold (I-DT) algorithms, for event classification. However, the increasing manual threshold setting introduces subjectivity.

[0006] In the field of SP detection, an existing adaptive speed threshold algorithm (NH) still requires manual parameter adjustment; the remodnav algorithm extends the NH algorithm, but requires a lot of manual parameter tuning. With the increase of thresholds and the increase of post-processing steps, the complexity of eye movement event detection also increases. In order to simplify these complexities, people have begun to turn to machine learning models for comprehensive eye movement event detection. From random forest algorithms to convolutional neural networks (CNNs), to hybrid models that combine CNNs with recurrent neural networks (RNNs), the detection capabilities of these methods have been continuously improved. Unsupervised learning is used for SP event discrimination; hybrid network models (1DCNN-BLSTM) are used for automatic detection. Temporal convolutional networks (TCNs) have made progress in providing sample-level SP detection, but sometimes have deficiencies in event-level recognition. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes an eye movement event detection method and system based on attention-driven and data segmentation, and uses an attention-enhanced one-dimensional convolutional neural network and a bidirectional long short-term memory network hybrid network model (AttSeqNet) for eye movement event detection. The architecture introduces an attention module in the 1DCNN-BLSTM structure to automatically learn the appropriate weights of the context feature vector, further improving eye movement event detection.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions:

[0009] Eye movement event detection method based on attention drive and data segmentation, including:

[0010] Obtain the eye movement data of the target person to be detected within a set time period and pre-process it;

[0011] The preprocessed eye movement data is converted into time series data, input into the hybrid network model AttSeqNet, and the eye movement event detection results are output;

[0012] Among them, when the time series data is input into the hybrid network model, it first passes through the convolutional neural network module, slides the moving window on the sequence data, and extracts the spatial feature information of the eye movement data; then, it passes through the bidirectional long short-term memory BLSTM module to extract the temporal feature information of the eye movement data, and then the extracted spatial and temporal features are input into the attention mechanism module to generate a new spatial context information feature vector, and finally the final probability distribution is output through softmax to realize the detection of eye movement events.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions:

[0014] The eye movement event detection system based on attention drive and data segmentation includes:

[0015] A data acquisition module is used to acquire the eye movement data of the target person to be detected within a set time period and pre-process it;

[0016] The detection module is used to convert the preprocessed eye movement data into time series data, input it into the hybrid network model AttSeqNet, and output the eye movement event detection results;

[0017] Among them, when the time series data is input into the hybrid network model, it first passes through the convolutional neural network module, slides the moving window on the sequence data, and extracts the spatial feature information of the eye movement data; then, it passes through the bidirectional long short-term memory BLSTM module to extract the temporal feature information of the eye movement data, and then the extracted spatial and temporal features are input into the attention mechanism module to generate a new spatial context information feature vector, and finally the final probability distribution is output through softmax to realize the detection of eye movement events.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the attention-driven and data-segmented eye movement event detection method is implemented.

[0020] According to some embodiments, the present disclosure adopts the following technical solutions:

[0021] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the eye movement event detection method that realizes the attention drive and data segmentation.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The eye movement event detection method based on attention drive and data segmentation disclosed in the present invention makes up for the defect of strong subjectivity of the existing threshold-based eye movement event detection algorithm, and can realize automatic recognition and end-to-end detection of eye movement events.

[0024] The smooth pursuit detection effect obtained by the architecture proposed in this disclosure is better than other existing algorithms. The introduction of the attention mechanism further improves the detection performance of the model, automatically assigns appropriate weights to the context feature vector, and improves the detection accuracy of smooth pursuit events. The introduction of a time-based data segmentation strategy can alleviate the overfitting caused by the high imbalance of the data set, reasonably segment the training set and the test set, and use the newly divided data set to train the model, which further improves the detection effect of eye movement events.

[0025] The present disclosure can be used to detect abnormal eye movement patterns of Parkinson's patients, calculate eye movement features based on eye movement events output by the model, better identify smooth pursuit events from fixation and saccade events, and study the differences in eye movement patterns between Parkinson's patients and healthy people when chasing moving targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0027] Figure 1 A data collection experiment scene diagram of an embodiment of the present disclosure;

[0028] Figure 2 This is a model architecture diagram of an embodiment of the present disclosure;

[0029] Figure 3 A comparison diagram of eye movement coordinate data before and after filtering according to an embodiment of the present disclosure;

[0030] Figure 4 is the position of the viewpoint on the screen of the embodiment of the present disclosure;

[0031] Figure 5 A method for calculating event matching IoU in an embodiment of the present disclosure;

[0032] Figure 6 0-3 represents four eye movement events: fixation, saccade, smooth pursuit and noise. DETAILED DESCRIPTION

[0033] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0036] Example 1

[0037] In one embodiment of the present disclosure, a method for detecting eye movement events based on attention drive and data segmentation is provided, comprising:

[0038] Step 1: The input data used to train the model is a public dataset, and a time-based data segmentation strategy needs to be applied to divide it into a training set and a test set;

[0039] Step 2: Obtain the subject's eye movement data and preprocess it, extract features from the processed data, and convert it into time feature series data for model input;

[0040] Step 3: After the time series data is input into the hybrid network model, it first passes through the convolutional neural network module, slides the moving window on the sequence data, and extracts the spatial feature information of the eye movement data; then, it passes through the bidirectional long short-term memory BLSTM module to extract the temporal feature information of the eye movement data;

[0041] Step 4: Input the extracted spatial and temporal features into the attention mechanism module to generate a new spatial context information feature vector, and then output the final probability distribution through softmax to achieve eye movement event detection;

[0042] Step 5: Use sample-level F1 score, event-level F1 score, and IoU to evaluate the performance of the model.

[0043] As an embodiment, the specific implementation process of the eye movement event detection method based on attention drive and data segmentation disclosed in the present invention is as follows:

[0044] First, the methods of data acquisition include:

[0045] 1) If Figure 1 As shown in the figure, four SP experimental scenes of different complexity were designed, namely, straight line, triangle, pentagon and square, to compare the influence of the complexity of the experimental scene on the results. The eye movement data acquisition equipment used Lenovo Xiaoxin Pro 16 and Tobii X2-60 60Hz eye tracking equipment, with a screen resolution of 2560*1600 and a screen size of 16 inches. At the beginning of the experiment, the observer sat about 60cm in front of the screen and clicked the start button. The blue ball moved at a constant speed along the red line. The observer was told to follow the blue ball with his eyes. In order to eliminate the influence of eye fatigue on the experimental results, the experiments were carried out in a random order.

[0046] The collected eye movement data are summarized as follows: The subjects were patients over 50 years old and normal elderly people. Parkinson's patients and healthy people were similar in age, gender and years of education. 124 volunteers were recruited from Parkinson's patients and healthy people. A relatively slow experimental group (Group A) and a relatively fast experimental group (Group B) were set up to compare the effect of the speed of the uniformly moving ball on the experimental results. Most participants only participated in one of the two experiments, and a few participants participated in both groups of experiments. Several groups of data were excluded due to missing eye movement data and excessive noise. Table 1 shows the remaining data used in the experiment. Group A collected approximately 402,661 eye movement data sample points in approximately 1.86 hours, and Group B collected 334,670 eye movement data sample points in approximately 1.55 hours.

[0047] Table 1: Basic information of Parkinson's disease dataset

[0048]

[0049] After constructing the dataset, the data needs to be preprocessed, including:

[0050] Due to the relaxation of eyelids in the elderly, the eyelids open and close less frequently, generating more noise. The spatial accuracy of eye movement data refers to the reliability of the measurement results when the eyes are not moving: although the eyes always move slightly (such as nystagmus or drift), there are variable errors or noise in the signal. One way to estimate the size of the noise is to calculate the position change between samples during fixation.

[0051] For the calculation of the noise level, the present disclosure uses a moving window method to determine the root mean square deviation (RMSD) of each window, and then uses the median RMSD of all windows in each experiment as the RMSD of the experimental data, and then calculates the average of all experimental RMSD as the RMSD (in°) of the data set. Before the calculation, it is assumed that the human eye is located in the middle of the screen, that is, when the angular coordinates of the viewpoint are located in the middle of the screen as 0°, the angular coordinates of other sample points are calculated. The size of the sliding window means that fixations with a duration less than this window size will be excluded.

[0052]

[0053] As shown in formula (1), α H represents the angular range of a single pixel measured along the horizontal direction, α V Represents the same quantity measured in the vertical direction. The horizontal resolution of the screen is represented by Res x Given, and Res y Dim corresponds to the resolution in the vertical direction. x The value range of Dim represents the width of the screen, and y The value range of represents the height. Furthermore, D represents the viewing distance of the screen.

[0054] px2 deg=(α H +α V ) / 2

[0055]

[0056]

[0057] The formula for calculating the noise of eye movement data is (2), where RMS xyRepresents the noise level of a single fixation point, N represents the number of samples contained in the fixation point, x and y represent the horizontal and vertical coordinates of the fixation point on the screen (in pixels), and px2deg represents the number of pixels per degree.

[0058] Since the original data is noisy with signal loss and oscillation, first-order linear interpolation is used for the positions where the signal is missing, and then the interpolated data is filtered using a moving median filter with a window size set to 19. Figure 3 The eye movement data before and after filtering are shown. After the denoising method is used, the average RMSD level of the group B data set is reduced from 0.7748° to 0.1003°, and the average RMSD level of the group A data set is reduced from 0.9299° to 0.1035°. Because the model of the present disclosure is trained on a public data set at a frequency of 250 Hz, the PD data set must be resampled at a frequency of 60 Hz to 250 Hz. The existing first-order spline interpolation method is used to resample the eye movement data to meet the input requirements of the model. The present disclosure uses spline interpolation and then extracts the input data required by the model. Sample interpolation uses a cubic polynomial to fit the curve within the sample points. By constraining the first-order derivative and second-order derivative of the polynomial at the sampling point, the velocity and acceleration characteristics of the sampling point before and after interpolation remain unchanged. In addition, this method will produce relatively smooth interpolation results. The coordinate comparison of the eye movement data of group B before and after filtering is as follows Figure 3 and Figure 4 shown.

[0059] 2) Convert the preprocessed eye movement data into time series data, input it into the hybrid network model AttSeqNet, and output the eye movement event detection results;

[0060] First, the hybrid network model AttSeqNet is trained, including:

[0061] GazeCom is used for training and testing. The dataset includes 5 types of labels: unknown, noise, fixation, saccade, and SP. It receives input consisting of a combination of speed and direction features, which are calculated at 5 different time scales: 4, 8, 16, 32, and 64ms. Specifically, the displacement vector of the gaze position in a time window of each size on the screen is calculated, centered on a given sample. The speed features are expressed in degrees per second (□ / s); however, the direction features are calculated in radians and are relative to the horizontal vector from left to right. Data samples longer than 21 seconds were excluded when training the model because the duration of each video stimulus was only 21 seconds; the eye movement data of one of the observers (SSK) was adjusted from 500Hz to 250Hz.

[0062] A network model was established to automatically and simultaneously detect three eye movement events: fixation, saccade and smooth pursuit. The amplitude abnormalities of saccade and smooth pursuit in Parkinson's patients were analyzed respectively.

[0063] The model was implemented using Tensorflow and Keras (version 2.9.1) and Python (version 3.10). In order to obtain faster convergence speed and higher computational efficiency, the Adam optimizer was used instead of the Rmsprop optimizer to train the model, and the learning rate was 0.001 and the batch size was 5000. Dropout was used to alleviate overfitting, with p = 0.3. After each epoch of training, the model with the smallest loss value of the validation set was retained as the optimal model.

[0064] The dataset is unbalanced and different observers show similar SP characteristics when watching the same video files. Therefore, the choice of cross-validation procedure becomes crucial.

[0065] Leave One Video Out (LOVO) cross-validation was used, which included running 18 training and testing runs. In each video, the model was trained on all the data from 17 videos and then tested on all the eye tracking data collected in the remaining video clips. This approach ensured that the model's eye movement label generator did not see any data examples from the test videos during the training phase. The labeled outputs of all test sets were summarized before the final evaluation. There are disadvantages to using ordinary methods to randomly divide the training and validation sets. In traditional methods, different observers show similar gaze patterns for the same dynamic stimulus scenario. Based on this feature, they used a clustering method to identify SP events. It is also mentioned that overfitting can be alleviated by improving the validation set splitting strategy, which was experimentally verified on the 1DCNN-BLSTM standard model. Therefore, if a random split method is used, randomly extracting 10% of the proportion as the validation set in each original cross-validation process can easily lead to too many similar data patterns in the training and validation sets, resulting in overfitting.

[0066] The present invention improves the division strategy of the training set and the validation set, adopts the time segmentation method, and for the eye movement data collected by all observers corresponding to the same video file, the validation set is composed of the data corresponding to the same time period. In the model, the data window size remains unchanged at about 1s, and the proportion remains unchanged. The similarity between the training set and the validation set is greatly reduced, which reduces overfitting and improves the detection effect of SP.

[0067] Furthermore, the trained hybrid network model AttSeqNe is a network model that combines an attention-driven one-dimensional convolutional neural network with a bidirectional long short-term memory network. The specific application method is as follows:

[0068] When the time series data is input into the hybrid network model, it first passes through the convolutional neural network module, slides the moving window on the sequence data, and extracts the spatial feature information of the eye movement data; then, it passes through the bidirectional long short-term memory BLSTM module to extract the temporal feature information of the eye movement data, and then the extracted spatial and temporal features are input into the attention mechanism module to generate a new spatial context information feature vector, and finally the final probability distribution is output through softmax to realize the detection of eye movement events.

[0069] Specifically, 1D-CNN is a convolutional layer specifically designed for processing sequence data. As shown in formula (3), unlike two-dimensional or three-dimensional convolution, 1D-CNN slides on sequence data by moving the window to extract the spatial feature information of the data. The convolution kernel is connected to the corresponding input symbol x at each point p. p-K+1：p The result of the convolution operation is summed with the convolution kernel parameter w and the bias b, and activated by the ReLU function (max(0, x)), and the output result y of the layer at point p is obtained. p (x). In the model, K is equal to 3. The main purpose of the convolutional layer is to prepare for the subsequent model input, so the pooling layer is not used.

[0070]

[0071] 1D-CNN is used to extract the spatial features of the eye movement feature sequence to prepare for further input into the bidirectional long short-term memory (BLSTM) network. BLSTM includes two different LSTMs that process the input data in a forward and reverse manner to improve accuracy and efficiency. As shown in formula (4), and denote the forward and reverse hidden state vectors at time t, respectively.

[0072]

[0073] An attention mechanism is introduced between the BLSTM layer and the fully connected output layer to adaptively learn the weights of context vectors with different features. The extracted features are input into the attention module to generate new spatial context information features through the following steps.

[0074] The first step is to generate a spatial attention matrix as shown in formula (5) and calculate the similarity score. Where w is the spatial attention matrix, c is the original feature, and their dot product is calculated.

[0075] score(w,c i )= <w,c i > (5)

[0076]

[0077]

[0078] Then, the original score is normalized to a probability distribution where the sum of all feature weights is 1 through softmax, as shown in Equation (6). i represents the normalized attention score, score(w, c i ) represents w and c i The similarity score between .

[0079] Finally, the original feature vectors are weighted and summed to obtain the attention value, as shown in formula (7). i represents the normalized attention score, c i represents the original feature, and the attention value represents the final probability distribution of the feature.

[0080] In AttSeqNet, data passes through an input layer and enters four convolutional layers before passing through two bidirectional recurrent layers. It is then connected to an attention layer before being directed to a fully connected layer to produce the output result. The convolutional layers extract spatial features from the raw input data. The recurrent layers model event sequences to capture temporal features. They detect the start and end times of fixations, saccades, and SPs. Our method can adaptively learn feature weights to improve the feature representation for detecting eye movement events, creating a highly interpretable design.

[0081] Table 3: Model structure and hyperparameters (N - number of event classes)

[0082]

[0083]

[0084] Performance Analysis

[0085] For sample-level evaluation, the multi-class problem is treated as a binary classification problem, and the eye movement event labels detected by the algorithm are compared with the true labels, considering only one eye movement event at a time. For each sample point, there will be hits and misses. At the sample level, assume that the present disclosure wants to predict SP. Recall indicates how many SP events are successfully predicted among all true SP events. Precision indicates how many of the predicted SP events are consistent with the actual events. The f1 score is the harmonic mean of these two indicators and is representative.

[0086] Regarding event-level evaluation, in this approach, each individual predicted sample label is considered for each corresponding event on the ground truth, and the event is classified based on the majority vote of these labels. However, it is worth noting that this approach may oversimplify the eye movement classification problem.

[0087] Different from the method of finding the maximum match, some studies have proposed the method of finding the earliest match. Hooge et al. calculated the event-level evaluation score by searching for the earliest matching event with the same classification as the real event, but this method does not consider the event intersection area, which may lead to inaccuracy. If event A occurs before event B, but the intersection area of ​​event B and the actual event is large, it is still unreasonable to match event B with the actual event.

[0088] To address this issue, an IoU threshold is introduced in the event-level matching calculation. The intersection over union (IoU) ratio between the ground truth eye movement events and the algorithm-labeled events is calculated. Only when the IoU of two matching events is greater than this threshold, it will be marked as a "hit", which is a true positive. If no events meet the criteria, they will be marked as "misses", which are false negatives. The IoU corresponding to these events will be set to 0. The average "match IoU" of all events can be calculated based on this, combined with the event-level f1 value, as an event-level evaluation metric. The IoU metric represents the accuracy of eye movement event-level detection. It measures the degree of overlap between the predicted event and the actual event. The higher the IoU, the better the accuracy. IoU is usually set to 0.5 because it provides a clear interpretation of the results. The first goal is to prevent the occurrence of two events that match the real event. The second goal is to limit the relative displacement of two matching events with the same duration to no more than one-third of the duration.

[0089] Performance in Amplitude and Duration Detection: We first evaluated the average amplitude and duration differences between detected eye movement events and true events and compared them with similar existing methods. As shown in Table 4, the metrics of our model deviate slightly from those reported in the literature; however, this difference is minimal and can be ignored. For the TCN model, we reproduced some results that were not explicitly mentioned in their original paper. To ensure fairness, we configured TCN with the same input feature dimensions and context window size as our model.

[0090] Table 4: Differences between the basic data of the algorithm and the real label

[0091]

[0092] The analysis shows that the event detector is closely correlated with the ground-truth for fixation and saccade events. However, in SP detection, it tends to underestimate the average amplitude and duration compared to the true value. This discrepancy may stem from the limited context window size of 257 samples, equivalent to one second at 250Hz, which limits the model's ability to capture longer SP events. As a result, the detected SP events appear shorter and smaller in amplitude. However, among all the evaluated models, the disclosed enhanced AttSeqNet+ shows the highest overall accuracy in reflecting the characteristics of the real dataset.

[0093] Table 5: IoU values ​​and event-level F1 scores when IoU is greater than 0.5

[0094]

[0095] Comparison of IoU and f1 score: As shown in Table 5, the IoU (Intersection over Union) values ​​of eye movement events detected by various algorithms and the event-level f1 score with IoU>0.5 are analyzed. Although the IoU of SP detection in the AttSeqNet+ model is slightly lower than that of the AttSeqNet model (0.2%), it outperforms other algorithms in SP detection. It is worth noting that the IoU of the disclosed model is 9.2% higher than that of the standard 1DCNN-BLSTM model

[27] . In addition, the disclosed model also improves by 1.5% in SP detection over the 1DCNN-BLSTM final model. On average, the disclosed model achieves the highest IoU among all models, especially under the strict standard (IoU>0.5), where the event-level f1 score of SP detection of the AttSeqNet+ model is 6.0% higher than that of the Startsev final model.

[0096] Table 6: Sample-level and event-level F1 scores

[0097]

[0098] Sample-level and event-level F1 score evaluation: Table 6 discusses the performance of the disclosed algorithm at the sample and event levels. The AttSeqNet+ model achieves a 3.5% higher f1 score for SP detection at the sample level than the 1DCNN-BLSTM final model. At the event level, it is 5.5% higher than the existing best model. Compared with the TCNs model, which improves the sample-level F1 score by dilating the convolutional network, the disclosed model performs well at the event level, especially under stricter criteria (IoU>0.5). While the TCN model requires post-processing to filter out SP events of shorter duration, the disclosed AttSeqNet+ model achieves direct end-to-end SP event detection without the need for such post-processing. Figure 6The confusion matrix shown further illustrates the efficiency and accuracy of the disclosed model.

[0099] Example 2

[0100] In one embodiment of the present disclosure, an eye movement event detection system based on attention drive and data segmentation is provided, comprising:

[0101] A data acquisition module is used to acquire the eye movement data of the target person to be detected within a set time period and pre-process it;

[0102] The detection module is used to convert the eye movement data into time series data, input it into the hybrid network model AttSeqNet, and output the eye movement event detection results;

[0103] Among them, when the time series data is input into the hybrid network model, it first passes through the convolutional neural network module, slides the moving window on the sequence data, and extracts the spatial feature information of the eye movement data; then, it passes through the bidirectional long short-term memory BLSTM module to extract the temporal feature information of the eye movement data, and then the extracted spatial and temporal features are input into the attention mechanism module to generate a new spatial context information feature vector, and finally the final probability distribution is output through softmax to realize the detection of eye movement events.

[0104] Example 3

[0105] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the attention-driven and data-segmented eye movement event detection method is implemented.

[0106] Example 4

[0107] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the eye movement event detection method that realizes the attention drive and data segmentation.

[0108] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. An eye movement event detection method based on attention drive and data segmentation, characterized in that: include: Obtain the eye movement data of the target person to be detected within a set time period and pre-process it; The preprocessed eye movement data is converted into time series data, input into the hybrid network model AttSeqNet, and the eye movement event detection results are output; In the hybrid network model AttSeqNet, when the time series data is input into the hybrid network model, it first passes through the convolutional neural network module, slides the moving window on the sequence data, and extracts the spatial feature information of the eye movement data; then, it passes through the bidirectional long short-term memory BLSTM module to extract the temporal feature information of the eye movement data, and then the extracted spatial and temporal features are input into the attention mechanism module to generate a new spatial context information feature vector, and finally the final probability distribution is output through softmax to realize the detection of eye movement events; The IoU threshold is introduced in the event-level matching calculation to calculate the intersection ratio IoU between the ground truth eye movement events and the algorithm-marked events; only when the IoU of two matching events is greater than this threshold, it is marked as a hit, which is a true positive number; if no event meets the standard, it is marked as missed, which is a false negative; the IoU corresponding to the above events is set to 0; the average matching IoU of all events is calculated based on this, combined with the f1 value at the event level, as an evaluation indicator at the event level; The IoU metric indicates the accuracy of eye movement event-level detection, measuring the degree of overlap between predicted events and actual events; The higher the IoU, the better the accuracy; IoU is usually set to a specific value to provide a clear explanation of the results; the first goal of introducing the IoU threshold is to prevent the occurrence of two events that match the real events, and the second goal is to limit the relative displacement of two matching events with the same duration to no more than one third of the duration.

2. The eye movement event detection method based on attention drive and data segmentation as claimed in claim 1, characterized in that: The eye movement data is an eye movement data sample point, which is the viewpoint coordinates of the eyeball. Linear interpolation is performed on the viewpoint coordinate data, and then the linear interpolation data is filtered to remove noise.

3. The eye movement event detection method based on attention drive and data segmentation as claimed in claim 1, characterized in that: In the convolutional neural network module, a 1D-CNN convolution layer is included. The convolution kernel of the 1D-CNN slides by performing a convolution operation with the corresponding input symbol at each data point. The result of the convolution operation is summed with the convolution kernel parameters and the bias, and activated by the ReLU function to obtain the spatial features of the eye movement data.

4. The eye movement event detection method based on attention drive and data segmentation as claimed in claim 1, characterized in that: The eye movement data is then input into the bidirectional long short-term memory network module. The bidirectional long short-term memory network module includes two different LSTMs, which process the input data in a forward and reverse manner respectively to extract the temporal feature information of the eye movement data.

5. The eye movement event detection method based on attention drive and data segmentation as claimed in claim 1, characterized in that: An attention mechanism is introduced between the bidirectional long short-term memory network module and the fully connected output layer to adaptively learn the weights of context vectors with different features. The spatial and temporal features are input into the attention module to generate an attention matrix, and the similarity score is calculated to generate a new spatial context information feature vector.

6. The eye movement event detection method based on attention drive and data segmentation as claimed in claim 5, characterized in that: The original score is normalized to a probability distribution where the sum of all feature weights is 1 through softmax, and the weighted sum of the original feature vector is performed to obtain the attention value, which represents the final probability distribution of the feature.

7. The eye movement event detection method based on attention drive and data segmentation as claimed in claim 1, characterized in that: When the hybrid network model AttSeqNet is trained, the event segmentation method is used to divide the training set and the validation set. The validation set is composed of data corresponding to the same time period. The size of the data window remains unchanged for the set time and the proportion remains unchanged.

8. Eye movement event detection system based on attention drive and data segmentation, characterized in that: include: A data acquisition module is used to acquire the eye movement data of the target person to be detected within a set time period and pre-process it; The detection module is used to convert the preprocessed eye movement data into time series data, input it into the hybrid network model AttSeqNet, and output the eye movement event detection results; In the hybrid network model AttSeqNet, when the time series data is input into the hybrid network model, it first passes through the convolutional neural network module, slides the moving window on the sequence data, and extracts the spatial feature information of the eye movement data; then, it passes through the bidirectional long short-term memory BLSTM module to extract the temporal feature information of the eye movement data, and then the extracted spatial and temporal features are input into the attention mechanism module to generate a new spatial context information feature vector, and finally the final probability distribution is output through softmax to realize the detection of eye movement events; The IoU threshold is introduced in the event-level matching calculation to calculate the intersection ratio IoU between the ground truth eye movement events and the algorithm-marked events; only when the IoU of two matching events is greater than this threshold, it is marked as a hit, which is a true positive number; if no event meets the standard, it is marked as missed, which is a false negative; the IoU corresponding to the above events is set to 0; the average matching IoU of all events is calculated based on this, combined with the f1 value at the event level, as an evaluation indicator at the event level; The IoU metric indicates the accuracy of eye movement event-level detection, measuring the degree of overlap between predicted events and actual events; The higher the IoU, the better the accuracy; IoU is usually set to a specific value to provide a clear explanation of the results; the first goal of introducing the IoU threshold is to prevent the occurrence of two events that match the real events, and the second goal is to limit the relative displacement of two matching events with the same duration to no more than one third of the duration.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the attention-driven and data-segmented eye movement event detection method as described in any one of claims 1-7 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the attention-driven and data-segmented eye movement event detection method as described in any one of claims 1-7.

Images