Eye movement video data processing method, device, computer equipment and storage medium

Abstract

The present application relates to the field of video data processing technology, and discloses an eye movement video data processing method, apparatus, computer equipment, and storage medium, which include: obtaining eye movement video data; extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; extracting time features based on all frames in the eye movement video data to determine one-dimensional time features of the eye movement video data; performing spatiotemporal fusion of the two-dimensional spatial features and the one-dimensional time features to obtain eye movement features of the eye movement video data; and classifying the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task. If the two-dimensional spatial features and the one-dimensional time features are regarded as spatiotemporally separated features, the present application classifies the completion status of the eye movement task through spatiotemporally separated feature extraction and fusion, thereby realizing a method for efficiently judging the completion status of the eye movement task.

Description

Technical Field

[0001] The present application relates to the technical field of video data processing, and in particular to a method, apparatus, computer equipment and storage medium for processing eye movement video data. Background Art

[0002] Related art methods for eye movement detection primarily focus on statistical analysis of eye movement parameters (e.g., fixation duration, saccade velocity, and saccade latency). While statistical analysis based on eye movement detection parameters can assess the completion of eye movement tasks, this approach has limitations. A new, more efficient method for classifying eye movement task completion is needed. Summary of the Invention

[0003] This application aims to solve, at least to some extent, one of the technical problems in the related art. To this end, this application proposes a method, apparatus, computer device, and storage medium for processing eye movement video data. The main technical solutions adopted in this application include:

[0004] In a first aspect, an embodiment of the present application provides a method for processing eye movement video data, the method comprising: obtaining eye movement video data; wherein the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of an eye movement task; performing feature extraction on each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; performing time feature extraction based on all frames in the eye movement video data to determine one-dimensional time features of the eye movement video data; performing spatiotemporal fusion of the two-dimensional spatial features and the one-dimensional time features to obtain eye movement features of the eye movement video data; and classifying based on the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task.

[0005] In a second aspect, an embodiment of the present application provides an eye movement video data processing device, comprising: a video data acquisition module for acquiring eye movement video data; wherein the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of an eye movement task; a spatial feature extraction module for extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; a temporal feature extraction module for extracting temporal features based on all frames in the eye movement video data to determine one-dimensional temporal features of the eye movement video data; a spatiotemporal feature fusion module for spatiotemporally fusing two-dimensional spatial features and one-dimensional temporal features to obtain eye movement features of the eye movement video data; an eye movement task classification module for classifying based on the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task.

[0006] The present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when the computer program is executed by a processor.

[0008] The present invention provides a computer program product, comprising a computer program, which implements the steps of any one of the above methods when executed by a processor.

[0009] In the above embodiment, eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task; on the one hand, feature extraction is performed on each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; on the other hand, temporal feature extraction is performed based on all frames in the eye movement video data to determine one-dimensional temporal features of the eye movement video data; thereby, the two-dimensional spatial features and the one-dimensional temporal features are spatiotemporally fused to obtain eye movement features of the eye movement video data; and then, classification is performed based on the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task. If the two-dimensional spatial features and the one-dimensional temporal features are regarded as spatiotemporally separated features, then in the above embodiment, the completion status of the eye movement task is classified by extracting and fusing the spatiotemporally separated features, thereby realizing a method for efficiently judging the completion status of the eye movement task. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1a A thumbnail of eye-tracking video data provided according to one embodiment of the present application;

[0012] Figure 1b This is an example diagram of a 3D convolution kernel extraction process provided according to one embodiment of the present application;

[0013] Figure 1c This is a flowchart of a method for processing eye movement video data according to one embodiment of the present application;

[0014] Figure 2a A flowchart of extracting two-dimensional spatial features according to one embodiment of the present application;

[0015] Figure 2b This is an example diagram of a structure for extracting two-dimensional spatial features according to one embodiment of the present application;

[0016] Figure 3a A flowchart of extracting one-dimensional time features according to an embodiment of the present application;

[0017] Figure 3b This is a diagram illustrating an example structure for extracting one-dimensional time features according to an embodiment of the present application;

[0018] Figure 4 A network structure diagram of a series fusion solution provided according to one embodiment of the present application;

[0019] Figure 5 A network structure diagram of a parallel fusion solution provided according to one embodiment of the present application;

[0020] Figure 6a A flowchart of classification of eye movement task completion status provided according to one embodiment of the present application;

[0021] Figure 6b A schematic diagram of standard relative positions for a gaze task provided according to one embodiment of the present application;

[0022] Figure 6c A schematic diagram of standard relative positions for a scanning task provided according to one embodiment of the present application;

[0023] Figure 7a A module structure diagram provided according to an embodiment of the present application;

[0024] Figure 7b A module structure diagram provided according to another embodiment of the present application;

[0025] Figure 8 This is a flowchart of a method for processing eye movement video data according to another embodiment of the present application;

[0026] Figure 9 1 is a structural block diagram of an eye movement video data processing device according to one embodiment of the present application;

[0027] Figure 10 The figure is a diagram of the internal structure of a computer device according to one embodiment of the present application. DETAILED DESCRIPTION

[0028] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0029] In the prior art, the current methods for detecting eye movements mainly focus on statistical analysis of eye movement parameters (such as gaze time, saccade speed, saccade latency, etc.), or subjective judgment by experienced doctors to perform classification assessments of eye movement completion. Statistical analysis based on eye movement detection parameters requires high accuracy and frame rate of eye movement detection instruments. The required inspection instruments are expensive and sensitive to vision, making them unsuitable for people with poor vision (preferably visual acuity <0.5). However, the judgment of eye movements by doctors requires rich clinical experience, is difficult to popularize and apply quickly, and the results are relatively subjective.

[0030] In addition, please refer to Figure 1a Eye movement video data has three dimensions: width, height and depth. They are picture height (height) 102, picture width (width) 104 and frame length (depth) 106. The picture of a certain frame represents the spatial information of the eye movement and the static test point at the current time, and the depth information represents the motion information of the eye movement in continuous time. Therefore, it is particularly important to correctly extract the features of spatial information and motion information. Traditional 3D convolution kernels generally use the following when processing video data: Figure 1b The 3D convolution kernel extraction method shown in the figure has a large number of intermediate parameters, which can easily lead to model failure and low efficiency in classifying the completion of eye movement tasks.

[0031] In summary, according to an embodiment of the present application, an embodiment of a method for processing eye movement video data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] In this embodiment, a method for processing eye movement video data is provided. Figure 1c is a flow chart of a method for processing eye movement video data according to an embodiment of the present application. Figure 1c As shown, the process includes the following steps:

[0033] S110: Obtain eye movement video data.

[0034] The eye movement video data can be obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task. Specifically, the eye movement trajectory video data recorded when the subject completes the eye movement task can be collected as the eye movement video data.

[0035] S120 , extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data.

[0036] Among them, the two-dimensional spatial features can refer to features extracted from each frame of video, which can reflect certain properties of eye movement. Specifically, the two-dimensional spatial features of eye movement video data can be obtained by applying a convolutional neural network, and the two-dimensional spatial features can be extracted through the convolution layer. Exemplarily, the feature extraction of each frame in the eye movement video data can be performed by sliding the convolution kernel on each frame image and performing a weighted summation of the local area for each position to generate a feature map. This feature map can be used as the two-dimensional spatial features of the eye movement video data.

[0037] S130 , extracting time features based on all frames in the eye movement video data to determine a one-dimensional time feature of the eye movement video data.

[0038] Among them, the one-dimensional time features can be time series features extracted from eye movement video data, and these features can reflect the characteristics of eye movement video data that change over time. For example, the one-dimensional time features can refer to those features that reflect the changes in eye movement over time, such as the movement speed of the gaze point, acceleration, the latency of the saccade, and the distribution of the gaze time. Specifically, the one-dimensional time features of the eye movement video data can be obtained by defining one or more one-dimensional convolution kernels, and obtaining the one-dimensional time features of the eye movement video data through convolution operations. For example, these one-dimensional convolution kernels can be designed for specific time series characteristics obtained by conversion based on the eye movement video data, such as detecting speed changes or acceleration changes. Furthermore, the one-dimensional convolution kernel is applied to the time series data obtained by conversion based on the eye movement video data, and a convolution operation is performed on the time series data by sliding extraction to obtain the one-dimensional time features of the eye movement video data.

[0039] S140 , performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain eye movement features of the eye movement video data.

[0040] Among them, eye movement features are a series of features used to characterize different eye movement patterns. For example, these features can be used to describe and distinguish the scanning path of the eyeball and the duration of the gaze point, etc. Spatiotemporal fusion refers to a video data processing method that combines spatial features and temporal features. Specifically, the process of spatiotemporal fusion includes combining the two-dimensional spatial features and one-dimensional temporal features extracted from the eye movement video data to form a comprehensive feature. This comprehensive feature can simultaneously reflect the spatial changes and temporal dynamic characteristics of eye movement. It can be used as the eye movement feature of eye movement video data. For example, the spatiotemporal fusion of two-dimensional spatial features and one-dimensional temporal features can be achieved by merging and splicing the two-dimensional spatial features and the one-dimensional temporal features.

[0041] S150 , classifying the eye movement task based on the eye movement characteristics and the target task category to which the eye movement task belongs, and obtaining the completion status of the eye movement task.

[0042] Among them, the target task category to which the eye movement task belongs can refer to some task categories that associate eye movement data with specific tasks or behaviors when performing eye movement video data analysis. Exemplarily, the target task category can be any eye movement task in which the fixation point and the test point have relative position requirements. Exemplarily, the target task category can include at least one of a fixation task, a saccade task, a reverse saccade task, a horizontal tracking task, and a vertical tracking task. Each task category has its specific eye movement pattern and eye movement characteristics based on the eye movement fixation point and the test point. The completion status of the eye movement task refers to data that is used to judge whether a specific eye movement task is successfully completed or achieves the expected goal by classifying the eye movement characteristics. Exemplarily, the completion status of the eye movement task can be binary, that is, classification is performed based on the eye movement characteristics and the target task category to which the eye movement task belongs, to obtain whether the eye movement task is successfully completed or the eye movement task is failed to complete. Optionally, the completion status of the eye movement task can also be multi-level, that is, it is classified based on the eye movement characteristics and the target task category to which the eye movement task belongs, and the eye movement task completion degree is obtained from 1 to 5. The specific completion status will be determined by the nature of the target task category to which the eye movement task belongs.

[0043] In the above embodiment, eye movement video data is first obtained. The eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task. Feature extraction is performed on each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. Time feature extraction is performed based on all frames in the eye movement video data to determine the one-dimensional time features of the eye movement video data. The two-dimensional spatial features and the one-dimensional time features are spatiotemporally fused to obtain the eye movement features of the eye movement video data. Classification is performed based on the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task. If the two-dimensional spatial features and the one-dimensional time features are regarded as spatiotemporally separated features, the completion status of the eye movement task is classified by spatiotemporally separated feature extraction and fusion in the above embodiment, thereby realizing a method for efficiently judging the completion status of the eye movement task.

[0044] In some embodiments, please refer to the attached Figure 2a , extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data, including:

[0045] S210 , extracting spatial features from the picture frames in the eye movement video data to obtain a spatial feature map.

[0046] Among them, spatial feature extraction can refer to the process of extracting features that can describe the content of each frame from each frame of the video. For example, the extracted features may include color, texture, shape, edge and other information. Specifically, through the sliding operation of the convolution kernel, for each position in the picture frame in the eye movement video data, the local area covered by the convolution kernel and the convolution kernel itself are element-wise multiplied and summed to obtain a value representing the feature response of the position, which is a spatial feature map. This spatial feature map can be a two-dimensional array, each element of which represents the feature response of the corresponding position in the original image. For example, if Figure 2b As shown, assuming that the video matrix of the input eye movement video data is x, where x can be a 4-dimensional matrix, including width w, which is used to represent the number of horizontal pixels in the video frame and determines the horizontal resolution of the video; height h, which is used to represent the number of vertical pixels in the video frame and determines the vertical resolution of the video; depth d, which is used to represent the number of frames in the video matrix, that is, the total length of the video, in frames; color c, which represents the color information of each pixel, can include the three basic color channels RGB (red, green, and blue), each channel containing the intensity information of the corresponding color. It can be understood that feature extraction for each frame of the eye movement video data can be performed by sliding a convolution kernel with a width, height, and depth specification of (3x3x1) to obtain a spatial feature map of the eye movement video data.

[0047] S220: Perform dimensionality reduction processing on the spatial feature map through global average pooling to obtain spatial distribution weight data.

[0048] Among them, Global Average Pooling (GAP) is a pooling operation that averages all pixel values ​​of each channel of the feature map to generate a single value, which is the spatial distribution weight data. For example, f KJ The spatial feature obtained by extracting spatial features from the frames in the eye movement video data is represented as a spatial feature map. KJ Characterize the spatial weight map obtained after the spatial feature map is reduced in dimension by global average pooling, that is, the spatial distribution weight data. Preferably, it is assumed that the dimension of the input spatial feature map is (W×H×C), where W is the width of the spatial feature map, that is, the number of pixels of the spatial feature map in the horizontal direction; H is the height of the spatial feature map, that is, the number of pixels of the spatial feature map in the vertical direction; C is the number of spatial feature map channels, that is, the number of different feature channels in the spatial feature map. The spatial feature map is reduced in dimension by global average pooling (GAP), and all pixel values ​​of each channel of the spatial feature map are averaged to obtain the spatial distribution weight data. This process can be achieved by the method shown in the following formula:

[0049] g KJ =Sigmoid(GAP(f KJ ))

[0050] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f KJ Represents the spatial features obtained after spatial feature extraction from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained by dimensionality reduction of the spatial feature map through global average pooling, that is, the spatial distribution weight data.

[0051] S230: Perform weighted processing on the spatial feature map using the spatial distribution weight data to obtain a two-dimensional spatial feature.

[0052] Specifically, by applying the spatial distribution weight data to the spatial feature map and performing element-by-element multiplication, the spatial feature map of each channel will obtain a distribution weight according to its corresponding global average value, and finally a spatial attention map is obtained, which is a two-dimensional spatial feature.

[0053] For example, the above process can be implemented by the following formula:

[0054] f′ KJ =f KJ ×g KJ

[0055] Among them, f KJ Represents the spatial features obtained after spatial feature extraction from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained by dimensionality reduction of the spatial feature map through global average pooling, that is, the spatial distribution weight data; f′ KJ Representation is the spatial attention map obtained after weighted processing, that is, the two-dimensional spatial feature.

[0056] In the above embodiment, the spatial feature map is subjected to dimensionality reduction processing through global average pooling, which reduces the dimension of the spatial feature map while retaining important information in the spatial feature map, so that the model can pay more attention to those spatial areas that are more important to the current task, thereby improving the effectiveness of feature extraction and further improving the classification accuracy and efficiency of the completion of eye movement tasks.

[0057] In some embodiments, please refer to the attached Figure 3a , based on all frames in the eye movement video data, temporal features are extracted to determine the one-dimensional temporal features of the eye movement video data, including:

[0058] S310 , extracting temporal features from all frames in the eye movement video data to obtain a temporal feature graph.

[0059] Among them, temporal feature extraction can obtain temporal features by analyzing the temporal information contained in each frame of eye movement video data. For example, temporal feature extraction can be used to capture the temporal dependency between frames, such as the continuity of actions or the movement trend of objects to obtain temporal features. Specifically, one or more one-dimensional convolution kernels can be defined, and a temporal feature map can be obtained by sliding the 1D convolution kernel on the time dimension to extract the information of the time series. For example, Figure 3b As shown in the figure, assume that the video matrix of the input eye movement video data is x, where x can be a 4-dimensional matrix containing width w, which represents the number of horizontal pixels in the video frame and determines the horizontal resolution of the video; height h, which represents the number of vertical pixels in the video frame and determines the vertical resolution of the video; depth d, which represents the number of frames in the video matrix, that is, the total duration of the video in frames; and color c, which represents the color information of each pixel and can include the three basic color channels RGB (red, green, and blue), each of which contains the intensity information of the corresponding color. At this time, temporal features are extracted from all frames in the eye movement video data. The temporal feature map of the eye movement video data can be obtained by sliding a convolution kernel with a width, height, and depth specification of (1x1x3).

[0060] S320: Perform dimensionality reduction processing on the temporal feature graph through global maximum pooling to obtain temporal distribution weight data.

[0061] Among them, Global Max Pooling (GMP) is a pooling operation that is performed independently on each channel and searches for the maximum value along the time dimension (i.e., the number of frames). Each channel will eventually obtain a single value, which represents the maximum response of the channel in all time frames. Specifically, it is applied to the time dimension to reduce the dimension of the time feature map by global max pooling, that is, selecting the maximum value in the time series in each channel to obtain the time distribution weight data. For example, f SJ The temporal feature obtained by extracting temporal features from all frames of the eye movement video data is the temporal feature graph. SJ Characterize the temporal weight map obtained by reducing the dimension of the temporal feature map by global average pooling, that is, the temporal distribution weight data. Preferably, it is assumed that the dimension of the input temporal feature map is (T×C), where T is the number of video frames and C is used to represent the number of channels of each frame. The temporal feature map is reduced in dimension by global maximum pooling (GMP) to obtain the temporal distribution weight data. This process can be implemented by the method shown in the following formula:

[0062] g SJ =Sigmoid(GMP(f SJ ))

[0063] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f SJ Represents the temporal features obtained by extracting temporal features from all frames of the eye movement video data, i.e., the temporal feature graph; g SJ Represents the time weight map obtained by dimensionality reduction of the time feature map through global average pooling, that is, the time distribution weight data.

[0064] S330: Perform weighted processing on the time feature graph using the time distribution weight data to obtain a one-dimensional time feature.

[0065] Similarly, weighted processing can be performed by applying the time distribution weight data to the time feature map and performing element-by-element multiplication to obtain the time attention vector, which is a one-dimensional time feature.

[0066] For example, the above process can be implemented by the following formula:

[0067] f′ SJ =f SJ ×g SJ

[0068] Among them, f SJ Represents the temporal features obtained by extracting temporal features from all frames of the eye movement video data, i.e., the temporal feature graph; gSJ Represents the time weight map obtained by dimensionality reduction of the time feature map through global average pooling, that is, the time distribution weight data; f′ SJ Represent the time attention vector obtained after weighted processing, that is, the one-dimensional time feature.

[0069] In the above embodiment, the temporal features are reduced in dimension through global maximum pooling, and the time distribution weight data is used to emphasize the key frames in the video, thereby highlighting the key time frames, which helps to improve the effectiveness of feature extraction and further improves the classification accuracy and efficiency of the eye movement task completion.

[0070] In some embodiments, performing spatiotemporal fusion of two-dimensional spatial features and one-dimensional temporal features to obtain eye movement features of eye movement video data includes: performing serial fusion of one-dimensional temporal features and two-dimensional spatial features to obtain eye movement features.

[0071] Serial fusion refers to the process of sequentially combining or concatenating features of different dimensions to create a new feature matrix. Specifically, in serial fusion, two-dimensional convolution sliding feature extraction is first performed on the frames in the eye movement video data to obtain two-dimensional spatial features. Furthermore, using these two-dimensional spatial features as input, one-dimensional convolution feature extraction is performed on all frames in the eye movement video data to extract their temporal features, ultimately obtaining a comprehensive feature matrix containing both spatial and temporal information, namely the eye movement features.

[0072] For example, assuming that the video matrix of the input eye movement video data is x, KJ can be used to represent the two-dimensional spatial feature extraction module, which is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. Similarly, SJ can be used to represent the one-dimensional temporal feature extraction module, which is used to extract temporal features based on all frames in the eye movement video data, determine the one-dimensional temporal features of the eye movement video data, and use F(x) to represent the eye movement features obtained by spatiotemporal fusion of the two-dimensional spatial features and the one-dimensional temporal features. Please refer to Figure 4 It can be understood that, first, the KJ module 401 uses a two-dimensional convolutional neural network to extract features from each frame of the eye movement video data, thereby obtaining two-dimensional spatial features of the eye movement video data. Furthermore, the two-dimensional spatial features are used as input to the SJ module 402, which uses a one-dimensional convolutional neural network to extract temporal features from all frames of the eye movement video data. Ultimately, a spatiotemporal feature matrix that combines two-dimensional spatial features and one-dimensional temporal features is obtained, namely, the eye movement features.

[0073] In the above implementation, a serial fusion scheme combines spatial and temporal features to better understand and analyze the dynamic characteristics of eye movement video data. Compared with methods that use 3D convolution to obtain spatiotemporal features, this method reduces the number of intermediate parameters while achieving better classification performance. This further improves the accuracy and efficiency of classifying eye movement task completion.

[0074] In some embodiments, performing spatiotemporal fusion of two-dimensional spatial features and one-dimensional temporal features to obtain eye movement features of eye movement video data includes: performing parallel fusion of one-dimensional temporal features and two-dimensional spatial features to obtain eye movement features.

[0075] Parallel fusion refers to the process of extracting features independently from both spatial and temporal dimensions simultaneously and then fusing them to form a comprehensive feature matrix encompassing multiple dimensions. Specifically, in parallel fusion, two-dimensional convolution sliding feature extraction is performed on the frames in the eye movement video data to obtain two-dimensional spatial features. At the same time, temporal feature extraction is performed on all frames in the eye movement video data to obtain one-dimensional temporal features. Furthermore, the two-dimensional spatial features and one-dimensional temporal features are spatiotemporally fused through weighted sum or product fusion to obtain the eye movement features of the eye movement video data.

[0076] For example, assuming that the video matrix of the input eye movement video data is x, KJ can be used to represent the two-dimensional spatial feature extraction module, which is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. Similarly, SJ can be used to represent the one-dimensional temporal feature extraction module, which is used to extract temporal features based on all frames in the eye movement video data, determine the one-dimensional temporal features of the eye movement video data, and use F(x) to represent the eye movement features obtained by spatiotemporal fusion of the two-dimensional spatial features and the one-dimensional temporal features. Please refer to Figure 5 It can be understood that the KJ module 501 uses a two-dimensional convolutional neural network to extract features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data. At the same time, the SJ module 502 uses a one-dimensional convolutional neural network to extract time features from all frames of the eye movement video data to obtain one-dimensional time features of the eye movement video data. Furthermore, the process of spatiotemporal fusion of the independently extracted two-dimensional spatial features and one-dimensional time features is achieved by element-by-element multiplication. For each time point, the corresponding elements of the two-dimensional spatial features and the one-dimensional time features are multiplied to obtain the eye movement features.

[0077] In this implementation, a parallel fusion scheme combines spatial and temporal features to better understand and analyze the characteristics of eye movement video data. Compared to methods that derive spatiotemporal features through 3D convolution, this approach reduces the number of intermediate parameters while achieving better classification performance. This further improves the accuracy and efficiency of classifying eye movement task completion.

[0078] In some embodiments, classification is performed based on the eye movement characteristics and the target task category of the eye movement task to obtain the completion status of the eye movement task, including:

[0079] Mapping is performed based on the eye movement characteristics to determine whether the eye movement task is completed correctly or incorrectly under the target task category.

[0080] Specifically, after extracting eye movement features through a convolutional neural network based on a pre-trained model, the eye movement features can be mapped to the fully connected layer and the output layer for processing under a specific target task category, and the probability distribution of whether the eye movement task is completed correctly or incorrectly is output. Furthermore, the probability value of the eye movement task completion corresponding to the classification node output conversion of the fully connected layer is converted, and by comparing the probability values ​​of correct completion and incorrect completion, a completion status with a higher probability value is finally output and used as the completion status of the eye movement task. For example, if the probability value of correct completion is greater than the probability value of incorrect completion, it indicates that the eye movement task is completed correctly under the target task category, and the correct completion is used as the final output of the eye movement task completion status; if the probability value of incorrect completion is greater than the probability value of correct completion, it indicates that the eye movement task is completed incorrectly under the target task category, and the incorrect completion is used as the final output of the eye movement task completion status.

[0081] Alternatively, in some embodiments, please refer to the attached Figure 6a , there is a test point in the picture frame, and the eye movement feature data is used to describe the actual relative position between the test point and the fixation point. The target task category can be any eye movement task that requires a relative position between the fixation point and the test point. For example, it can include any one of a fixation task, a saccade task, a reverse saccade task, a horizontal tracking task, and a vertical tracking task. The target task category has a standard relative position. Classification is performed based on the eye movement features and the target task category of the eye movement task to obtain the completion status of the eye movement task, including:

[0082] S610: Compare the actual relative position with the standard relative position to obtain consistency between the actual relative position and the standard relative position.

[0083] The test point refers to the point that indicates where the eye should look during the eye movement task. Specifically, the state of the test point will be different for different target task categories. For example, please refer to Figure 6b In the fixation task, the test point 602 refers to the point that appears in the center of the screen and the subject is required to fixate on. At this time, the subject's fixation point 604 is as follows: Figure 6b Similarly, please refer to Figure 6c In the saccade task, the test point 602 refers to the point where the subject's gaze point 604 changes to a new position after being gazed at. Figure 6c As shown. The standard relative position refers to the ideal or expected positional relationship between the test point and the fixation point in the eye movement task, which also includes the standard motion trajectory of the fixation point in the eye movement task. Exemplarily, for different target task categories, the standard relative position can be determined based on the intuitive assessment of the relative positional relationship between the test point and the fixation point by an experienced doctor. The actual relative position refers to the relative position between the test point and the fixation point displayed in the eye movement video data, which also includes the actual motion trajectory of the fixation point.

[0084] Specifically, the consistency between the actual relative position and the standard relative position can be obtained by classifying and evaluating the degree of match between the actual relative position and the standard relative position. For example, if the actual relative position matches the standard relative position, the consistency between the actual relative position and the standard relative position is consistent; if the actual relative position does not match the standard relative position, the consistency between the actual relative position and the standard relative position is inconsistent. Optionally, a threshold can be set to judge the consistency. For example, if the difference in the degree of consistency is within the range of this threshold, the consistency is considered to be good; if the difference in the degree of consistency is outside the range of this threshold, the consistency is considered to be poor.

[0085] S620: If the consistency indicates that the actual relative position is consistent with the standard relative position, determine that the eye movement task is completed correctly.

[0086] Specifically, if the consistency indicates that the actual relative position is consistent with the standard relative position, that is, under the target task category, the classification results of the actual relative position and the standard relative position are consistent, then the completion of the target task category by the subject shown in the eye movement video data is correct.

[0087] S630: If the consistency condition indicates that the actual relative position is inconsistent with the standard relative position, determine that the completion condition of the eye movement task is a completion error.

[0088] Specifically, if the consistency situation indicates that the actual relative position is inconsistent with the standard relative position, that is, under the target task category, the classification results of the actual relative position and the standard relative position are inconsistent, then the completion status of the subject in the eye movement video data in completing the target task category is a completion error.

[0089] In the above embodiment, by comparing the actual relative position between the test point and the fixation point obtained based on the eye movement feature data of the subject under different target task categories with the standard relative position between the test point and the fixation point under the target task category, the target task category of the eye movement task is classified to obtain the completion status of the eye movement task under the target task category. Furthermore, the degree of dependence on the physician's experience is reduced, and the completion status of the eye movement task can be judged more objectively. And only focusing on the information about the test point and the fixation point in the eye movement video data, the accuracy requirements for the eye movement detection instrument are low, which reduces the detection cost. And the vision requirements for the subject are also low (experiments have found that the best eye vision ≥ 0.1 can complete the detection), thereby improving the classification efficiency of the eye movement task completion status, suitable for use by a wider range of people, and is expected to be promoted to mobile terminals such as mobile phones and tablets.

[0090] In some embodiments, the two-dimensional spatial features are obtained by extracting features through a two-dimensional convolution module, and the one-dimensional temporal features are obtained by extracting features through a one-dimensional convolution module, and the two-dimensional convolution module and the one-dimensional convolution module are connected in series or in parallel.

[0091] The two-dimensional convolution module constructs an n×n convolution kernel in a planar space based on a convolutional neural network, and performs convolution operations on the two-dimensional plane of the eye movement video data to obtain two-dimensional spatial features. The one-dimensional convolution module constructs a convolution kernel along the vertical axis based on a convolutional neural network, and performs convolution operations on the depth level of the eye movement video data to obtain one-dimensional temporal features.

[0092] For example, the two-dimensional convolution module and the one-dimensional convolution module can be connected in series. Figure 7a , wherein the two-dimensional convolution module 702 is used to extract features from each frame in the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. The one-dimensional convolution module 704 is used to extract temporal features based on all frames in the eye movement video data to determine the one-dimensional temporal features of the eye movement video data. It can be understood that first, the two-dimensional convolution module 702 is used to extract features from each frame in the eye movement video data using a two-dimensional convolutional neural network to obtain the two-dimensional spatial features of the eye movement video data. Further, the two-dimensional spatial features are used as the input of the one-dimensional convolution module 704, and the one-dimensional convolutional neural network is used to extract temporal features from all frames in the eye movement video data, and finally a spatiotemporal feature that integrates the two-dimensional spatial features and the one-dimensional temporal features is obtained, that is, the eye movement feature.

[0093] Similarly, the 2D convolution module and the 1D convolution module can be connected in parallel. Figure 7b , wherein the two-dimensional convolution module 702 is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. The one-dimensional convolution module 704 is used to extract temporal features based on all frames in the eye movement video data to determine the one-dimensional temporal features of the eye movement video data. Through the two-dimensional convolution module 702, a two-dimensional convolutional neural network is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. At the same time, through the one-dimensional convolution module 704, a one-dimensional convolutional neural network is used to extract temporal features from all frames in the eye movement video data to obtain the one-dimensional temporal features of the eye movement video data. Further, the independently extracted two-dimensional spatial features and one-dimensional temporal features are spatially and temporally fused, and the process is completed by element-by-element multiplication. Finally, the spatiotemporal features that fuse the two-dimensional spatial features and the one-dimensional temporal features are obtained, that is, the eye movement features.

[0094] In the above implementation, a serial fusion scheme combines spatial and temporal features to better understand and analyze the dynamic characteristics of eye movement video data. Compared with methods that use 3D convolution to obtain spatiotemporal features, this method reduces the number of intermediate parameters while achieving better classification performance. This further improves the accuracy and efficiency of classifying eye movement task completion.

[0095] This specification also provides a method for processing eye movement video data. Figure 8 , the method comprises the following steps:

[0096] S802: Obtain eye movement video data.

[0097] Among them, the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task.

[0098] S804: Extract spatial features from the picture frames in the eye movement video data to obtain a spatial feature map.

[0099] Specifically, through the sliding operation of the convolution kernel, for each position in the picture frame of the eye movement video data, the local area covered by the convolution kernel and the convolution kernel itself are element-wise multiplied and summed to obtain a value representing the feature response of the position, which is the spatial feature map.

[0100] S806: Perform dimensionality reduction processing on the spatial feature map through global average pooling to obtain spatial distribution weight data.

[0101] Specifically, the spatial feature map is reduced in dimension by global average pooling (GAP), and all pixel values ​​of each channel of the spatial feature map are averaged to obtain spatial distribution weight data. This process can be achieved by the method shown in the following formula:

[0102] g KJ =Sigmoid(GAP(f KJ ))

[0103] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f KJ Represents the spatial features obtained after spatial feature extraction from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained by dimensionality reduction of the spatial feature map through global average pooling, that is, the spatial distribution weight data.

[0104] S808: Perform weighted processing on the spatial feature map using the spatial distribution weight data to obtain a two-dimensional spatial feature.

[0105] Specifically, weighted processing can be performed by applying the spatial distribution weight data to the spatial feature map, and performing element-by-element multiplication so that the spatial feature map of each channel will obtain a distribution weight according to its corresponding global average value, and finally obtain a spatial attention map, which is a two-dimensional spatial feature.

[0106] For example, the above process can be implemented by the following formula:

[0107] f′ KJ =f KJ ×g KJ

[0108] Among them, f KJ Represents the spatial features obtained after spatial feature extraction from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained by dimensionality reduction of the spatial feature map through global average pooling, that is, the spatial distribution weight data; f′ KJ Representation is the spatial attention map obtained after weighted processing, that is, the two-dimensional spatial feature.

[0109] S810: Extract temporal features from all frames in the eye movement video data to obtain a temporal feature graph.

[0110] Specifically, one or more one-dimensional convolution kernels can be defined and 1D convolution kernels can be used to slide on the time dimension to extract the time series information, that is, to obtain a time feature map.

[0111] S812: Perform dimensionality reduction processing on the temporal feature map through global maximum pooling to obtain temporal distribution weight data.

[0112] The temporal feature map is reduced in dimension by global maximum pooling (GMP) to obtain temporal distribution weight data, which is used to characterize the importance of each channel in the time series. This process can be achieved by the method shown in the following formula:

[0113] g SJ =Sigmoid(GMP(f SJ ))

[0114] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f SJ Represents the temporal features obtained by extracting temporal features from all frames of the eye movement video data, i.e., the temporal feature graph; g SJ Represents the time weight map obtained by dimensionality reduction of the time feature map through global average pooling, that is, the time distribution weight data.

[0115] S814: Perform weighted processing on the time feature graph using the time distribution weight data to obtain a one-dimensional time feature.

[0116] Similarly, weighted processing can be performed by applying the time distribution weight data to the time feature map and performing element-by-element multiplication to obtain the time attention vector, which is a one-dimensional time feature.

[0117] For example, the above process can be implemented by the following formula:

[0118] f′ SJ =f SJ ×g SJ

[0119] Among them, f SJ Represents the temporal features obtained by extracting temporal features from all frames of the eye movement video data, i.e., the temporal feature graph; g SJ Represents the time weight map obtained by dimensionality reduction of the time feature map through global average pooling, that is, the time distribution weight data; f′ SJ Represent the time attention vector obtained after weighted processing, that is, the one-dimensional time feature.

[0120] S816 , performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain eye movement features of the eye movement video data.

[0121] For example, two-dimensional spatial features and one-dimensional temporal features can be serially fused to obtain eye movement features. Specifically, in the serial fusion, two-dimensional convolution sliding feature extraction is first performed on the frames in the eye movement video data to obtain two-dimensional spatial features. Furthermore, using these two-dimensional spatial features as input, temporal feature extraction can be performed on all frames in the eye movement video data, and one-dimensional convolution feature extraction can be performed. Ultimately, a comprehensive feature matrix containing spatial and temporal information is obtained, namely the eye movement features.

[0122] For example, one-dimensional time features and two-dimensional spatial features may be fused in parallel to obtain eye movement features.

[0123] Specifically, in parallel fusion, two-dimensional convolution sliding feature extraction is performed on the frames in the eye movement video data to obtain two-dimensional spatial features. At the same time, temporal feature extraction is performed on all frames in the eye movement video data to obtain one-dimensional temporal features. Furthermore, the two-dimensional spatial features and one-dimensional temporal features are spatiotemporally fused through weighted sum or product fusion to obtain eye movement features of the eye movement video data.

[0124] S818. Mapping is performed according to the eye movement characteristics to determine whether the eye movement task is completed correctly or incorrectly under the target task category.

[0125] It should be understood that, although the various steps in the above flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0126] The embodiment of this specification also provides an eye movement video data processing device 900, such as Figure 9 As shown, it includes: a video data acquisition module 910, a spatial feature extraction module 920, a temporal feature extraction module 930, a spatiotemporal feature fusion module 940 and an eye movement task classification module 950, wherein:

[0127] The video data acquisition module 910 is used to acquire eye movement video data, wherein the eye movement video data is acquired by collecting the movement of the subject's eye gaze point during the execution of the eye movement task.

[0128] The spatial feature extraction module 920 is used to extract features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data.

[0129] The time feature extraction module 930 is used to extract time features based on all frames in the eye movement video data to determine the one-dimensional time features of the eye movement video data.

[0130] The spatiotemporal feature fusion module 940 is used to perform spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain the eye movement features of the eye movement video data.

[0131] The eye movement task classification module 950 is used to perform classification based on the eye movement features and the target task category to which the eye movement task belongs, and obtain the completion status of the eye movement task.

[0132] In some embodiments, the spatial feature extraction module 920 is further configured to extract spatial features from frames in the eye movement video data to obtain a spatial feature map. The spatial feature map is subjected to dimensionality reduction processing using global average pooling to obtain spatial distribution weight data. The spatial feature map is then weighted using the spatial distribution weight data to obtain a two-dimensional spatial feature.

[0133] In some embodiments, the temporal feature extraction module 930 is further configured to extract temporal features from all frames in the eye movement video data to obtain a temporal feature map. The temporal feature map is subjected to dimensionality reduction processing using global maximum pooling to obtain temporal distribution weight data. The temporal feature map is weighted using the temporal distribution weight data to obtain a one-dimensional temporal feature.

[0134] In some embodiments, the spatiotemporal feature fusion module 940 is further configured to perform serial fusion of the one-dimensional temporal feature and the two-dimensional spatial feature to obtain the eye movement feature, and further configured to perform parallel fusion of the one-dimensional temporal feature and the two-dimensional spatial feature to obtain the eye movement feature.

[0135] In some embodiments, the eye movement task classification module 950 is further configured to perform mapping based on the eye movement features to determine whether the eye movement task is completed correctly or incorrectly under the target task category.

[0136] In some embodiments, an eye movement video data processing apparatus 900 further includes a two-dimensional convolution module and a one-dimensional convolution module. The two-dimensional convolution module is configured to construct a convolution kernel of size n×n in a planar space based on a convolutional neural network, and perform a convolution operation on the eye movement video data on the two-dimensional plane to obtain two-dimensional spatial features. The one-dimensional convolution module is configured to construct a convolution kernel of a longitudinal axis based on a convolutional neural network, and perform a convolution operation on the eye movement video data at the depth level to obtain one-dimensional temporal features.

[0137] The specific definition of an eye movement video data processing device can be found in the definition of an eye movement video data processing method described above and will not be repeated here. Each module in the above-mentioned eye movement video data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0138] In this embodiment, an eye movement video data processing device is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0139] The embodiment of the present application further provides a computer device, which may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an eye movement video data processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0140] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0141] The embodiments of the present application also provide a computer-readable storage medium, and the above-mentioned method according to the embodiment of the present application can be implemented in hardware and firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0143] An embodiment of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the method in any of the above embodiments.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer equipment. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present application is described with reference to a flowchart and / or block diagram of an eye movement video data processing method, apparatus, computer device, and storage medium according to an embodiment of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0146] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0148] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0149] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, since the computer device and computer-readable storage medium are substantially similar to the method embodiments, their description is relatively simple. For relevant portions, reference can be made to the description of the method embodiments.

[0150] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

[0151] Although the embodiments of the present application have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. A method for processing eye movement video data, characterized in that: The method comprises: Acquiring eye movement video data; wherein the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task; Performing feature extraction on each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; wherein spatial feature extraction refers to the process of extracting features that can describe the content of each frame of the eye movement video data through a two-dimensional convolution module; Temporal feature extraction is performed based on all frames in the eye movement video data to determine a one-dimensional temporal feature of the eye movement video data; wherein the temporal feature extraction is performed by sliding a one-dimensional convolution module on the time dimension to extract time series information to obtain a temporal feature graph for capturing the temporal dependency between frames in the eye movement video data, the two-dimensional convolution module is connected in series or in parallel with the one-dimensional convolution module; the two-dimensional spatial feature and the one-dimensional temporal feature are regarded as spatiotemporal separation features; Performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain eye movement features of the eye movement video data; wherein the spatiotemporal fusion includes serial fusion or parallel fusion, wherein the serial fusion refers to the process of combining or connecting features of different dimensions in sequence to obtain a new feature matrix, and the parallel fusion refers to the process of independently extracting features from both spatial and temporal dimensions and then fusing them to obtain a comprehensive feature matrix containing multiple dimensions; the eye movement features simultaneously reflect the spatial changes and temporal dynamic characteristics of eye movement; Classification is performed based on the eye movement characteristics and the target task category to which the eye movement task belongs, so as to obtain the completion status of the eye movement task.

2. The method according to claim 1, characterized in that Extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data includes: Extracting spatial features from the picture frames in the eye movement video data to obtain a spatial feature map; Performing dimensionality reduction processing on the spatial feature map by global average pooling to obtain spatial distribution weight data; The spatial feature map is weighted using the spatial distribution weight data to obtain the two-dimensional spatial feature.

3. The method according to claim 1, characterized in that The extracting time features based on all frames in the eye movement video data to determine the one-dimensional time features of the eye movement video data includes: Extracting time features from all frames in the eye movement video data to obtain a time feature graph; Performing dimensionality reduction processing on the temporal feature map through global maximum pooling to obtain temporal distribution weight data; The time characteristic graph is weighted using the time distribution weight data to obtain the one-dimensional time characteristic.

4. The method according to claim 1, wherein There is a test point in the picture frame, and the eye movement feature data is used to describe the actual relative position between the test point and the gaze point.

5. The method according to claim 4, characterized in that The target task category is any one of a fixation task, a saccade task, a reverse saccade task, a horizontal pursuit task, and a vertical pursuit task.

6. The method according to claim 5, characterized in that The target task category has a standard relative position; the classification based on the eye movement characteristics and the target task category of the eye movement task to obtain the completion status of the eye movement task includes: Comparing the actual relative position with the standard relative position to obtain consistency between the actual relative position and the standard relative position; If the consistency condition indicates that the actual relative position is consistent with the standard relative position, it is determined that the eye movement task is completed correctly.

7. The method according to claim 6, characterized in that The classifying based on the eye movement features and the target task category of the eye movement task to obtain the completion status of the eye movement task further includes: If the consistency condition indicates that the actual relative position is inconsistent with the standard relative position, it is determined that the completion of the eye movement task is an error.

8. An eye movement video data processing device, characterized in that: The device comprises: A video data acquisition module is used to acquire eye movement video data; wherein the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task; a spatial feature extraction module, configured to extract features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; wherein spatial feature extraction refers to the process of extracting features that can describe the content of each frame of the eye movement video data through a two-dimensional convolution module; A temporal feature extraction module is configured to extract temporal features based on all frames in the eye movement video data to determine a one-dimensional temporal feature of the eye movement video data; wherein the temporal feature extraction is performed by sliding a one-dimensional convolution module on the time dimension to extract time series information to obtain a temporal feature graph, which is configured to capture the temporal dependency between frames in the eye movement video data; the two-dimensional convolution module is connected in series or in parallel with the one-dimensional convolution module; the two-dimensional spatial features and the one-dimensional temporal features are regarded as spatiotemporally separated features; a spatiotemporal feature fusion module, configured to spatiotemporally fuse the two-dimensional spatial features and the one-dimensional temporal features to obtain eye movement features of the eye movement video data; wherein the spatiotemporal fusion includes serial fusion or parallel fusion, wherein serial fusion refers to the process of combining or connecting features of different dimensions in sequence to obtain a new feature matrix, and parallel fusion refers to the process of independently extracting features from both spatial and temporal dimensions and then fusing them to obtain a comprehensive feature matrix containing multiple dimensions; the eye movement features simultaneously reflect the spatial changes and temporal dynamic characteristics of eye movement; The eye movement task classification module is used to classify the eye movement task based on the eye movement characteristics and the target task category to which the eye movement task belongs, so as to obtain the completion status of the eye movement task.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images