Eye movement video data processing method and device, computer equipment and storage medium

By extracting the two-dimensional spatial characteristics and one-dimensional temporal characteristics of eye movement video data and performing time-space fusion, the problem of low classification efficiency of eye movement tasks in the existing technology is solved, and efficient judgment of the completion of eye movement tasks is achieved. It is applicable to People with poor vision have reduced the cost of testing.

CN119067924AActive Publication Date: 2024-12-03ZHONGSHAN OPHTHALMIC CENT SUN YAT SEN UNIV
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202411084382.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-03
Estimated Expiration
2044-08-08

AI Technical Summary

Technical Problem

The method of detecting eye movement in the prior art mainly depends on the statistical analysis of eye movement parameters, and it is difficult to efficiently classify the completion of eye movement tasks. It also has high vision requirements and is not suitable for people with poor vision, and it depends on doctors to judge that there is subjectiveness. and difficult to popularize.

Method used

By obtaining eye movement video data, extracting two-dimensional spatial features and one-dimensional temporal features, and performing time and space fusion, classifying task categories based on eye movement characteristics, achieving efficient judgment on the completion of eye movement tasks.

Benefits of technology

It improves the classification accuracy and efficiency of eye movement tasks, reduces the accuracy requirements for detection instruments, is suitable for people with poor vision, reduces the dependence on doctor experience, and reduces the detection cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067924A_ABST
    Figure CN119067924A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of video data processing, and discloses an eye movement video data processing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining eye movement video data; performing feature extraction on each frame of picture in the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; time feature extraction is carried out based on all picture frames in the eye movement video data, and one-dimensional time features of the eye movement video data are determined; carrying out space-time fusion on the two-dimensional space features and the one-dimensional time features to obtain eyeball movement features of the eye movement video data; and based on the eyeball movement features and the target task category to which the eyeball movement task belongs, performing classification to obtain a completion condition of the eyeball movement task. If the two-dimensional space features and the one-dimensional time features are regarded as the features of time-space separation, the completion conditions of the eyeball movement task are classified through feature extraction and fusion of time-space separation, and the method for efficiently judging the completion conditions of the eyeball movement task is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of video data processing, and in particular to an eye movement video data processing method, device, computer equipment and storage medium. Background Art

[0002] The methods for eye movement detection in related technologies mainly focus on statistical analysis of eye movement parameters (such as fixation time, saccade speed, saccade latency, etc.). Although statistical analysis based on eye movement detection parameters can be used to evaluate the completion of eye movement tasks, this method has limitations, and a new method with higher classification efficiency for eye movement task completion needs to be proposed. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present application proposes an eye movement video data processing method, device, computer equipment and storage medium. The main technical solutions adopted by the present application include:

[0004] In a first aspect, an embodiment of the present application provides a method for processing eye movement video data, the method comprising: obtaining eye movement video data; wherein the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of an eye movement task; performing feature extraction on each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; performing time feature extraction based on all frames in the eye movement video data to determine one-dimensional time features of the eye movement video data; performing spatiotemporal fusion of the two-dimensional spatial features and the one-dimensional time features to obtain eye movement features of the eye movement video data; and performing classification based on the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task.

[0005] In a second aspect, an embodiment of the present application provides an eye movement video data processing device, comprising: a video data acquisition module, for acquiring eye movement video data; wherein the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of an eye movement task; a spatial feature extraction module, for extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; a temporal feature extraction module, for extracting temporal features based on all frames in the eye movement video data to determine one-dimensional temporal features of the eye movement video data; a spatiotemporal feature fusion module, for spatiotemporally fusing two-dimensional spatial features and one-dimensional temporal features to obtain eye movement features of the eye movement video data; an eye movement task classification module, for performing classification based on the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task.

[0006] The present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0008] The present invention provides a computer program product, comprising a computer program, which implements the steps of any one of the above methods when executed by a processor.

[0009] In the above embodiment, the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task; on the one hand, the feature extraction is performed on each frame of the eye movement video data to obtain the two-dimensional spatial feature of the eye movement video data; on the other hand, the time feature extraction is performed based on all the frames in the eye movement video data to determine the one-dimensional time feature of the eye movement video data; thereby, the two-dimensional spatial feature and the one-dimensional time feature are spatiotemporally fused to obtain the eye movement feature of the eye movement video data; and then, the eye movement feature and the target task category to which the eye movement task belongs are classified to obtain the completion status of the eye movement task. If the two-dimensional spatial feature and the one-dimensional time feature are regarded as spatiotemporally separated features, the completion status of the eye movement task is classified by spatiotemporally separated feature extraction and fusion in the above embodiment, thereby realizing a method for efficiently judging the completion status of the eye movement task. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1a A thumbnail of eye-movement video data provided according to an embodiment of the present application;

[0012] Figure 1b This is an example diagram of a 3D convolution kernel extraction process provided according to an embodiment of the present application;

[0013] Figure 1c A flowchart of an eye movement video data processing method provided according to an embodiment of the present application;

[0014] Figure 2a A flowchart of extracting two-dimensional spatial features according to an embodiment of the present application;

[0015] Figure 2b This is a diagram showing an example of a structure for extracting two-dimensional spatial features according to an embodiment of the present application;

[0016] Figure 3a A flowchart of extracting one-dimensional time features according to an embodiment of the present application;

[0017] Figure 3b This is a diagram showing an example of a structure for extracting one-dimensional time features according to an embodiment of the present application;

[0018] Figure 4 A network structure diagram of a series fusion solution provided according to an embodiment of the present application;

[0019] Figure 5 A network structure diagram of a parallel fusion solution provided according to an embodiment of the present application;

[0020] Figure 6a A flowchart of classification of eye movement task completion status provided according to one embodiment of the present application;

[0021] Figure 6b A schematic diagram of standard relative positions of a gaze task provided according to an embodiment of the present application;

[0022] Figure 6c A schematic diagram of standard relative positions for a scanning task provided according to an embodiment of the present application;

[0023] Figure 7a A module structure diagram provided according to an embodiment of the present application;

[0024] Figure 7b A module structure diagram provided according to another embodiment of the present application;

[0025] Figure 8 A flowchart of an eye movement video data processing method provided according to another embodiment of the present application;

[0026] Fig. 9 is a structural block diagram of an eye movement video data processing device according to an embodiment of the present application;

[0027] Fig.10 The figure is a diagram of the internal structure of a computer device according to one embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0029] In the prior art, the current methods for detecting eye movements mainly focus on statistical analysis of eye movement parameters (e.g., fixation time, saccade speed, saccade latency, etc.), or classification evaluation of eye movement completion by subjective judgment of experienced doctors. Statistical analysis based on eye movement detection parameters requires high accuracy and frame rate of eye movement detection instruments, and the required inspection instruments are expensive and sensitive to vision, which is not suitable for people with poor vision (best eye vision <0.5). However, the judgment of eye movements by doctors requires rich clinical experience, is difficult to popularize and apply quickly, and the results are relatively subjective.

[0030] In addition, please refer to Figure 1a , eye movement video data has three dimensions: width, height and depth, namely, picture height (height) 102, picture width (width) 104 and frame length (depth) 106. The picture of a certain frame represents the spatial information of the eye movement and the static test point at the current time, and the depth information represents the motion information of the eye movement in continuous time. Therefore, it is particularly important to correctly extract the features of spatial information and motion information. Traditional 3D convolution kernels generally use the following when processing video data: Figure 1b The 3D convolution kernel extraction method shown in the figure may easily lead to the failure of model convergence due to the huge amount of intermediate parameters, which in turn leads to low efficiency in classifying the completion of eye movement tasks.

[0031] In summary, according to an embodiment of the present application, an embodiment of an eye movement video data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] In this embodiment, a method for processing eye movement video data is provided. Figure 1c is a flow chart of an eye movement video data processing method according to an embodiment of the present application. Figure 1c As shown, the process includes the following steps:

[0033] S110: Obtain eye movement video data.

[0034] The eye movement video data may be obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task. Specifically, the eye movement trajectory video data recorded when the subject completes the eye movement task may be collected as the eye movement video data.

[0035] S120 , extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data.

[0036] Among them, the two-dimensional spatial features may refer to the features extracted from each frame of the video, and these features can reflect certain properties of eye movement. Specifically, the two-dimensional spatial features of the eye movement video data can be obtained by applying a convolutional neural network, and the two-dimensional spatial features are obtained by extracting the convolution layer. Exemplarily, the feature extraction of each frame in the eye movement video data can be performed by sliding the convolution kernel on each frame image, and performing a weighted summation of the local area for each position to generate a feature map. This feature map can be used as the two-dimensional spatial features of the eye movement video data.

[0037] S130 , extracting time features based on all picture frames in the eye movement video data to determine a one-dimensional time feature of the eye movement video data.

[0038] Among them, the one-dimensional time features can be time series features extracted from the eye movement video data, and these features can reflect the characteristics of the eye movement video data changing over time. Exemplarily, the one-dimensional time features can refer to those features that reflect the changes in eye movement over time, such as the movement speed of the gaze point, the acceleration, the latency of the saccade, and the distribution of the gaze time. Specifically, the one-dimensional time features of the eye movement video data can be obtained by defining one or more one-dimensional convolution kernels, and obtaining the one-dimensional time features of the eye movement video data through convolution operations. Exemplarily, these one-dimensional convolution kernels can be designed for specific time series characteristics obtained by conversion based on the eye movement video data, such as detecting speed changes or acceleration changes. Further, the one-dimensional convolution kernel is applied to the time series data obtained by conversion based on the eye movement video data, and a convolution operation is performed on the time series data by sliding extraction to obtain the one-dimensional time features of the eye movement video data.

[0039] S140 , performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain eye movement features of the eye movement video data.

[0040] Among them, eye movement features are a series of features used to characterize different eye movement patterns. Exemplarily, these features can be used to describe and distinguish the eye's scanning path and the duration of the gaze point, etc. Spatiotemporal fusion refers to a video data processing method that combines spatial features with temporal features. Specifically, the process of spatiotemporal fusion includes combining the two-dimensional spatial features and one-dimensional temporal features extracted from the eye movement video data to form a comprehensive feature. This comprehensive feature can simultaneously reflect the spatial changes and temporal dynamic characteristics of eye movement. It can be used as the eye movement feature of eye movement video data. Exemplarily, the spatiotemporal fusion of two-dimensional spatial features and one-dimensional temporal features can be achieved by merging and splicing the two-dimensional spatial features and the one-dimensional temporal features.

[0041] S150, classifying based on the eye movement features and the target task category to which the eye movement task belongs, to obtain the completion status of the eye movement task.

[0042] Wherein, the target task category to which the eye movement task belongs can refer to some task categories that associate eye movement data with specific tasks or behaviors when performing eye movement video data analysis. Exemplarily, the target task category can be any eye movement task with relative position requirements between the fixation point and the test point, and exemplary, the target task category can include at least one of the fixation task, the saccade task, the reverse saccade task, the horizontal tracking task, and the vertical tracking task. Each task category has its specific eye movement pattern and eye movement characteristics based on the eye movement fixation point and the test point. The completion of the eye movement task refers to the data that is used to judge whether a specific eye movement task is successfully completed or reaches the expected goal by classifying the eye movement characteristics. Exemplarily, the completion of the eye movement task can be binary, that is, classification is performed based on the eye movement characteristics and the target task category to which the eye movement task belongs, and the eye movement task is successfully completed or the eye movement task fails to complete. Optionally, the completion status of the eye movement task can also be multi-level, that is, it is classified based on the eye movement characteristics and the target task category to which the eye movement task belongs, and the eye movement task completion degree is obtained from 1 to 5. The specific completion status will be determined by the nature of the target task category to which the eye movement task belongs.

[0043] In the above implementation, eye movement video data is first obtained. The eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task. Feature extraction is performed on each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. Time feature extraction is performed based on all frames in the eye movement video data to determine the one-dimensional time features of the eye movement video data. The two-dimensional spatial features and the one-dimensional time features are spatiotemporally fused to obtain the eye movement features of the eye movement video data. Classification is performed based on the eye movement features and the target task category to which the eye movement task belongs to obtain the completion status of the eye movement task. If the two-dimensional spatial features and the one-dimensional time features are regarded as spatiotemporally separated features, the completion status of the eye movement task is classified through the spatiotemporally separated feature extraction and fusion in the above embodiment, thereby realizing a method for efficiently judging the completion status of the eye movement task.

[0044] In some embodiments, please refer to the attached Figure 2a , extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data, including:

[0045] S210: Extract spatial features from picture frames in the eye movement video data to obtain a spatial feature map.

[0046] Among them, spatial feature extraction may refer to the process of extracting features that can describe the content of each frame of the video. Exemplarily, the extracted features may include information such as color, texture, shape, and edge. Specifically, through the sliding operation of the convolution kernel, for each position in the picture frame in the eye movement video data, the local area covered by the convolution kernel and the convolution kernel itself are element-wise multiplied and summed to obtain a value representing the feature response of the position, which is a spatial feature map. This spatial feature map can be a two-dimensional array, in which each element represents the feature response of the corresponding position in the original image. Exemplarily, such as Figure 2b As shown, it is assumed that the video matrix of the input eye movement video data is x, where x can be a 4-dimensional matrix, including width w, which is used to represent the number of horizontal pixels of the video frame, which determines the horizontal resolution of the video; height h, which is used to represent the number of vertical pixels of the video frame, which determines the vertical resolution of the video; depth d, which is used to represent the number of frames in the video matrix, that is, the total duration of the video, in frames; color c, which represents the color information of each pixel, can include three basic color channels of RGB (red, green, and blue), and each channel contains the intensity information of the corresponding color. It can be understood that feature extraction of each frame of the eye movement video data can be performed by sliding a convolution kernel with a width, height, and depth specification of (3x3x1) to obtain a spatial feature map of the eye movement video data.

[0047] S220, performing dimensionality reduction processing on the spatial feature map through global average pooling to obtain spatial distribution weight data.

[0048] Among them, global average pooling (GAP) is a pooling operation that averages all pixel values ​​of each channel of the feature map to generate a single value, which is the spatial distribution weight data. For example, f KJ The spatial feature obtained by extracting spatial features from the picture frames in the eye movement video data is the spatial feature map. KJ Characterize the spatial weight map obtained after the spatial feature map is reduced in dimension by global average pooling, that is, the spatial distribution weight data. Preferably, it is assumed that the dimension of the input spatial feature map is (W×H×C), where W is the width of the spatial feature map, that is, the number of pixels of the spatial feature map in the horizontal direction; H is the height of the spatial feature map, that is, the number of pixels of the spatial feature map in the vertical direction; C is the number of channels of the spatial feature map, that is, the number of different feature channels in the spatial feature map. The spatial feature map is reduced in dimension by global average pooling (GAP), and all pixel values ​​of each channel of the spatial feature map are averaged to obtain the spatial distribution weight data. This process can be implemented by the method shown in the following formula:

[0049] g KJ =Sigmoid(GAP(f KJ ))

[0050] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f KJ Represents the spatial features obtained after extracting spatial features from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained after dimensionality reduction of the spatial feature map through global average pooling, that is, the spatial distribution weight data.

[0051] S230, performing weighted processing on the spatial feature map using the spatial distribution weight data to obtain a two-dimensional spatial feature.

[0052] Specifically, by applying the spatial distribution weight data to the spatial feature map and performing element-by-element multiplication, the spatial feature map of each channel will obtain a distribution weight according to its corresponding global average value, and finally a spatial attention map is obtained, which is a two-dimensional spatial feature.

[0053] Exemplarily, the above process can be implemented by the following formula:

[0054] f′ KJ =f KJ ×g KJ

[0055] Among them, f KJ Represents the spatial features obtained after extracting spatial features from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained by reducing the dimension of the spatial feature map through global average pooling, that is, the spatial distribution weight data; f′ KJ Representation of the weighted spatial attention map, i.e., the two-dimensional spatial feature.

[0056] In the above implementation, the spatial feature map is dimensionality reduced by global average pooling, which reduces the dimension of the spatial feature map while retaining important information in the spatial feature map, so that the model can pay more attention to those spatial areas that are more important to the current task, thereby improving the effectiveness of feature extraction and further improving the classification accuracy and efficiency of the completion of eye movement tasks.

[0057] In some embodiments, please refer to the attached Figure 3a , extracting temporal features based on all frames in the eye movement video data, and determining the one-dimensional temporal features of the eye movement video data, including:

[0058] S310 , extracting time features from all frames in the eye movement video data to obtain a time feature graph.

[0059] Among them, the temporal feature extraction can obtain the temporal feature by analyzing the temporal information contained in each frame of the eye movement video data. Exemplarily, the temporal feature extraction can be used to capture the temporal dependency between frames, such as the continuity of actions or the movement trend of objects to obtain the temporal feature. Specifically, one or more one-dimensional convolution kernels can be defined, and the temporal feature graph can be obtained by sliding the 1D convolution kernel on the time dimension to extract the information of the time series. Exemplarily, as Figure 3b As shown, assume that the video matrix of the input eye movement video data is x, where x can be a 4-dimensional matrix, including width w, which is used to represent the number of horizontal pixels of the video frame, which determines the horizontal resolution of the video; height h, which is used to represent the number of vertical pixels of the video frame, which determines the vertical resolution of the video; depth d, which is used to represent the number of frames in the video matrix, that is, the total duration of the video, in frames; color c, which represents the color information of each pixel, can include three basic color channels of RGB (red, green, and blue), and each channel contains the intensity information of the corresponding color. At this time, the temporal feature extraction of all the frames in the eye movement video data can be performed by sliding operation of the convolution kernel with a width, height, and depth specification of (1x1x3) to obtain the temporal feature map of the eye movement video data.

[0060] S320, performing dimension reduction processing on the temporal feature graph through global maximum pooling to obtain temporal distribution weight data.

[0061] Among them, Global Max Pooling (GMP) is a pooling operation that is performed independently on each channel and searches for the maximum value along the time dimension (i.e., the number of frames). Each channel will eventually obtain a single value, which represents the maximum response of the channel in all time frames. Specifically, the time dimension is applied to reduce the dimension of the time feature map by global maximum pooling, that is, the maximum value of the time series is selected in each channel to obtain the time distribution weight data. For example, f SJ The temporal feature obtained by extracting temporal features from all frames of the eye movement video data is the temporal feature graph. SJ Characterize the time weight map obtained after the time feature map is reduced in dimension by global average pooling, that is, the time distribution weight data. Preferably, it is assumed that the dimension of the input time feature map is (T×C), where T is the number of video frames and C is used to represent the number of channels of each frame. The time feature map is reduced in dimension by global maximum pooling (GMP) to obtain the time distribution weight data. This process can be implemented by the method shown in the following formula:

[0062] g SJ =Sigmoid(GMP(f SJ ))

[0063] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f SJ Represents the temporal features obtained after temporal feature extraction from all frames of the eye movement video data, i.e., the temporal feature graph; g SJ Represents the time weight map obtained by dimensionality reduction of the time feature map through global average pooling, that is, the time distribution weight data.

[0064] S330, using the time distribution weight data to perform weighted processing on the time feature graph to obtain a one-dimensional time feature.

[0065] Similarly, weighted processing can be performed by applying the time distribution weight data to the time feature map and multiplying the elements by element to obtain the time attention vector, which is a one-dimensional time feature.

[0066] Exemplarily, the above process can be implemented by the following formula:

[0067] f′ SJ =f SJ ×g SJ

[0068] Among them, f SJ Represents the temporal features obtained after temporal feature extraction from all frames of the eye movement video data, i.e., the temporal feature graph; gSJ Represents the time weight map obtained by reducing the dimension of the time feature map through global average pooling, that is, the time distribution weight data; f′ SJ Represents the time attention vector obtained after weighted processing, that is, the one-dimensional time feature.

[0069] In the above implementation, the temporal features are reduced in dimension through global maximum pooling, and the time distribution weight data is used to emphasize the key frames in the video, thereby highlighting the key time frames, which helps to improve the effectiveness of feature extraction and further improves the classification accuracy and efficiency of the completion of eye movement tasks.

[0070] In some embodiments, two-dimensional spatial features and one-dimensional temporal features are spatiotemporally fused to obtain eye movement features of eye movement video data, including: one-dimensional temporal features and two-dimensional spatial features are serially fused to obtain eye movement features.

[0071] Among them, serial fusion refers to the process of combining or connecting features of different dimensions in sequence to obtain a new feature matrix. Specifically, in serial fusion, firstly, two-dimensional convolution sliding feature extraction is performed on the picture frames in the eye movement video data to obtain two-dimensional spatial features. Furthermore, these two-dimensional spatial features are used as input, and one-dimensional convolution feature extraction is performed on all the picture frames in the eye movement video data to obtain their temporal features, and finally a comprehensive feature matrix containing spatial and temporal information is obtained, which is the eye movement feature.

[0072] For example, assuming that the video matrix of the input eye movement video data is x, KJ can be used to represent a two-dimensional spatial feature extraction module, which is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. Similarly, SJ can be used to represent a one-dimensional temporal feature extraction module, which is used to extract temporal features based on all frames in the eye movement video data, determine the one-dimensional temporal features of the eye movement video data, and use F(x) to represent the eye movement features obtained by spatiotemporal fusion of the two-dimensional spatial features and the one-dimensional temporal features. Please refer to Figure 4 It can be understood that, first, the KJ module 401 uses a two-dimensional convolutional neural network to extract features from each frame of the eye movement video data, and obtains the two-dimensional spatial features of the eye movement video data. Further, the two-dimensional spatial features are used as input to the SJ module 402, and the one-dimensional convolutional neural network is used to extract temporal features from all frames in the eye movement video data, and finally a spatiotemporal feature matrix that combines two-dimensional spatial features and one-dimensional temporal features is obtained, that is, the eye movement features.

[0073] In the above implementation, the spatial and temporal features are combined through a series fusion scheme to better understand and analyze the dynamic features of eye movement video data. Compared with the method of obtaining spatiotemporal features through 3D convolution, it has better classification performance while reducing the amount of intermediate parameters, and further improves the classification accuracy and efficiency of the completion of eye movement tasks.

[0074] In some embodiments, two-dimensional spatial features and one-dimensional temporal features are spatiotemporally fused to obtain eye movement features of eye movement video data, including: one-dimensional temporal features and two-dimensional spatial features are parallelly fused to obtain eye movement features.

[0075] Parallel fusion refers to the process of extracting features independently from both spatial and temporal dimensions and then fusing them to obtain a comprehensive feature matrix containing multiple dimensions. Specifically, in parallel fusion, two-dimensional convolution sliding feature extraction is performed on the picture frames in the eye movement video data to obtain two-dimensional spatial features, while temporal feature extraction is performed on all picture frames in the eye movement video data to obtain one-dimensional temporal features. Furthermore, the two-dimensional spatial features and the one-dimensional temporal features are spatiotemporally fused by weighted sum or product fusion to obtain the eye movement features of the eye movement video data.

[0076] For example, assuming that the video matrix of the input eye movement video data is x, KJ can be used to represent a two-dimensional spatial feature extraction module, which is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. Similarly, SJ can be used to represent a one-dimensional temporal feature extraction module, which is used to extract temporal features based on all frames in the eye movement video data, determine the one-dimensional temporal features of the eye movement video data, and use F(x) to represent the eye movement features obtained by spatiotemporal fusion of the two-dimensional spatial features and the one-dimensional temporal features. Please refer to Figure 5 It can be understood that the KJ module 501 uses a two-dimensional convolutional neural network to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. At the same time, the SJ module 502 uses a one-dimensional convolutional neural network to extract time features from all frames in the eye movement video data to obtain the one-dimensional time features of the eye movement video data. Furthermore, the process of spatiotemporal fusion of the independently extracted two-dimensional spatial features and one-dimensional time features is achieved by element-by-element multiplication. For each time point, the corresponding elements of the two-dimensional spatial features and the one-dimensional time features are multiplied to obtain the eye movement features.

[0077] In the above implementation, the spatial and temporal features are combined through a parallel fusion scheme to better understand and analyze the characteristics of eye movement video data. Compared with the method of obtaining spatiotemporal features through 3D convolution, it has better classification performance while reducing the amount of intermediate parameters. It further improves the classification accuracy and efficiency of the completion of eye movement tasks.

[0078] In some embodiments, classification is performed based on the eye movement features and the target task category of the eye movement task to obtain the completion status of the eye movement task, including:

[0079] Mapping is performed based on the eye movement features to determine whether the eye movement task is completed correctly or incorrectly under the target task category.

[0080] Specifically, after extracting eye movement features through a convolutional neural network based on a pre-trained model, under a specific target task category, the eye movement features are mapped to the fully connected layer and the output layer for processing, and the probability distribution of correct or incorrect completion of the eye movement task is output. Further, the probability value of the eye movement task completion corresponding to the classification node output conversion of the fully connected layer is output, and by comparing the probability values ​​of correct completion and incorrect completion, a completion status with a high probability value is finally output and used as the completion status of the eye movement task. Exemplarily, if the probability value of correct completion is greater than the probability value of incorrect completion, it indicates that the eye movement task is completed correctly under the target task category, and the correct completion is used as the final output of the eye movement task completion status; if the probability value of incorrect completion is greater than the probability value of correct completion, it indicates that the eye movement task is completed incorrectly under the target task category, and the incorrect completion is used as the final output of the eye movement task completion status.

[0081] Optionally, in some embodiments, please refer to the attached Figure 6a , there is a test point in the picture frame, and the eye movement feature data is used to describe the actual relative position between the test point and the fixation point. The target task category can be any eye movement task that requires a relative position between the fixation point and the test point, and can include, for example, any of a fixation task, a saccade task, a reverse saccade task, a horizontal tracking task, and a vertical tracking task. The target task category has a standard relative position. Classification is performed based on the eye movement features and the target task category of the eye movement task to obtain the completion status of the eye movement task, including:

[0082] S610: Compare the actual relative position with the standard relative position to obtain consistency between the actual relative position and the standard relative position.

[0083] The test point refers to the point that indicates the direction the eye should look during the eye movement task. Specifically, the state of the test point will be different for different target task categories. For example, please refer to Figure 6b In the fixation task, the test point 602 refers to the point that appears in the center of the screen and the subject is required to fixate on. At this time, the subject's fixation point 604 is as follows: Figure 6b Similarly, please refer to Figure 6c In the saccade task, the test point 602 refers to the point that will change to a new position after being fixed on, and the eye movement is required to shift to the point of fixation. At this time, the subject's fixation point 604 is as follows: Figure 6c As shown. The standard relative position refers to the ideal or expected positional relationship between the test point and the fixation point in the eye movement task, which also includes the standard motion trajectory of the fixation point in the eye movement task. Exemplarily, for different target task categories, the standard relative position can be determined based on an intuitive assessment of the relative positional relationship between the test point and the fixation point by a senior doctor. The actual relative position refers to the relative position between the test point and the fixation point displayed in the eye movement video data, which also includes the actual motion trajectory of the fixation point.

[0084] Specifically, the consistency between the actual relative position and the standard relative position can be obtained by classifying and evaluating the matching degree between the actual relative position and the standard relative position. Exemplarily, if the actual relative position matches the standard relative position, the consistency between the actual relative position and the standard relative position is consistent; if the actual relative position does not match the standard relative position, the consistency between the actual relative position and the standard relative position is inconsistent. Optionally, a threshold can be set to judge the consistency. Exemplarily, if the difference in the degree of consistency is within the range of this threshold, the consistency is considered to be good; if the difference in the degree of consistency is outside the range of this threshold, the consistency is considered to be poor.

[0085] S620: If the consistency condition indicates that the actual relative position is consistent with the standard relative position, it is determined that the eye movement task is completed correctly.

[0086] Specifically, if the consistency situation indicates that the actual relative position is consistent with the standard relative position, that is, under the target task category, the classification results of the actual relative position and the standard relative position are consistent, then the completion of the target task category by the subject shown in the eye movement video data is correct.

[0087] S630: If the consistency status indicates that the actual relative position is inconsistent with the standard relative position, determine that the completion status of the eye movement task is a completion error.

[0088] Specifically, if the consistency situation indicates that the actual relative position is inconsistent with the standard relative position, that is, under the target task category, the classification results of the actual relative position and the standard relative position are inconsistent, then the completion of the target task category by the subject shown in the eye movement video data is a completion error.

[0089] In the above-mentioned implementation, by comparing the actual relative position between the test point and the fixation point obtained based on the eye movement feature data of the subject under different target task categories and the standard relative position between the test point and the fixation point under the target task category, the target task category of the eye movement task is classified to obtain the completion of the eye movement task under the target task category. Further, the degree of dependence on the experience of the physician is reduced, and the completion of the eye movement task can be judged more objectively. And only focusing on the information about the test point and the fixation point in the eye movement video data, the accuracy requirements of the eye movement detection instrument are relatively low, reducing the detection cost. And the vision requirements for the subject are also relatively low (experimentally found that the best eye vision ≥ 0.1 can complete the detection), thereby improving the classification efficiency of the completion of the eye movement task, suitable for a wider range of people, and is expected to be promoted to mobile terminals such as mobile phones and tablets.

[0090] In some embodiments, the two-dimensional spatial feature is obtained by extracting features through a two-dimensional convolution module, and the one-dimensional time feature is obtained by extracting features through a one-dimensional convolution module. The two-dimensional convolution module is connected in series or in parallel with the one-dimensional convolution module.

[0091] Among them, the two-dimensional convolution module is a module that constructs a convolution kernel of size n×n in a plane space based on a convolutional neural network, and performs a convolution operation on the two-dimensional plane of the eye movement video data to obtain two-dimensional spatial features. The one-dimensional convolution module is a module that constructs a convolution kernel of the longitudinal axis based on a convolutional neural network, and performs a convolution operation on the depth level of the eye movement video data to obtain one-dimensional temporal features.

[0092] Exemplarily, the two-dimensional convolution module and the one-dimensional convolution module may be connected in series. Figure 7a , wherein the two-dimensional convolution module 702 is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. The one-dimensional convolution module 704 is used to extract temporal features based on all frames in the eye movement video data to determine the one-dimensional temporal features of the eye movement video data. It can be understood that first, through the two-dimensional convolution module 702, a two-dimensional convolutional neural network is used to extract features from each frame in the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. Furthermore, the two-dimensional spatial features are used as the input of the one-dimensional convolution module 704, and the one-dimensional convolutional neural network is used to extract temporal features from all frames in the eye movement video data, and finally a spatiotemporal feature that combines the two-dimensional spatial features and the one-dimensional temporal features is obtained, that is, the eye movement feature.

[0093] Similarly, a 2D convolution module and a 1D convolution module can be connected in parallel. Figure 7b , wherein the two-dimensional convolution module 702 is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. The one-dimensional convolution module 704 is used to extract temporal features based on all frames in the eye movement video data to determine the one-dimensional temporal features of the eye movement video data. Through the two-dimensional convolution module 702, a two-dimensional convolutional neural network is used to extract features from each frame of the eye movement video data to obtain the two-dimensional spatial features of the eye movement video data. At the same time, through the one-dimensional convolution module 704, a one-dimensional convolutional neural network is used to extract temporal features from all frames in the eye movement video data to obtain the one-dimensional temporal features of the eye movement video data. Further, the independently extracted two-dimensional spatial features and one-dimensional temporal features are spatially and temporally fused, and the process is completed by element-by-element multiplication. Finally, the spatiotemporal features that fuse the two-dimensional spatial features and the one-dimensional temporal features, i.e., the eye movement features, are obtained.

[0094] In the above implementation, the spatial and temporal features are combined through a series fusion scheme to better understand and analyze the dynamic features of eye movement video data. Compared with the method of obtaining spatiotemporal features through 3D convolution, it has better classification performance while reducing the amount of intermediate parameters, and further improves the classification accuracy and efficiency of the completion of eye movement tasks.

[0095] This specification also provides a method for processing eye movement video data. Figure 8 , the method comprises the following steps:

[0096] S802: Obtain eye movement video data.

[0097] Among them, the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task.

[0098] S804: Extract spatial features from the picture frames in the eye movement video data to obtain a spatial feature map.

[0099] Specifically, through the sliding operation of the convolution kernel, for each position in the picture frame in the eye movement video data, the local area covered by the convolution kernel and the convolution kernel itself are element-wise multiplied and summed to obtain a value representing the feature response of the position, which is the spatial feature map.

[0100] S806: Perform dimensionality reduction processing on the spatial feature map through global average pooling to obtain spatial distribution weight data.

[0101] Specifically, the spatial feature map is reduced in dimension by global average pooling (GAP), and all pixel values ​​of each channel of the spatial feature map are averaged to obtain spatial distribution weight data. This process can be implemented by the method shown in the following formula:

[0102] g KJ =Sigmoid(GAP(f KJ ))

[0103] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f KJ Represents the spatial features obtained after extracting spatial features from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained after dimensionality reduction of the spatial feature map through global average pooling, that is, the spatial distribution weight data.

[0104] S808. Perform weighted processing on the spatial feature map using the spatial distribution weight data to obtain a two-dimensional spatial feature.

[0105] Specifically, weighted processing can be performed by applying the spatial distribution weight data to the spatial feature map, and performing element-by-element multiplication so that the spatial feature map of each channel obtains a distribution weight according to its corresponding global average value, and finally a spatial attention map is obtained, which is a two-dimensional spatial feature.

[0106] Exemplarily, the above process can be implemented by the following formula:

[0107] f′ KJ =f KJ ×g KJ

[0108] Among them, f KJ Represents the spatial features obtained after extracting spatial features from the picture frames in the eye movement video data, that is, the spatial feature map; g KJ Represents the spatial weight map obtained by reducing the dimension of the spatial feature map through global average pooling, that is, the spatial distribution weight data; f′ KJ Representation of the weighted spatial attention map, i.e., the two-dimensional spatial feature.

[0109] S810: Extract time features from all frames in the eye movement video data to obtain a time feature graph.

[0110] Specifically, one or more one-dimensional convolution kernels can be defined and the 1D convolution kernel can be used to slide on the time dimension to extract the time series information, that is, to obtain a time feature map.

[0111] S812. Perform dimensionality reduction processing on the temporal feature graph through global maximum pooling to obtain temporal distribution weight data.

[0112] The time feature map is reduced in dimension by global maximum pooling (GMP) to obtain time distribution weight data, which is used to characterize the importance of each channel in the time series. This process can be achieved by the method shown in the following formula:

[0113] g SJ =Sigmoid(GMP(f SJ ))

[0114] Among them, Sigmoid is an activation function whose output value is between 0 and 1, which is used to convert the pooling result into weight data; f SJ Represents the temporal features obtained after temporal feature extraction from all frames of the eye movement video data, i.e., the temporal feature graph; g SJ Represents the time weight map obtained by dimensionality reduction of the time feature map through global average pooling, that is, the time distribution weight data.

[0115] S814. Perform weighted processing on the time feature graph using the time distribution weight data to obtain a one-dimensional time feature.

[0116] Similarly, weighted processing can be performed by applying the time distribution weight data to the time feature map and performing element-by-element multiplication to obtain the time attention vector, which is a one-dimensional time feature.

[0117] Exemplarily, the above process can be implemented by the following formula:

[0118] f′ SJ =f SJ ×g SJ

[0119] Among them, f SJ Represents the temporal features obtained after temporal feature extraction from all frames of the eye movement video data, i.e., the temporal feature graph; g SJ Represents the time weight map obtained by reducing the dimension of the time feature map through global average pooling, that is, the time distribution weight data; f′ SJ Represents the time attention vector obtained after weighted processing, that is, the one-dimensional time feature.

[0120] S816, performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain eye movement features of the eye movement video data.

[0121] Exemplarily, the two-dimensional spatial features and the one-dimensional temporal features can be fused in series to obtain the eye movement features. Specifically, in the series fusion, the picture frames in the eye movement video data are first subjected to two-dimensional convolution sliding feature extraction to obtain the two-dimensional spatial features. Further, these two-dimensional spatial features are used as input to extract the temporal features of all the picture frames in the eye movement video data, and one-dimensional convolution feature extraction can be performed to finally obtain a comprehensive feature matrix containing spatial and temporal information, namely the eye movement features.

[0122] Exemplarily, one-dimensional time features and two-dimensional space features may be fused in parallel to obtain eye movement features.

[0123] Specifically, in parallel fusion, two-dimensional convolution sliding feature extraction is performed on the picture frames in the eye movement video data to obtain two-dimensional spatial features, and time feature extraction is performed on all the picture frames in the eye movement video data to obtain one-dimensional time features. Further, the two-dimensional spatial features and the one-dimensional time features are spatiotemporally fused by weighted sum or product fusion to obtain the eye movement features of the eye movement video data.

[0124] S818. Mapping is performed according to the eye movement characteristics to determine whether the eye movement task is completed correctly or incorrectly under the target task category.

[0125] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0126] The present specification also provides an eye movement video data processing device 900, such as Fig. 9 As shown, it includes: a video data acquisition module 910, a spatial feature extraction module 920, a temporal feature extraction module 930, a spatiotemporal feature fusion module 940 and an eye movement task classification module 950, wherein:

[0127] The video data acquisition module 910 is used to acquire eye movement video data, wherein the eye movement video data is acquired by collecting the movement of the subject's eye gaze point during the execution of the eye movement task.

[0128] The spatial feature extraction module 920 is used to extract features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data.

[0129] The time feature extraction module 930 is used to extract time features based on all frames in the eye movement video data to determine the one-dimensional time features of the eye movement video data.

[0130] The spatiotemporal feature fusion module 940 is used to perform spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain the eye movement features of the eye movement video data.

[0131] The eye movement task classification module 950 is used to classify the eye movement task based on the eye movement characteristics and the target task category to which the eye movement task belongs, so as to obtain the completion status of the eye movement task.

[0132] In some embodiments, the spatial feature extraction module 920 is further used to extract spatial features from the picture frames in the eye movement video data to obtain a spatial feature map. The spatial feature map is subjected to dimensionality reduction processing by global average pooling to obtain spatial distribution weight data. The spatial feature map is subjected to weighted processing using the spatial distribution weight data to obtain a two-dimensional spatial feature.

[0133] In some embodiments, the temporal feature extraction module 930 is further used to extract temporal features from all frames in the eye movement video data to obtain a temporal feature graph. The temporal feature graph is subjected to dimensionality reduction processing by global maximum pooling to obtain temporal distribution weight data. The temporal feature graph is subjected to weighted processing using the temporal distribution weight data to obtain a one-dimensional temporal feature.

[0134] In some embodiments, the spatiotemporal feature fusion module 940 is further used to fuse the one-dimensional time feature and the two-dimensional space feature in series to obtain the eye movement feature, and is further used to fuse the one-dimensional time feature and the two-dimensional space feature in parallel to obtain the eye movement feature.

[0135] In some embodiments, the eye movement task classification module 950 is further used to perform mapping according to the eye movement features to determine whether the eye movement task is completed correctly or incorrectly under the target task category.

[0136] In some embodiments, an eye movement video data processing device 900 further includes a two-dimensional convolution module and a one-dimensional convolution module, wherein the two-dimensional convolution module is used to construct a convolution kernel of a plane space of size n×n based on a convolution neural network, and perform a convolution operation on the two-dimensional plane of the eye movement video data to obtain a two-dimensional spatial feature. The one-dimensional convolution module is used to construct a convolution kernel of a longitudinal axis based on a convolution neural network, and perform a convolution operation on the depth level of the eye movement video data to obtain a one-dimensional temporal feature.

[0137] The specific definition of an eye movement video data processing device can be found in the definition of an eye movement video data processing method above, which will not be repeated here. Each module in the above eye movement video data processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0138] In this embodiment, an eye movement video data processing device is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0139] The embodiment of the present application also provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an eye movement video data processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0140] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] The embodiment of the present application also provides a computer-readable storage medium, and the above-mentioned method according to the embodiment of the present application can be implemented in hardware and firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0143] An embodiment of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the method in any of the above embodiments.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices or computer equipment. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0145] The present application is described with reference to a flowchart and / or block diagram of an eye movement video data processing method, device, computer device and storage medium according to an embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for computer devices and computer-readable storage media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0150] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application, etc., should all be included in the scope of the claims of the present application.

[0151] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for processing eye movement video data, characterized in that: The method comprises: Acquire eye movement video data; wherein the eye movement video data is obtained by collecting the movement of the subject's eye gaze point during the execution of the eye movement task; Extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; Extracting time features based on all picture frames in the eye movement video data to determine a one-dimensional time feature of the eye movement video data; Performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain eye movement features of the eye movement video data; Classification is performed based on the eye movement characteristics and the target task category to which the eye movement task belongs, so as to obtain the completion status of the eye movement task.

2. The method according to claim 1, characterized in that The step of extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data includes: Extracting spatial features from picture frames in the eye movement video data to obtain a spatial feature map; Performing dimensionality reduction processing on the spatial feature map by global average pooling to obtain spatial distribution weight data; The spatial feature map is weighted using the spatial distribution weight data to obtain the two-dimensional spatial feature.

3. The method according to claim 1, characterized in that The step of extracting time features based on all frames in the eye movement video data to determine a one-dimensional time feature of the eye movement video data includes: Extracting time features from all frames in the eye movement video data to obtain a time feature graph; Performing dimensionality reduction processing on the temporal feature graph by global maximum pooling to obtain temporal distribution weight data; The time characteristic graph is weighted by using the time distribution weight data to obtain the one-dimensional time characteristic.

4. The method according to claim 1, characterized in that The step of performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain the eye movement features of the eye movement video data includes: The one-dimensional time feature and the two-dimensional space feature are fused in series to obtain the eye movement feature.

5. The method according to claim 1, characterized in that The step of performing spatiotemporal fusion on the two-dimensional spatial features and the one-dimensional temporal features to obtain the eye movement features of the eye movement video data includes: The one-dimensional time feature and the two-dimensional space feature are fused in parallel to obtain the eye movement feature.

6. The method according to claim 1, characterized in that The classifying based on the eye movement feature and the target task category of the eye movement task to obtain the completion status of the eye movement task includes: Mapping is performed according to the eye movement features to determine whether the eye movement task is completed correctly or incorrectly under the target task category.

7. The method according to any one of claims 1 to 6, characterized in that: The two-dimensional spatial feature is obtained by extracting features through a two-dimensional convolution module, and the one-dimensional time feature is obtained by extracting features through a one-dimensional convolution module. The two-dimensional convolution module is connected in series or in parallel with the one-dimensional convolution module.

8. An eye movement video data processing device, characterized in that: The device comprises: A video data acquisition module, used to acquire eye movement video data; wherein the eye movement video data is acquired by collecting the movement of the subject's eye gaze point during the execution of the eye movement task; A spatial feature extraction module, used for extracting features from each frame of the eye movement video data to obtain two-dimensional spatial features of the eye movement video data; A time feature extraction module, used for extracting time features based on all frames in the eye movement video data to determine a one-dimensional time feature of the eye movement video data; A spatiotemporal feature fusion module, used for spatiotemporally fusing the two-dimensional spatial feature and the one-dimensional temporal feature to obtain the eye movement feature of the eye movement video data; The eye movement task classification module is used to classify the eye movement task based on the eye movement characteristics and the target task category to which the eye movement task belongs, so as to obtain the completion status of the eye movement task.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Real-time action positioning method based on time-space cross attention

    CN115019239A

  • Spatial-temporal feature fusion nystagmus video classification method and system

    CN115482408A

  • LED display screen brightness adjusting system and method thereof

    CN116645917A

  • Method and system for regulating and controlling brightness of corn lamp without stroboflash

    CN116963365A

  • Eye movement feature map acquisition method and device, terminal and storage medium

    CN117373105A