Time sequence and time frequency fusion driven high-precision eye movement behavior detection method

Abstract

The embodiment of the invention relates to the technical field of eye movement behavior detection, and discloses a high-precision eye movement behavior detection method driven by time sequence and time frequency fusion, and the method comprises the steps: carrying out the short-time Fourier transform of an original eye movement data sequence of a collected to-be-detected eye movement behavior, converting the original eye movement data sequence of the time domain into time-frequency domain information; performing feature extraction on time sequence information in the original eye movement data sequence by adopting a plurality of parallel Mamba networks to obtain time sequence features, and performing feature extraction on time-frequency domain information by using a convolutional neural network to obtain time-frequency features; splicing and fusing the time sequence features and the time frequency features to obtain fused features; inputting the fusion features into a bidirectional gating cycle unit network for sequence modeling; and performing classification prediction on the output characteristics of the bidirectional gating loop unit network by using a full connection layer to obtain a detection result of the to-be-detected eye movement behavior. The method effectively improves the accuracy and robustness of eye movement behavior detection.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of eye movement behavior detection, and in particular to a high-precision eye movement behavior detection method driven by time series and time-frequency fusion. Background Art

[0002] Eye movement behavior detection technology is playing an increasingly important role as a key research tool in psychology, neuroscience, human-computer interaction and other fields. By tracking and analyzing eye movements, researchers can reveal the distribution of human attention, cognitive processes and visual information processing mechanisms in different situations. This technology has been widely used in user interface design, advertising effectiveness evaluation, driver fatigue detection, clinical psychology diagnosis, education research and other fields.

[0003] Most of the eye movement detection technologies proposed so far rely on time series data, using traditional classification algorithms to identify and classify different eye movement events, such as fixations, saccades, and late tremors. These technologies usually only focus on the time information in the sequence, process the structure and characteristics of the time series itself, and have achieved certain results.

[0004] However, in addition to the time dimension information, time series data also contains rich frequency information. These time-frequency domain features are also of great value for accurately detecting eye movement behavior. Most traditional methods ignore the frequency domain information of time series data and fail to effectively integrate time and frequency information for eye movement behavior detection. The lack of time-frequency domain information will make it difficult to accurately identify some complex eye movement behaviors, which in turn affects the overall detection accuracy of the eye movement behavior detection system. Therefore, how to combine time and frequency information to improve the accuracy and robustness of eye movement behavior detection has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the present application proposes a high-precision eye movement behavior detection method driven by timing and time-frequency fusion, which effectively improves the accuracy and robustness of eye movement behavior detection by combining timing information and frequency domain information.

[0006] In order to achieve the above-mentioned purpose, an embodiment of the present application proposes a high-precision eye movement behavior detection method driven by time series and time-frequency fusion, comprising the following steps: performing a short time Fourier transform (STFT) on the collected original eye movement data sequence of the eye movement behavior to be detected, and converting the original eye movement data sequence in the time domain into time-frequency domain information; using multiple parallel Mamba networks to extract features of the time series information in the original eye movement data sequence to obtain time series features, and using a convolutional neural network to extract features of the time-frequency domain information to obtain time-frequency features; splicing and fusing the time series features and the time-frequency features to obtain fused features; inputting the fused features into a bidirectional gated recurrent unit network for sequence modeling to capture global dependencies; using a fully connected layer to classify and predict the output features of the bidirectional gated recurrent unit network to obtain detection results for the eye movement behavior to be detected.

[0007] In order to achieve the above-mentioned purpose, an embodiment of the present application also proposes a high-precision eye movement behavior detection system driven by time series and time-frequency fusion, including: an STFT preprocessing module, which is used to perform short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, and convert the original eye movement data sequence in the time domain into time-frequency domain information; a time series extraction module, which is used to use multiple parallel Mamba networks to extract features of the time series information in the original eye movement data sequence to obtain time series features; a time-frequency extraction module, which is used to use a convolutional neural network to extract features of the time-frequency domain information to obtain time-frequency features; a splicing and fusion module, which is used to splice and fuse the time series features and the time-frequency features to obtain fused features; a sequence modeling module, which is used to input the fused features into a bidirectional gated recurrent unit network for sequence modeling to capture global dependencies; a classification prediction module, which is used to use a fully connected layer to classify and predict the output features of the bidirectional gated recurrent unit network to obtain detection results for the eye movement behavior to be detected.

[0008] In order to achieve the above-mentioned purpose, an embodiment of the present application also proposes an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a high-precision eye movement behavior detection method driven by timing and time-frequency fusion as described above.

[0009] In order to achieve the above objectives, an embodiment of the present application also proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a high-precision eye movement behavior detection method driven by timing and time-frequency fusion as described above.

[0010] The embodiment of the present application proposes a high-precision eye movement behavior detection method driven by time series and time-frequency fusion, which performs a short-time Fourier transform on the original eye movement data sequence, and converts the original eye movement data sequence in the time domain into time-frequency domain information in order to extract time-frequency features. This effectively captures the frequency information that is not fully utilized in traditional methods, and the introduction of time-frequency domain information significantly improves the accuracy of eye movement behavior classification. Subsequently, multiple parallel Mamba networks are used to process long-distance dependent sequence information to extract time series features, and convolutional neural networks are combined to extract time-frequency features. Through the parallel structure and smaller convolution kernel design, the model can reduce the model size and the number of parameters while maintaining the ability to express complex features, effectively improving the computational efficiency. Afterwards, the timing features and time-frequency features are concatenated and fused to obtain fused features, and the fused features are input into the bidirectional gated recurrent unit network for sequence modeling to capture global dependencies. Finally, a fully connected layer is used to classify and predict the output features of the bidirectional gated recurrent unit network to obtain the detection results of the eye movement behavior to be detected. This not only solves the technical difficulties of integrating sequence and frequency information, but also significantly improves the accuracy and robustness of classification and detection of eye movement behaviors such as fixation, saccade, and late tremor.

[0011] In some optional embodiments, a short-time Fourier transform is performed on the collected original eye movement data sequence of the eye movement behavior to be detected, and the conversion of the original eye movement data sequence in the time domain into time-frequency domain information includes: performing a short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, expanding the time information of the original eye movement data sequence to the frequency dimension, capturing the frequency change characteristics of different time points, and thus converting the original eye movement data sequence in the time domain into time-frequency domain information; in the process of performing the short-time Fourier transform, the continuity and integrity of the time-frequency domain information are ensured by setting the length and overlap of the time window.

[0012] In some optional embodiments, the plurality of parallel Mamba networks are composed of three parallel branches, each branch is composed of a linear layer and a state space model connected in series, and the number of channels of each branch is 32.

[0013] In some optional embodiments, for an input X={x1, x2, ..., x t ,…,x T}, after being mapped by the linear layer, the output of the linear layer is expressed by the formula:

[0014] h t =W·x t +b;

[0015] Among them, W is the weight matrix of linear transformation, b is the bias term, and x t is the tth item in the original eye movement data sequence, ht is the tth item of the output of the linear layer;

[0016] The output of the linear layer is further processed by the state-space model, and the output of the state-space model is expressed as:

[0017] y t =C·s t +D·h t ;

[0018] s t =A·s t-1 +B·h t-1 ;

[0019] Among them, A is the state transfer matrix, B is the input mapping matrix, C and D are the observation matrices, and s t represents the hidden state at the current time step t, s t-1 represents the hidden state of the previous time step t-1, y t is the tth item of the output of the state-space model.

[0020] In some optional embodiments, the convolutional neural network uses a convolution kernel size of (3,7), a step size of (1,1), a number of layers of 3, and a number of channels of each layer of 32.

[0021] In some optional embodiments, the splicing and fusing of the time series features and the time-frequency features to obtain the fused features includes: splicing the outputs of the state space model of the three parallel branches based on the channel dimension, and using a linear layer to map the number of channels after splicing to 32 to obtain the mapped time series features; keeping the length dimension of the time-frequency features unchanged, fusing the width dimension to the channel dimension, and using a linear layer to map the fused channel dimension to 64 to obtain the mapped time-frequency features; splicing the mapped time series features and the mapped time-frequency features based on the channel dimension to obtain the fused features.

[0022] In some optional embodiments, the bidirectional gated recurrent unit network is composed of 4 layers of bidirectional gated recurrent units, the dimension of its hidden layer is 64, and the dimension of its output is also 64. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1It is a flowchart of a high-precision eye movement behavior detection method driven by time series and time-frequency fusion provided in one embodiment of the present application;

[0025] Figure 2 It is a schematic diagram of the principle of a high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion provided in one embodiment of the present application;

[0026] Figure 3 is a structural schematic diagram of a high-precision eye movement behavior detection model provided in one embodiment of the present application;

[0027] Figure 4 is a schematic diagram of the structure of a Mamba network provided in one embodiment of the present application;

[0028] Figure 5 is a structural schematic diagram of a high-precision eye movement behavior detection system driven by time sequence and time-frequency fusion provided in another embodiment of the present application;

[0029] Figure 6 It is a structural schematic diagram of an electronic device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0031] An embodiment of the present application proposes a high-precision eye movement behavior detection method driven by timing and time-frequency fusion, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described by taking the server as an example. The implementation details of the high-precision eye movement behavior detection method driven by timing and time-frequency fusion proposed in this embodiment are described in detail below. The following content is only the relevant implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0032] The specific process of the high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0033] Step 101 , performing short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, and converting the original eye movement data sequence in the time domain into time-frequency domain information.

[0034] In a specific implementation, when there is a need to detect eye movement behavior, the server first collects the original eye movement data sequence of the eye movement behavior to be detected, and then performs a short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, thereby converting the original eye movement data sequence in the time domain into time-frequency domain information.

[0035] In one example, the server performs short-time Fourier transform on the collected raw eye movement data sequence of the eye movement behavior to be detected, expands the time information of the raw eye movement data sequence to the frequency dimension, thereby capturing the frequency change characteristics at different time points, and then converts the raw eye movement data sequence in the time domain into time-frequency domain information. In the process of performing short-time Fourier transform, the server ensures the continuity and integrity of the time-frequency domain information by reasonably setting the length and overlap of the time window.

[0036] In one example, the server may perform a short-time Fourier transform on the original eye movement data sequence using a rectangular window with a size of 6 and a step size of 1.

[0037] Step 102, using multiple parallel Mamba networks to extract features from the time series information in the original eye movement data sequence to obtain time series features, and using a convolutional neural network to extract features from the time-frequency domain information to obtain time-frequency features.

[0038] In the specific implementation, after obtaining the time-frequency domain information, the server has completed the data preprocessing process and then enters the eye movement behavior recognition process, including feature extraction, feature fusion, feature modeling and classification prediction. This process is actually implemented by the eye movement behavior detection model. The structure of the eye movement behavior detection model can be as follows: Figure 2 As shown in the figure, it can be roughly divided into feature extraction network, feature fusion network, feature modeling network and classification network. The feature extraction network consists of two parts, namely multiple parallel Mamba networks and a convolutional neural network. Multiple parallel Mamba networks are used to extract features of the temporal information in the original eye movement data sequence to obtain temporal features. Convolutional neural networks are used to extract features of time-frequency domain information to obtain time-frequency features.

[0039] In an example, the specific structure of a Mamba network can be as follows Figure 3As shown in the figure, multiple parallel Mamba networks are composed of three parallel branches, each of which is composed of a linear layer and a state space model in series, and the number of channels in each branch is 32. Multiple parallel Mamba networks can well handle the long-distance dependency characteristics of the original eye movement data sequence, thereby capturing the subtle and complex dynamic changes in the original eye movement data sequence.

[0040] For input X={x1,x2,…,x t ,…,x T}, after being mapped by the linear layer, the output of the linear layer is expressed by the formula:

[0041] h t =W·x t +b;

[0042] Among them, W is the weight matrix of linear transformation, b is the bias term, and x t is the tth item in the original eye movement data sequence, h t is the tth item of the output of the linear layer;

[0043] The output of the linear layer is further processed by the state-space model, and the output of the state-space model is expressed as:

[0044] y t =C·s t +D·h t ;

[0045] s t =A·s t-1 +B·h t-1 ;

[0046] Among them, A is the state transfer matrix, B is the input mapping matrix, C and D are the observation matrices, and s t represents the hidden state at the current time step t, s t-1 represents the hidden state of the previous time step t-1, y t is the tth item of the output of the state-space model.

[0047] In one example, the convolutional neural network uses a convolution kernel size of (3,7), a stride of (1,1), 3 layers, and 32 channels in each layer. Multi-level convolution operations can extract local patterns and multi-scale information of time-frequency features, which is crucial for identifying eye movement changes in a short period of time.

[0048] Step 103, concatenate and fuse the time series features and the time-frequency features to obtain fused features.

[0049] In the specific implementation, after completing the feature extraction, we can enter the feature fusion process, which is realized by the fusion network of the eye movement behavior detection model. The fusion network splices and fuses the time series features and time-frequency features to obtain the fusion features.

[0050] In one example, the fusion network first concatenates the outputs of the state-space model of the three parallel branches based on the channel dimension, and uses a linear layer to map the number of channels after concatenation to 32 to obtain the mapped time series features. After that, the time-frequency features retain the length dimension unchanged, the width dimension is fused to the channel dimension, and a linear layer is used to map the fused channel dimension to 64 to obtain the mapped time-frequency features. Finally, the mapped time series features and the mapped time-frequency features are concatenated based on the channel dimension to obtain the fused features. The feature fusion process maps the number of channels to a unified dimension, ensuring that the model can simultaneously use the information of these two types of features for a more comprehensive eye movement behavior analysis.

[0051] Step 104, input the fused features into a bidirectional gated recurrent unit network for sequence modeling to capture global dependencies.

[0052] In the specific implementation, after completing feature fusion, feature modeling can be performed, which is achieved by the feature modeling network. The feature modeling network uses a bidirectional gated recurrent unit network. The fused features are input into the bidirectional gated recurrent unit network for sequence modeling, which can capture forward and backward dependencies and global dependencies.

[0053] In one example, a bidirectional gated recurrent unit network consists of 4 layers of bidirectional gated recurrent units (bi-GRU), with a hidden layer dimension of 64 and an output dimension of 64.

[0054] Step 105: Use the fully connected layer to classify and predict the output features of the bidirectional gated recurrent unit network to obtain a detection result of the eye movement behavior to be detected.

[0055] In the specific implementation, after completing feature modeling, the final classification prediction can be performed. The classification prediction is implemented by a fully connected layer. The output features of the bidirectional gated recurrent unit network are classified and predicted using the fully connected layer to obtain the detection results of the eye movement behavior to be detected. The fully connected layer maps high-dimensional features to a specific classification space (such as fixation, saccade, post-saccade tremor, etc.), and its output is a specific classification probability to ensure the accuracy and robustness of eye movement behavior detection.

[0056] In one example, when iteratively training an eye movement behavior detection model, the cross entropy loss function can be used to calculate the classification error to guide model training. The cross entropy loss function can effectively measure the difference between the model prediction result and the actual label, and adjust the model parameters through the error feedback mechanism, and finally obtain a trained eye movement behavior detection model.

[0057] In this embodiment, short-time Fourier transform is first performed on the original eye movement data sequence, and the original eye movement data sequence in the time domain is converted into time-frequency domain information in order to extract time-frequency features, which effectively captures the frequency information that is not fully utilized in the traditional eye movement behavior detection method, and the introduction of time-frequency domain information significantly improves the accuracy of eye movement behavior classification. Subsequently, multiple parallel Mamba networks are used to process the sequence information of long-distance dependence to extract the time series features, and the time-frequency features are extracted in combination with the convolutional neural network. Through the parallel structure and the design of a smaller convolution kernel, the model can be reduced in size and parameter amount while maintaining the ability to express complex features, effectively improving the computational efficiency. After that, the time series features and the time-frequency features are spliced ​​and fused to obtain fused features, and the fused features are input into the bidirectional gated recurrent unit network for sequence modeling, capturing global dependencies, and finally using the fully connected layer to classify and predict the output features of the bidirectional gated recurrent unit network to obtain the detection results of the eye movement behavior to be detected, which not only solves the technical difficulties of integrating sequence and frequency information, but also significantly improves the classification and detection accuracy and robustness of eye movement behaviors such as gaze, saccade, and late tremor.

[0058] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0059] Another embodiment of the present application proposes a high-precision eye movement behavior detection system driven by timing and time-frequency fusion. The implementation details of the high-precision eye movement behavior detection system driven by timing and time-frequency fusion of this embodiment are described in detail below. The following content is only the relevant implementation details provided for easy understanding and is not necessary for the implementation of this solution.

[0060] The specific structure of the high-precision eye movement behavior detection system driven by time sequence and time-frequency fusion proposed in this embodiment can be as follows: Figure 5 As shown, it includes: an STFT preprocessing module 201, a time series extraction module 202, a time-frequency extraction module 203, a splicing and fusion module 204, a sequence modeling module 205 and a classification prediction module 206.

[0061] The STFT preprocessing module 201 is used to perform short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, and convert the original eye movement data sequence in the time domain into time-frequency domain information.

[0062] The time series extraction module 202 is used to extract features of the time series information in the original eye movement data sequence using multiple parallel Mamba networks to obtain time series features.

[0063] The time-frequency extraction module 203 is used to extract features from the time-frequency domain information using a convolutional neural network to obtain time-frequency features.

[0064] The splicing and fusion module 204 is used to splice and fuse the time series features and the time-frequency features to obtain fused features.

[0065] The sequence modeling module 205 is used to input the fused features into the bidirectional gated recurrent unit network for sequence modeling to capture global dependencies.

[0066] The classification prediction module 206 is used to use the fully connected layer to perform classification prediction on the output features of the bidirectional gated recurrent unit network to obtain the detection result of the eye movement behavior to be detected.

[0067] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0068] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above method embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above method embodiment.

[0069] In another embodiment, in order to verify the superiority of the high-precision eye movement behavior detection method driven by time series and time-frequency fusion (hereinafter referred to as OURS), we conducted comparative experiments on OURS and other methods on the same dataset Lund2013. The experimental results are shown in Table 1.

[0070] Table 1: Event-level Cohen's Kappa comparison of OURS and other methods

[0071] method Watch Eye saccade Post-saccadic concussion NH2010 0.639 0.798 0.350 MNH 0.837 0.759 0.598 IRF 0.780 0.848 0.616 gazeNet 0.959 0.947 0.776 OURS 0.969 0.950 0.803

[0072] As can be seen from Table 1, the accuracy of OURS in the three-category eye movement behavior detection of fixation, saccade and post-saccade oscillation is significantly higher than that of other methods, especially in the detection of fixation, which is improved by more than 1.5% compared with traditional methods.

[0073] Another embodiment of the present application provides an electronic device, the structure of which can be as follows: Figure 6 As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute a high-precision eye movement behavior detection method driven by a timing and time-frequency fusion as described in the above-mentioned method embodiments.

[0074] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects the circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0075] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0076] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a high-precision eye movement behavior detection method driven by timing and time-frequency fusion as described in the above method embodiments.

[0077] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (such as a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0078] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion, characterized in that: include: Performing short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, and converting the original eye movement data sequence in the time domain into time-frequency domain information; Multiple parallel Mamba networks are used to extract the time series information in the original eye movement data sequence to obtain time series features, and convolutional neural networks are used to extract the time-frequency domain information to obtain time-frequency features. The time series features and time-frequency features are concatenated and fused to obtain fused features; The fused features are input into a bidirectional gated recurrent unit network for sequence modeling to capture global dependencies; The fully connected layer is used to classify and predict the output features of the bidirectional gated recurrent unit network to obtain the detection results of the eye movement behavior to be detected.

2. According to claim 1, a high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion, characterized in that: The short-time Fourier transform is performed on the collected original eye movement data sequence of the eye movement behavior to be detected, and the original eye movement data sequence in the time domain is converted into time-frequency domain information, including: Performing short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, expanding the time information of the original eye movement data sequence to the frequency dimension, capturing the frequency change characteristics at different time points, and thus converting the original eye movement data sequence in the time domain into time-frequency domain information; In the process of short-time Fourier transform, the length and overlap of the time window are set to ensure the continuity and integrity of the time-frequency domain information.

3. According to claim 1, a high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion, characterized in that: Multiple parallel Mamba networks are composed of three parallel branches, each of which is composed of a linear layer and a state space model in series, and the number of channels in each branch is 32.

4. According to claim 3, a high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion, characterized in that: For input X={x1,x2,…,x t ,…,x T }, after being mapped by the linear layer, the output of the linear layer is expressed by the formula: h t =W·x t +b; Among them, W is the weight matrix of linear transformation, b is the bias term, and x t is the tth item in the original eye movement data sequence, h t is the tth item of the output of the linear layer; The output of the linear layer is further processed by the state-space model, and the output of the state-space model is expressed as: y t =C·s t +D·h t ; s t =A·s t-1 +B·h t-1 ; Among them, A is the state transfer matrix, B is the input mapping matrix, C and D are the observation matrices, and s t represents the hidden state at the current time step t, s t-1 represents the hidden state of the previous time step t-1, y t is the tth item of the output of the state-space model.

5. According to claim 3, a high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion, characterized in that: The convolutional neural network uses a convolution kernel size of (3,7), a stride of (1,1), 3 layers, and 32 channels in each layer.

6. According to claim 5, a high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion, characterized in that: The splicing and fusing of the time series features and the time-frequency features to obtain the fused features includes: The outputs of the state space model of the three parallel branches are concatenated based on the channel dimension, and the number of channels after concatenation is mapped to 32 using a linear layer to obtain the mapped time series features; The length dimension of the time-frequency feature is kept unchanged, the width dimension is fused into the channel dimension, and the fused channel dimension is mapped to 64 using a linear layer to obtain the mapped time-frequency feature; The mapped time series features and the mapped time-frequency features are concatenated based on the channel dimension to obtain fused features.

7. A high-precision eye movement behavior detection method driven by time sequence and time-frequency fusion according to any one of claims 1 to 6, characterized in that: The bidirectional gated recurrent unit network consists of 4 layers of bidirectional gated recurrent units, with a hidden layer dimension of 64 and an output dimension of 64.

8. A high-precision eye movement behavior detection system driven by time sequence and time-frequency fusion, characterized in that: include: The STFT preprocessing module is used to perform short-time Fourier transform on the collected original eye movement data sequence of the eye movement behavior to be detected, and convert the original eye movement data sequence in the time domain into time-frequency domain information; A time series extraction module is used to extract the time series information in the original eye movement data sequence using multiple parallel Mamba networks to obtain time series features; The time-frequency extraction module is used to extract features from the time-frequency domain information using a convolutional neural network to obtain time-frequency features; The splicing and fusion module is used to splice and fuse the time series features and the time-frequency features to obtain fusion features; The sequence modeling module is used to input the fused features into the bidirectional gated recurrent unit network for sequence modeling and capture global dependencies; The classification prediction module is used to use the fully connected layer to classify and predict the output features of the bidirectional gated recurrent unit network to obtain the detection results of the eye movement behavior to be detected.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a high-precision eye movement behavior detection method driven by timing and time-frequency fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a high-precision eye movement behavior detection method driven by time series and time-frequency fusion as described in any one of claims 1 to 7 is implemented.

Images